Add generic EventSource and EventSink

[matzew]

Knative Eventing offers only a small number of event sources, such as ApiServerSource or PingSource. Besides that it provides a ContainerSource to allow users to run their own integrations, leveraging the SinkBinding API. The same is true for event sinks with the JobSink and the KafkaSink in the eventing-kafka-broker repo.

In order to offer a rich set of Event Sources and Event Sinks for Knative Eventing, we would like to add two new generic CRDs: EventSource and EventSink.

Initially the new CRDs would leverage a fixed set of Apache Camel Kamelets. Those Kamelets would run as regular deployments and in the case of "sources" those Kamelets would leverage the SinkBinding API. Behind the new CRD these Kamelets acting either as true, native "source" or "sink" for Knative.

With Kamelets we can provide connectivity for many products and services, including Amazon Web Services, Azure, Google Cloud, or even traditional messaging systems like AMQP 1.0.

Essentially the Event Source/Sink CRDs would act like a shim to propagate configuration to the underlying Deployment of the Kamelets being used..

There will be two new APIs. At a high-level they would be like:

apiVersion: sources.knative.dev/v1alpha1
kind: EventSource
metadata:
  ...
spec:
  {tbd}
  sink:
    ref:
      apiVersion: eventing.knative.dev/v1
      kind: Broker
      name: default

and for the "sinks"

apiVersion: eventing.knative.dev/v1
kind: EventSink
metadata:
  ...
spec:
  {tbd}

NEXT: In a dedicated feature track I want to go into details on the API proposal as well as the implementation and requirements for this new proposal.

[matzew]

I have created a feature track for this: https://docs.google.com/document/d/1UOEbi2kBgH65QaYa4fhvZRxlkUWrqKrptbe9PFDw1bc/edit

[tylerbowers]

Will this be a step in for many of the Triggermesh event sources? I just started using Knative and got excited when I saw those event sources from Triggermesh in the knative docs. This excitement wore off when I found out that Triggermesh disbanded last year and there's no one maintaining the project anymore.

[matzew]

@tylerbowers hey Tyler! Thanks for reaching out.

The idea of of this is to add two new "generic" types:

• IntegrationSource
• IntegrationSink

Behind the secenes we are leveraging Apache Camel Kamelets for the connectivity. See here for the currently available supported "sources" or "sinks" -> https://github.com/knative-extensions/eventing-integrations

[tylerbowers]

@matzew Really cool stuff, thanks for the clarification.

[mukundjalan]

As pointed by @pierDipi I am requesting to add a source that can do something similar to what is mentioned in triggermesh/triggermesh#1544 (comment)

[matzew]

@mukundjalan we have already (not yet released, but on main) some support for AWS Source:

• sqs
• s3
• ddbStreams

here is a PR that drafts an initial doc:

• Adding first draft of IntegrationSource doc docs#6156

And some preview rendering

NOTE: there will be api changes. next knative release is January - but the nightly builds have this already.

[matzew]

@tikr7 take a look at the comment above. That shows the "simple SQS" source for Knative Eventing.
This is part of the next 1.17 release, and directly included into the "core" installation

[matzew]

https://knative.dev/blog/articles/consuming_sqs_data_with_integrationsource/

is an early preview blog post, using SQS

[tikr7]

@matzew this looks very promising! I can't wait for it :)

[matzew]

@mukundjalan with the SQS option, as discussed here it is possible to receive notifications from the EventBridge.

In AWS console, for your eventbus / EventBridge, create a Rule, and use an SQS queue as target. From this SQS queue you can receive the CloudEvent notifications in Knative