Skip to content

Commit

Permalink
Update readme examples
Browse files Browse the repository at this point in the history
  • Loading branch information
buinauskas committed Sep 20, 2023
1 parent ad82640 commit 020b0d5
Show file tree
Hide file tree
Showing 8 changed files with 66 additions and 23 deletions.
27 changes: 15 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,11 @@ This connector has not yet been published to Confluent Hub. To install it, downl
install it using `confluent-hub` command line tool.

```sh
wget https://github.com/vinted/kafka-connect-vespa/releases/download/v1.0.2/vinted-kafka-connect-vespa-1.0.2-SNAPSHOT.zip -O /tmp/vinted-kafka-connect-vespa-1.0.2-SNAPSHOT.zip -q
wget https://github.com/vinted/kafka-connect-vespa/releases/download/v1.0.3/vinted-kafka-connect-vespa-1.0.3-SNAPSHOT.zip -O /tmp/vinted-kafka-connect-vespa-1.0.3-SNAPSHOT.zip -q
```

```sh
confluent-hub install --no-prompt /tmp/vinted-kafka-connect-vespa-1.0.2-SNAPSHOT.zip
confluent-hub install --no-prompt /tmp/vinted-kafka-connect-vespa-1.0.3-SNAPSHOT.zip
```

### Important
Expand Down Expand Up @@ -62,8 +62,8 @@ also warrant a higher number of connections.

`vespa.max.streams.per.connection`
This determines the maximum number of concurrent, in-flight requests for
this Sets the maximum number of streams per HTTP/2 client, which is
maxConnections \* maxStreamsPerConnection. Prefer more streams over more
these Sets the maximum number of streams per HTTP/2 client, which is
maxConnections \* maxStreamsPerConnection. Prefer more streams to more
connections, when possible. The feed client automatically throttles load
to achieve the best throughput, and the actual number of streams per
connection is usually lower than the maximum.
Expand Down Expand Up @@ -100,7 +100,7 @@ The period of consecutive failures before shutting down.
`vespa.namespace`
User specified part of each document ID in that sense. Namespace can not
be used in queries, other than as part of the full document ID. However,
it can be used for document selection, where id.namespace can be
it can be used for document selection, where namespace can be
accessed and compared to a given string, for instance. An example use
case is visiting a subset of documents. Defaults to topic name if not specified.

Expand Down Expand Up @@ -183,6 +183,9 @@ Valid options are ignore', 'warn', and 'fail'.

#### Examples

Connector configuration examples can be found in the [config](config) directory. But here are some quick ones to get
started.

##### Standalone Example

This configuration is used typically along
Expand All @@ -196,8 +199,8 @@ topics=music
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
vespa.endpoint=http://vespa:8080/
vespa.namespace=mynamespace
vespa.document.type=mydocumenttype
vespa.namespace=default
vespa.document.type=music
```

##### Distributed Example
Expand All @@ -216,23 +219,23 @@ the distributed connect worker(s).
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"vespa.endpoint": "http://vespa:8080/",
"vespa.namespace": "mynamespace",
"vespa.document.type": "mydocumenttype"
"vespa.namespace": "default",
"vespa.document.type": "music"
}
}
```

Use curl to post the configuration to one of the Kafka Connect Workers. Change `http://localhost:8088/` the endpoint of
Use curl to post the configuration to one of the Kafka Connect Workers. Change `http://localhost:8083/` the endpoint of
one of your Kafka Connect worker(s).

Create a new instance.

```bash
curl -s -X POST -H 'Content-Type: application/json' --data @connector.json http://localhost:8088/connectors
curl -s -X POST -H 'Content-Type: application/json' --data @connector.json http://localhost:8083/connectors
```

Update an existing instance.

```bash
curl -s -X PUT -H 'Content-Type: application/json' --data @connector.json http://localhost:8088/connectors/TestSinkConnector1/config
curl -s -X PUT -H 'Content-Type: application/json' --data @connector.json http://localhost:8083/connectors/TestSinkConnector1/config
```
2 changes: 1 addition & 1 deletion config/quickstart-vespa-json-converter.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "false",
"vespa.url": "http://vespa:8080/",
"vespa.namespace": "mynamespace",
"vespa.namespace": "default",
"vespa.document.type": "music"
}
}
22 changes: 22 additions & 0 deletions config/quickstart-vespa-malformed-documents-with-dlq.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"config": {
"name": "VespaSinkConnector1",
"connector.class": "com.vinted.kafka.connect.vespa.VespaSinkConnector",
"tasks.max": "1",
"topics": "music",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"vespa.url": "http://vespa:8080/",
"vespa.namespace": "default",
"vespa.document.type": "music",
"vespa.dryrun": "false",
"vespa.drop.invalid.message": "true",
"vespa.behavior.on.malformed.documents": "WARN",
"errors.tolerance": "all",
"errors.log.enable": "true",
"errors.log.include.messages": "true",
"errors.deadletterqueue.topic.name": "dlq",
"errors.deadletterqueue.topic.replication.factor": 1,
"errors.deadletterqueue.context.headers.enable": "true"
}
}
10 changes: 2 additions & 8 deletions config/quickstart-vespa-malformed-documents.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,10 @@
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"vespa.url": "http://vespa:8080/",
"vespa.namespace": "mynamespace",
"vespa.namespace": "default",
"vespa.document.type": "music",
"vespa.dryrun": "false",
"vespa.drop.invalid.message": "true",
"vespa.behavior.on.malformed.documents": "WARN",
"errors.tolerance": "all",
"errors.log.enable": "true",
"errors.log.include.messages": "true",
"errors.deadletterqueue.topic.name": "dlq",
"errors.deadletterqueue.topic.replication.factor": 1,
"errors.deadletterqueue.context.headers.enable": "true"
"vespa.behavior.on.malformed.documents": "WARN"
}
}
12 changes: 12 additions & 0 deletions config/quickstart-vespa-raw-operations.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"config": {
"name": "VespaSinkConnector1",
"connector.class": "com.vinted.kafka.connect.vespa.VespaSinkConnector",
"tasks.max": "1",
"topics": "music",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"vespa.url": "http://vespa:8080/",
"vespa.operational.mode": "raw"
}
}
12 changes: 12 additions & 0 deletions config/quickstart-vespa-topic-as-documenttype.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"config": {
"name": "VespaSinkConnector1",
"connector.class": "com.vinted.kafka.connect.vespa.VespaSinkConnector",
"tasks.max": "1",
"topics": "music",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"vespa.url": "http://vespa:8080/",
"vespa.namespace": "default"
}
}
2 changes: 1 addition & 1 deletion config/quickstart-vespa.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"vespa.url": "http://vespa:8080/",
"vespa.namespace": "mynamespace",
"vespa.namespace": "default",
"vespa.document.type": "music"
}
}
2 changes: 1 addition & 1 deletion config/quickstart-vespa.properties
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,5 @@ topics=music
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
vespa.url=http://vespa:8080/
vespa.namespace=mynamespace
vespa.namespace=default
vespa.document.type=music

0 comments on commit 020b0d5

Please sign in to comment.