A connector to pinecone.io
builtin
connector. If you want to create a non-builtin
connector,
you need to replace --sink-type pinecone
with --archive /path/to/pulsar-io-pinecone.nar
. You can find the button to download the nar
package at the beginning of the document.
--sink-config
is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see Configuration Properties for reference.
pulsar-admin
are similar to those of pulsarctl
. You can find an example for StreamNative Cloud Doc.Name | Type | Required | Sensitive | Default | Description |
---|---|---|---|---|---|
apiKey | string | True | True | None | The API key for the Pinecone service. Find this in the Pinecone dashboard. |
indexName | string | True | False | None | The name of the Pinecone index to which you want to write data. Find this in the Pinecone dashboard. |
namespace | string | True | False | None | The name of the Pinecone namespace to which you want to write data. Find this in the Pinecone dashboard. |
dimensions | integer | False | False | None | The number of dimensions required by the index. If a request is made to upsert data into an index with a different number of dimensions, the request will fail. If not provided the connector will make it’s best attempt to upsert the data and if the connection fails due to a mismatch the message will eventually be DLQ’d. |
queryMetadata | JSON | False | False | None | The metadata to be associated with the request to the index.This should be a JSON object in the form {"key": "value", "key2": "value2" } . |
pinecone-upsert-successful
pinecone-upsert-failed
pinecone-connector-active
pinecone-upsert-failed-no-config
pinecone-upsert-failed-no-client
pinecone-upsert-failed-no-index-connection
pinecone-upsert-failed-parsing-error
pinecone-upsert-failed-dimension-error
at-most-once
, at-least-once
, and effectively-once
.
Currently, the Pinecone sink connector provides the at-least-once delivery guarantee.
mvn clean install
.
Similar configuration can be setup when using an image mounted with a config file
defining environment variables or when using in Kubernetes.
This table lists the schema types that currently are supported to be converted.
Schema | Supported | |
---|---|---|
AVRO | No | |
PRIMITIVE | Yes | (only bytes and string) |
PROTOBUF_NATIVE | Yes | |
PROTOBUF | No | |
JSON | Yes | |
KEY_VALUE | No |