metadataServiceUri | The metadata service uri for bookkeeper client, for example, zk://localhost:2181/ledgers | Y | N/A |
pulsarWebServiceUrl | The pulsar web service URL, for example, http://localhost:8080 | Y | N/A |
pulsarServiceUrl | The pulsar protocol service URL, for example, pulsar://localhost:6650 | Y | N/A |
offloadProvider | The offloader driver’s name, for example, delta or iceberg | Y | N/A |
maxContainerCountPerTopic | The maximum containers will be used by a single topic. | N | 2 |
containerSizeInBytes | The maximum bytes for each container. By default, the container can fill with 24 MB messages to write. | N | 25165824 |
enableSharedMessageContainer | Enable shared message container or not. Once the shared message container is enabled, all the offload processors will get the container from the shared message container pool | N | true |
sharedMessageContainerPoolMemory | The max shared message container pool memory size. Default: 0.125 * maxDirectMemorySize | N | 0.125 * maxDirectMemory |
offloadSubName | offload subscription name | N | __OFFLOAD |
initialDelayTime | The initial delay time in milliseconds | N | 10000 |
maxRetryTimes | The max retry times of the delay, when the backoff reaches the max retry number, the delay time won’t increase anymore | N | 5 |
maxContainerSwitchTime | The max time that message container waiting to switch. Time unit: second. Default: 60 seconds. | N | 60 |
maxReadCacheSize | Amount of memory to use for caching data payload in offload framework. This memory is allocated from JVM direct memory and it’s shared across all the topics running in the same broker. By default, uses 1/8th of available direct memory. | N | Math.max(64MB, (int) (0.125 * maxDirectMemory())) |
readContainerBatchSize | The bytes size of entries one read container hold to read. Default is 5MB. | N | 5242880 |
offloadWriterMaxRetryTimes | Offload writer the max retry times of the delay to retry for retryable write error. | N | 3 |
embeddedStatServiceEnabled | Enable the embedded stat service | N | true |
statServicePort | Stat service default port | N | 8090 |
offloadWriterThreads | The number of thread pool threads for lakehouse writer | N | 2 |
offloadReaderThreads | The number of thread pool threads for Lakehouse reader | N | 2 |
offloadCoreThreads | The number of thread pool threads for the core processor | N | 2 |
offloadHighPriorityTaskThreads | Maximum number of thread pool threads for the core processor high-priority task | N | 2 |
cluster | The tiered storage offloading cluster name, and it will be shown on the Prometheus metrics label. If you run the offloader within the Pulsar broker, you can set it the same with the Pulsar cluster name. Default: pulsar | N | pulsar |
schemaRegistryType | The schema registry type, we support ‘pulsar’,‘kop’,‘fisher’, default is ‘pulsar’. | N | pulsar |
messageFormat | The msg’s format, we support ‘avro’,‘json’,‘csv’,‘kop’, default is ‘avro’. | N | avro |
skipDirtyData | Whether to skip the dirty data, if skip, it will also ack the msg although encode failed. | N | false |
appendPulsarMetaField | Whether to append pulsar meta filed(such as: **ledgerId, **entryId). If disabled, can’t read the msg from tiered storage. | N | true |
partitionKey | The custom partition key in the lakehouse table, support multi partition key split by ’,’. It only works when ‘appendPulsarMetaField’ is disabled. Default value is ‘none’ | N | none |
dataManagedByPulsar | Whether the data lifecycle is managed by Pulsar or not. If setting it to false, all the delete request from Pulsar won’t be processed to the tiered storage | N | true |