V4.0.5.3
StreamNative Weekly Release Notes v4.0.5.3
Download
Distributions
Packages
Images
General Changes
Apache Pulsar
(#24443) [fix][txn] Fix negative unacknowledged messages in transactions by ensuring that the batch size is added into CommandAck
(#24421) [fix][build] Add missing name to submodules
(#24459) [improve][broker] change to warn log level for ack validation error
(#24441) [fix][ml] Enhance OpFindNewest to support skip non-recoverable data
(#24454) [improve][broker][branch-4.0] Update to Oxia 0.6.0 and use new group-id
(#24448) [refactor][broker] Expose the managedLedger field for the sub class
KoP
Add lookup cache for transaction marker channel manager
StreamNative Pulsar Plugins
Fix the build issue
Upgrade zk version to 3.9.3 to avoid CVE-2024-51504
Bump com.fasterxml.jackson.core:jackson-core from 2.13.0 to 2.15.0 in sn-pulsar-tool
pulsarctl
(#1801) Enhance GetStatsCmd to include additional options for backlog statistics
StreamNative Tiered storage
[Branch-4.0] Cherry pick #1051 #1061
StreamNative Ursa storage
aed7cdec Use maven 3.9.9 settings
24cd9b7e Change command
036a9127 Change command
a1f57661 change interactiveMode mode
e6117b73 change interactiveMode mode
09d4c7c7 change interactiveMode mode
0680adc3 Add local maven settings to skip twitter downloading
b023c907 Add local maven settings to skip twitter downloading
Fix build script
Refactor pulsar external table
Support load pulsar client token from file
Use the existing resources to init lakehouse reader
Renable the pulsar e2e tests
Get ledger metadata from Oxia
31275451 fix cherry-pick issue
Change default entrySerDeType to PULSAR_BATCHED_RAW_PARQUET
support register managedledger meatadata in oxia
Pulsar support reading from parquet
store messageId into parquet file
Store ledger metadata to oxia
Use jar instead of nar for offloader
Add serialization type in the metadata
Revert to use normal file as the index file
Support save pulsar entry without parsing batch
Support skip system topic and black topics
Fix the bookkeeperApi can not get the index by secondary key
Using MapFile to speed up the seek performance
Support deleting the compacted data
Update the offload flag according to the ursa storage state
Make pulsar compaction worker enable iceberg external table writer
Reuse the pulsar storage configuration for the pulsar offloader
Update metadata store ledgers info after checking offloaded flag.
Fix the prepare task name compatibility issue
[cleanup] remove unused code in the managedLedgerWithTs
Support generate the Ursa offset when committing the task
Update to Oxia 0.6.0 and use new group-id
Make bookkeeperStorageApi implement the StorageApi
Pulsar offloader leader support
Support write without parse content with schema
Support compress the index files
Make the tasks in the CommitTaskProvider sorted by the stream id and start offset
Pulsar offloader support offload to iceberg format
Support read/write index file with hadoop lib to cloud storage
Fix pulsar lakehouse reader memory leak issue
Integration test for the pulsar protocol compaction process
Refactored to allow support for low-latency storage class
Adapt the comapction process with pulsar related code
Fix the NPE when serializing the bytes schema version
Use static instances for compression codecs
Pin version for commons-lang3
Support bookkeeper entry reader to let compaction service read from bookkeeper
Support read and write bookkeeper entries
Introduce the api for bookkeeper generate offsets
Support trimming topic with the mark-deleted-offset