> ## Documentation Index > Fetch the complete documentation index at: https://docs.streamnative.io/llms.txt > Use this file to discover all available pages before exploring further. # V4.1.0.0 ## StreamNative Weekly Release Notes v4.1.0.0 #### General Changes ### AoP Add qpid dependencies Add Jiwei Guo as code owner of AoP Add back qpid-test-utils package Use Github package wildcard repository url \[test] fix mock zookeeper change Upgrade artifact github action version Fix SN bom version Fix build script Use SN bom remove useless check in tests Fix the publish latency unit ### MoP Fix MQTT message error handling and improve connection responses Add support MQTT5 features related doc Add Cong to the codeowners Fix subscription authorization PREFIX mode Fix listener error fix topic authentication issue Fix connection event error Fix user properties lost when enable authorization Remove jacoco upload jobs Use Github package wildcard repository url Fix proxy conn multi broker Fix mock zookeeper change Upgrade artifact github action version Fix build script Use SN bom Remove yahoo dependency MoP accept ws proxy connections Add mop proxy admin part Fix mop producer publish metric Make the proxy adapter worker thread configrable Fix the auth data is NPE error Fix broker enable dedup cause client publish failed Seperate proxy and broker a single module Refactor MoP to prepare for split Proxy to seperate module ### KoP Fix Kafka Connect's topic replay loop might be stuck when all messages have been compacted out Pin avro version to 1.12.0 for tests Fix possible deadlock of system topic access due to blocking call when holding the lock Fix incorrect ListOffsets result on a compacted topic Add partition name to error logs in PartitionLog and UrsaPartitionLog classes Fix retention.ms may overflow when converting to the Pulsar retention policy Adapt TopicCompactionService interface changes Auth SN github maven repo before claude review Do not set setReplicationClusters on createTopicIfNotExist Fix consumer close might be stuck when SyncGroup is in progress Fix topic name reference in AlterPartitionReassignments Ignore proxy module in CI Add support for checking schema compatibility against specific versions Remove key distribution verification from PulsarNonBatchedFormatTester Increase timeout for testTwoTopicsGroupState to improve reliability Disable context7 mcp server in Claude github actions Fix message consumption count in BasicEndToEndPulsarTest Add Claude Code GitHub Workflow Support configurable separator for topic Use a scheduler to handle the KafkaClientPool expired key removal Fix broken master due to upstream change Add lookup cache for transaction marker channel manager Don't replay topic for producer state when the topic is empty Fix invalid negative timestamp when fetch offset and timestamp \[Ursa] Support unload topic via rest api Fix flaky UrsaPartitionLogTest.testClosingWithPendingProduce \[test] Remove ignore annotation for compaction test Enhance AdminProxyTest to wait for topic policy updates before alter configs Fix wrong content length when getting schemas containing non-ASCII text Remove kafka tenant and kafka namespace config Add developers and project name Fix Schema Registry returns internal error if the request body is an invalid JSON Remove migration service Support registering the same schema with different subjects for Oxia schema registry Support consume message via rest api Fix consume failure caused by duplicate entry recycling Support a custom managed ledger that extends ManagedLedgerImpl Add namespace filter in dashboard Fix InitProducerId could always fail for the same transactional id Update dispatchable position after update KSN transaction max read position \[Ursa] Support unload all zones when the zone is not specified in the clientId Fix ClientIdBasedLookupTest flaky test Disable automatic group metadata migration in setup to fix flaky test Migrate nexus staging to maven central Improve logs for topic lookup Improve KSN Metrics Dashboard Update to Oxia 0.6.0 and use new group-id Add Swagger Maven plugin for REST API documentation generation Fix get topic by name in KsnRestServlet and update related tests Reduce unnecessary time-consuming topic replay for producer state recovery \[refactor] Create PartitionLog only after ProducerStateManager#recover is done Change the default value of the kopAllowedNamespaces Support topic retention policy configuration Update build script to copy kafka-rest-servlet as a jar instead of nar Remove unnecessary dependencies for rest api Bump Pulsar to 4.1.0-SNAPSHOT Add kafka-rest-servlet to asset directory in build script Support produce message via rest api Unified metrics topic scope format Exclude netty-codec from test Support schema registry RBAC verification at cluster level Fix UrsaPartitionLog not taking snapshot during shutdown \[schema-registry] Fix get schema by subject and id behavior Fix test compatibility request \[Ursa] Reduce unnecessary time-consuming topic replay for producer state recovery Support Scram with JWT token Sasl Return timestamp for the ListOffset request Fixed NPE when removing partition log \[Ursa] Skip producer state recovery for consumers and non-idempotent producers Add kafka-common jar to build script assets Avoid replaying topics when handling ListOffsets requests Fix InitProducerID request transaction operation Support configuring default compatibility level Change log level from error to info for successful authorization in SimpleAclAuthorizer \[schema-registry] Use the dot for all subject related requests Add KSN rest admin servlet Fix flaky testProducerStateRecovery \[Ursa] Add per-topic partition storage size metrics for ursa Filter system topic when list kafka topic Remove the log when the EventManager receives events after shutdown Use try load group and offsets to avoid print scheduling loading log Improve error logging for topic deletion failures Remove unused jaxb-api dependency from pom.xml Add namespace label to metrics Fix ServiceConfiguration cast to KafkaServiceConfiguration Fix pulsar-kafka-schema-registry compilation failure: Compilation failure Schedule load group metadata on owner broker \[Ursa] Fix namespace bundle for topic not served by this instance \[Ursa] Support unload ursa partition log Support KSN RBAC part3 Support handle DESCRIBE\_LOG\_DIRS request Bump Pulsar to 4.1.0-SNAPSHOT \[Ursa] Support idempotent producer Remove error logs if a response completes with a known Kafka exception Support KSN RBAC Part2 Add a separated config to specify number of worker threads \[ursa] Fix messages might be skipped for a consumer that subscribes multiple partitions Fix NPE when Accept Header is empty in SchemaRegistry request \[proxy] Fix memory leak for requests that are not forwarded to brokers In the `pulsar_non_batched` entry format a single batched message is encoded as a non-batched message Only take snapshot for producer state except when the topic is unloaded Exclude lz4-java dependency due to CVE Don't block requests for unknown request field Fix pulsar\_non\_batched out of order sequence number error Fix lookup reference schema Fix DeadLock issue when resolving the reference schema SNIP-143: Fallback to base64 encoding only when the byte array is not a valid UTF-8 string Fix OxiaSchemaStorage use the wrong range scan start key Add new PulsarNonBatchedEntryFormatter class to encode batched messages to non-batched messages Support content negotiation for schema registry HTTP service \[ursa] Support configuring consistent hashing virtual nodes Fix dead group might not be removed when storing group metadata in metadata store Retry loading a partition of transaction metadata if it failed to load last time Disable geo-replication for system topics Fix memory leak due to temporary buffers are not released in decode \[ursa] Update default load balance strategy to CONSISTENT\_HASHING Remove nexus staging repository Fix connections will always be disconnected when RbacAuthorizationProvider is not configured Use Github package wildcard repository url Remove the override updateRates method since it's removed from PersistentMessageExpiryMonitor Change the ConsumerGroupMetricsCollector error log to warn Support RBAC authorization validation - Part1 Don't print error logs when creating a topic that already exists Don't print error logs when receiving GET\_TELEMETRY\_SUBSCRIPTION requests \[CI] Run smoke-tests on CI Do not modify the Entry's data when decode Cherry pick PRs from branch-4.0 to master branch Avoid blocking I/O threads when lookup due to unstable DNS Refactor the Ursa partition log cache to make fields immutable as much as possible Support independent schema registry service \[test] Update SimpleLoadBalanceTest's load manager config to use `ServiceUnitStateMetadataStoreTableViewImpl` \[ursa] Re-balance the topic when broker up Fix UrsaStorageTest.testListOffsetsFromClosedManagedLedger Remove the mixed\_kafka format and simplify ProducerAppendInfo Fix failed to generate schemaId/version if the schema-id-gen/version-gen nodes are empty Fix testIdempotentProduce will fail when running individually Support wildcard characters for kopAllowedNamespaces \[test] fix mock zookeeper change \[ursa] Move the consume logic to use managed cursor directly Support import mode for oxia schema registry Fix get schema to return the correct Protobuf type instead of Avro for Protobuf schemas Fix DescribeConfigs compatibility with Sarama 1.42.1 or earlier Add a simple consistent hashing based load balancer implementation Remove the complicated multi-tenant metadata feature and enable transaction by default Change LEO outdated log to Debug log Catch the `TooLongFrameException` as warn log Invalidate UrsaPartitionLog from cache if it's failed Use slf4j-log4j2-impl as the logger in tests Fix the ConsumerGroupMetricsCollector admin client url Revert "Revert using sn-bom (#957)" Add a simple partition index based load balancer implementation Update Pulsar stats for Ursa Upgrade the kafka-clients dependency to 3.9.0 for new requests \[ursa] Fix shadow managed ledgers' metadata nodes not deleted Revert using sn-bom Fix group metadata inconsistency after loading from the metadata store Update pulsar consumer stats from ksn Fix flaky OffsetTopicWriteTimeoutTest.testSyncGroup Fix possible thread safety issue when accessing GroupMetadata Update the default offset retention time from 3 days to 7 days Handle schema registry authorization compatibility issue Fix system topics not filtered for billing metrics Add pulsar-kafka-schema-registry jar to image Support schema registry RBAC Use SN bom Delete the corresponding shadow topics when the source topic is deleted Add metric for consumer lag Fix possible deadlock cause by NamespaceBundleFactory.getBundle Close the topic during ListOffsets if ML is already closed Remove the unnecessary asyncMarkDelete call after reading from the cursor Add more logs for empty assignment Update clientId format to allow including both zone id and additional information from users Update the shadow namespace name format \[Ursa] Don't fail with OFFSET\_OUT\_OF\_RANGE when LEO is less than the fetch offset Unload topic when fail to create cursor Retain only the value associated with the valid topic name key when fetching offsets Fix data\_in\_bytes is always 0 when Ursa is enabled Fix managed ledger config not respected for Ursa producers Fix Ursa might not be able to produce after the topic is unloaded Handle Kafka multi-tenant format topic name in schema registry Remove producer state snapshot when topic deleted Filter the pulsar.dedup cursor z-node on the topic deletion event handler Fix memory leak for producers when Ursa is enabled Ignore all compaction tests when entryFormat is pulsar Disable bundle ownership transferring for bundles in shadow namespaces Remove useless log for the PulsarSchemaStorage Fix UrsaPartitionLog initialization Introduce DelayedRemovalCache to cleanup ManagedLedger when Ursa is enabled Add metrics for Ursa topic initialization Fix Kafka headers are not converted correctly when entryFormat is pulsar Return a user-friendly error when the Fetch requests are not supported or topic is blank Add message duplication tests when Ursa is disabled Support billing metrics for SN cloud Support handling produce requests from non-owner brokers \[fix] Filter duplicated topic when fetch offsets Adopt a more efficient and reliable approach for compacted topic replay Enable geo replication test Support librdkafka's PARTITION\_EOF feature Re-implement the EventManager Avoid creating transaction related systems topics when enabling ursa engine Remove all coordinator epoch usages Change the default entryFormat to kafka Refactor the transaction implementation to make it align with Kafka Fix response status code for Schema Registry ### StreamNative Pulsar Plugins upgrade avro to 1.12.0 fix export duplicated JVM metrics on AuditLogMetrics Upgrade pulsar placement policy bk dependency to 4.17.2 Upgrade zk version to 3.9.3 to avoid CVE-2024-51504 Bump com.fasterxml.jackson.core:jackson-core from 2.13.0 to 2.15.0 in sn-pulsar-tool Fix the build issue \[detector] Print more info when a corrupted value is received by Kafka consumer Return empty when getting global topic policies instead of returning exception Upgrade commons-beanutils version to fix CVE-2025-48734 Add test to verify the sts module \[detector] Enable idempotence for Kafka detector Fix the backup tool can not use sts to authenticate \[cluster-rollout] add orphan unload job cleanup logic Fix the time ticker leak issue which caused high CPU usage Update Go SDK rbac & oidc dependencies Update x/net and go 1.24 Fix jlink command's compress argument fix(detector): enable pprof by default feat(detector): support disable pulsar protocol detection. \[graceful-rollout] set unload retry max make integration test for all release branch Add environment to e2e pipeline add cloud package integration tests Use Github package wildcard repository url ursa-storage Fix rest api due to producer min compress size 4a25f8bca Update Go toolchain to 1.22.12 Upgrade deps to fix CVEs Fix go x/net cve \[fix]\[cluster-rollout] handle namespace not found greacefully \[test] Fix mock zookeeper change Add cert expiration detector feat(detector): support pprof fix: discard superuser flag to avoid deadlock fix(detector): compatible with non-partitioned aliveness topic Change error log to warn when topic closed fix initialize for AuthenticationProviderMTls feat(IdentityPool): Add basic authentication support for identitypool Add AuthenticationProviderMTls for snoidc Fix deploy workflow Fix deploy package workflow Add plugin module deploy workflow fix azure package failed when list non-exists directory Support Topic Level Tracing and LogTraceRecorder Azure Blob Storage backed Package Management Service Exclude commons-io to fix cve Fix license commons plugin jar package Use WebIdentity way when AWS\_WEB\_IDENTITY\_TOKEN\_FILE exists fix test in BrokerUnloadJobResourcesTest Just add non fat jar to image Use SN bom Add pulsar-oxia-state-store jar package Change authentication failed log level to warn Upgrade aws sdk dependency version to v2 exlude netty for aws-jdk Reduce pulsar-rollout-plugin nar package size ### pulsarctl Upgrade go to 1.24.6 to fix CVE-2025-47907 ### Cloud Pulsar Plugins Change to use commons-lang3 ### Function Mesh Worker Service b8175ef4 fix build.sh Create a new sub module mesh-worker-common Generate OpenAPI docs for agent-functions Support set agent tools config Make MeshWorker able to run standalone and load additional servlets Support load ConnectorCatalog using label Update error msg in status Implement agent function Set minReplicas to parallelism when HPA is enabled Set default VPA by default when HPA is not enabled Support invalid name Fix resource error during update and get connectors Fix trigger function not support partitioned input topics error Find specified ServiceAccount using oauth2's client role and use it when exist validate function-mesh v0.24.1 exclude lz4-java for CVE reasons Unified resource scale for all Objects and make it configurable add security schemas to kafka connect openapi better error responses Use Github package wildcard repository url bump function-mesh to 0.24.0 masking the sensitive data in logs support insecure auth secret override Use ConnectRestException for Kafka Connectors Add default liveness probe: Validate k8s secret before creating functions/connectors inferring the tenant and namespace from request if empty Fix update error for functions/sinks/sources with secrets injected Support set log topic from configs for Sinks\&Sources Support set resources for kafka connect Use SN bom SNAPSHOT version Do not allow using system topics when creating sink\&source Reject request when kafka connect's name is longer than 29 Use SN bom to reduce artifact size Use large runner to avoid disk full issue Fix free disk job of ubuntu-latest runner bump kafka dep to 3.9.0 Replace the deprecated `getZooKeeperSessionTimeoutMillis` Add `extraDependency` field to FunctionMeshConnectorDefinition Support set pod annotations via CustomRuntimeOptions Set processingGuarantee for window functions Bump function-mesh to v0.23.0 ### StreamNative Tiered storage Introduce flag to control delta add file stats ### StreamNative Ursa storage Remove failed to deserilize log Add more metrics Run with different commit runner if task properties changed Expose parquet file reader cache config Fix NaN serialization issue in UrsaParquetFileWriter Fix nested enum serialization failed fix: Handle NPE when nested record default value has mismatched field names Refactor the commit process to allow recreate commit runner Pulsar worker support offload to delta Remove tmate in CI USe Hessian2 as the new task serialization. Fix delete package task bug Fix topic medata not found issue. Fix: Use shared static thread pool for PulsarLakehouseReader idle timeout Upgrade iceberg to 1.9.2 Unblock the compact process when encounter task deserialize exception Adapt new changes for TopicCompactionService interface Refresh the catalog instance when using open catalog Make update and delete compactTask async Uniform all places configuration name for the data source type Update the offload flag according to the each ledger state Rename engineType to dataSourceForCompaction Fix publish time Fix committed task delete leak Add engine type configuration Refactor the offload format by adding a util to convert the entry to KafkaMessage Fix deadlock by making getStreamId async in OffloadReadHandler Fix task compability issue Fix resource leak: close IndexFileReader in ParquetFileReader Only publish BlobNotFound exception task to DLQ Exclude OutOfMemoryError for DLQ Support publish commit failed tasks to DLQ Fix oxia lock leak when not acquired Optimize Compaction Service heap memory usage fix\[lock]: fixes memory leak on oxia distributed lock Introduce catalog factory for iceberg Add external table protobuf support for Ursa and Pulsar protocol Make the pulsar compaction worker not record column stats. Using bookkeeperStorageApi when configured pulsar client Add compaction leader metric doc Delete committed tasks and update oxia index Fix the error handling Remove unnecessary synchronized lock fix: handle UUID logical type with string base type in AvroToIcebergConverter Fix ConcurrentModificationException in CompactionTaskProvider.getTask Cleanup stream when deleting unloaded topics Add multi catalog user document Add compaction service throguhput rate limiter for reading from BookKeeper Support multiple catalog in namespace and topic level Optimize reset cursor Refactor update iceberg table properties Separate managed and external writer for ursa Add compaction leader metric Add failure reason for the iceberg external writer Fix CI Upgrade delta kernel to 4.0.0 Make the RawReader object pooled to avoid creating each time Delete the compact task if the topic doesn't exist \[refactor] Only open one parquet file for the lakehouse reader Using the GlobalOpenTelemetry to register the reader metrics Fix reader read failed in parallel Delta external table introduce temporary credential Fix the task compatibility issue. Introduce CustomKernelParquetWriter to support put write mode to improve memory usage Fix readIndexes bug Fix the typo for the classname of PreparedCompactStreamTask Introduce the task type to control the compaction handling Add offload cursor to block data expire Send the failed parse messages into a failure topic Fix the metrics tests Load configuration from the pulsar-client.conf file Close the catalog resrouce after using Delete the compaction task if it compacting the expired data Add ML Cloud Storage Developer Guide Reenable the pulsarE2ETest Speed up the CI process from 1 hour to 20mins by separating to the different groups Adapt for unity iceberg rest api Fix the offloaded ledgermetadata is not synced with the offload state chore: Add Claude Code Agents Allow to disable sync UrsaMLMetadata Support delete ledger from pulsar offload handler Add metrics for pulsar read/writer lakehouse path Redirect maven twitter repo to central Fix build script Refactor pulsar external table Support load pulsar client token from file Use the existing resources to init lakehouse reader Renable the pulsar e2e tests Get ledger metadata from Oxia Change default entrySerDeType to PULSAR\_BATCHED\_RAW\_PARQUET support register managedledger meatadata in oxia Pulsar support reading from parquet store messageId into parquet file Store ledger metadata to oxia Use jar instead of nar for offloader Add serialization type in the metadata Revert to use normal file as the index file Support save pulsar entry without parsing batch Support skip system topic and black topics Fix the bookkeeperApi can not get the index by secondary key Using MapFile to speed up the seek performance Support deleting the compacted data Update the offload flag according to the ursa storage state Make pulsar compaction worker enable iceberg external table writer Reuse the pulsar storage configuration for the pulsar offloader Update metadata store ledgers info after checking offloaded flag. Fix the prepare task name compatibility issue \[cleanup] remove unused code in the managedLedgerWithTs Support generate the Ursa offset when committing the task Update to Oxia 0.6.0 and use new group-id Make bookkeeperStorageApi implement the StorageApi Pulsar offloader leader support Support write without parse content with schema Support compress the index files Make the tasks in the CommitTaskProvider sorted by the stream id and start offset Pulsar offloader support offload to iceberg format Support read/write index file with hadoop lib to cloud storage Fix pulsar lakehouse reader memory leak issue Integration test for the pulsar protocol compaction process Refactored to allow support for low-latency storage class Using the 4.1.0-SNAPSHOT of pulsar Adapt the comapction process with pulsar related code Fix the NPE when serializing the bytes schema version Use static instances for compression codecs Pin version for commons-lang3 Support bookkeeper entry reader to let compaction service read from bookkeeper Support read and write bookkeeper entries Introduce the api for bookkeeper generate offsets Support trimming topic with the mark-deleted-offset Fail back to normal config when the external config miss. Optimize entry reader read batch Allow to configure the http client used by Azure Fix build script typo Support pulsar entry write into / read from the parquet file process \[doc] WAL Cloud Storage Developer Guide Update Metrics.md with accurate and concise descriptions Fix topic quarantine bug Implement new interface for Pulsar Add default timeout for all integration tests External table support wal compact Disable expire Iceberg snapshot by default Support fencing a managed ledger after closing Use nonRetriableQuaratine for task publish Improve the compaction worker handle stream task logic. Flaky test in SimpleStorageImplTest.java \[improve] Use ReentrantReadWriteLock in EntryCache Move out the integration test containers to a new module Reduce primitive schema retry times Format compaction service quarantine logic Support configure iceberg table properties with topic properties Throw exception when read empty entries in lakehouse worker. \[fix] throw exception upon EntryCache apis calls after EntryCache is closed Move the persistStorageApi initialize in the common place Fix real offset incorrect Fix semaphore not release bug Fix read lock not release bug Fix build failure GCS supports delete with lifecycle Throw exceptions when handle the recursive schema throw exception when hitting non-binary index while processing RAW type Support azure to delete object using lifecycle rules Delete the compact task when compaction worker read compacted wal file. Quarantine topic when get topic failed or get topic metadata failed fix deadlock from EntryIndexCache.invalidate Skip topics in pulsar tenant in compaction service Fix compaction service generates a lot of small parquet files Add Iceberg bigquery metastore catalog support Fix publish thread executor block issue. Skip some tests to speed up integration test Fix s3Table name format Remove sn-bom build in CI \[fix] write api support for v2 format Make the entry read instance in compaction not shared betweet the different task Fix the IllegalReferenceCountException by duplicated method execute Fix npe when check lakehouse commit table failed. Do ursa GCS performance and improvement Fix GCS deadlock issue Fix priorityqueue concurrentModification exception Recreate LakehouseStreamCompactWorker when encountered S3Exception Add more metrics for compaction service Use Github package wildcard repository url Support write entry with specified initial offset Support streamnative Delta External table. Add metrics for the read cache size in bytes Fix the aws config socket time not work issue. Add more netty config Fix the NPE when getting the non-exists blob Remove keys from lock when unlock \[improve] Ursa Live Entry Index Compaction Improve parquet read prefetch Use SN bom based image Fix taskWriter shared bug Fix avro convert to iceberg npe Format lakehouse exception Fix the memory leak issue in the test. Fix memory leak when write parquet failed Fix the record and schema convert issue for avro to iceberg Fix LakehouseKafkaReader NPE optimize the jvm opts for compact service. Remove serailize failed noise log tmp Do not catch handleRecord exception Fix dockerfile Remove the test jar from the image Revert #615 to avoid the caffine cache performance issue. Enable v3 serialization and deserialization Make compaction service compatible with Confluent schema registry Fix maxPendingAddRequestsUsedBytes integer overflow Fix the map value struct type issue in unity catalog. Fix avro record convert to Iceberg Record failed Fix retry the compact stream task failed issue. Fix parquet row writer issue Introduce delta kernel to replace of delta-standalone. Fix unity catalog schema issue \[Fix] Fix the iceberg namespace on s3 table catalog Limit the readcache by bytes Limit the pending add by bytes Support nested protobuf schema for the compaction Move the create table to the writer instead committer Add test case to cover msg payload schema content Remove copied class Remove the ursa-lakehouse dependency from the ursa-ml Add iceberg external table support Support kafka protobuf schema for the compaction Fix BrokerInterceptorTest.testAddBrokerEntryMetadataEntryRef flaky test. Add test unit to cover get wrong streamId issue. Fix CI Fix the json schema convert issue. Fix always use the wrong streamId when publish stream task. Set partitionkey for the rangeScan in the persistStorageApi Add the persistCache size into the meta Improve PersistCache with a simpler and thread-safe implementation Add EntryIndex protobuf feat: move the info log to debug to prevent server keep logging \[cleanup] Remove unused api Refactor compaction worker interface Replace guava cache with caffeine cache \[cleanup] Move the read cache into a single class Clean up the unused code of ursa-storage-core Remove the unused code Unity Catalog support clientId and clientSecret to authenticate. Use ubuntu-latest to run ci Fix Github Packages publish workflow Remove nexus snapshot distribution Use `____` to replace '-' for table name when create unity catalog table. Support publishing to GitHub Packages Fix ManagedCursorTest#testReadEntriesWithSkipDeletedEntries flaky test. Fix StorageWalManagedCursor#internalReadEntries only read one entry at each round. Fix s3 compact flaky test. \[TEST] Run integration test with image ksn 3.9.0 Fix parquet prefetch bug Add iceberg azure dependency Use separate thread pool for parquet reading Test master CI Fix default compaction thread bug Support load credential from file Fix flaky test TestCompactionServiceBaseFileStorage#simple fixed flaky test\_whenReadAgaint add cloud region 9d6fd76f \[Bug] fix duplicated entry id put in PersistCache.index Adjust the compaction default configuration according to the performance test Fix flaky test in TestGarbageCollection Add the ProfileCredentialsProvider into the auth chain Disable nonblocking dns and get rid of request limitation of S3 prefix improvement in dispatch logic Fix the default write buffer segment configuration Unity catalog support config User-agent. Adding stress-ng to CI Fix oxia read failed with small range Improve CI by running S3 and GCS integration tests In parallel Optimize PersistCache serialization to only persist used segments Support disable read cache expire by time Compaction service support GCS Delta table support partition column using topic partition index Compact service support azure blob protocol Parse the storage account name and blob container name from bucket for azure Check the table whether register to the unity catalog when commit action. If not, register the table to unity catalog. Compaction service support Azure storage. Modify delta table mapping name in unity catalog Rename azure to azureblob to make it more precise fix azure make data in heap Quarantine the compact task if it read WAL data fialed. Add the storage metrics for all the file storage type Allow to disable the lakehouse reader in the managedledger Enable ChecksumCRC32C for getting object from S3 Add core and lakehouse jars to image Support unity catalog. Add iceberg support Remove jar with dependencies when release Switch to sn-bom pom dependency \[cleanup] Remove the duplicated code Limit the read request from compaction to the storage api Get the compact task in the start offset order Separate the integration tests and ut in workflows Enable all the primitive types tests Grouping pending add requests by stream id Add read request limitation for the persistStorageApi read Guarantee managed ledger's stream id is always valid and never modified Add Yunze and Zike as code owner Avoid concurrently update metadata for ShadowManagedLedger Provided storage endpoint for azure storage \[WIP] Add primitive type support Fix the null filed value can't decode issue. Introduce lakehouse read prefetch cache manager Speed up get all task Separte the maxRequest config and write buffer segment config Fix the commit runner race condition issue. Update streamId if Shadow Managed Ledger stream ID is invalid Trigger the metadata update when calling getLastConfirmed entry \[improve]\[tests] covered Pulsar protocol retention and backlog quota \[improve]\[tests] covers Pulsar protocol encryption and compression tests Remove awssdk bundle dependency Avoiding the risk of pending add buffer release fix netty and json cve Fix the stream id is duplicated Do not reject the entries bigger than write buffer Support the azure file storage \[improve]\[tests] enabled ExtensibleLoadBalancer in integ tests and added delayed messaging test \[improve]\[tests] enabled concurrent tests in integ and enabled new load balancer Support creating the managedLedger in different brokers Add GCS FileStorage implementation Introduce task manager to avoid acquire lock between threads (Re-Commit) LakehouseKafkaReader add prefetch cache support \[fix]\[ml] fixed npe in getLastIndividualDeletedRange Improve the WAL catch up read throughput by prefetching cache \[improve]\[tests] cover Pulsar messaging tests \[fix]\[tests] Enable ursa-storage-test tests Fix trivy downloading db rate limitation Make mockito test scope ([#401)](https://github.com/streamnative/ursa-storage/pull/401))) Revert "Introduce compaction task provider to avoid acquire lock between multiple threads Fix the integer key leak issue Introduce compaction task provider to avoid acquire lock between multiple threads Fix aws credential not match issue Reduce package size Upgrade aws sdk version to fix cve Fix CVE-2024-7254 Trigger publish task by self, not wait all the topics. Add error log when put entry failed in the managedLedgder Update the Docker resource and readme to the latest stat Add json support check in publish task Bump org.scala-lang:scala-library from 2.13.0 to 2.13.9 CompactService support json schema Optimize quarantine logs in publish tasks Bump project version and pulsar version to 4.1.0-SNAPSHOT \[metrics] Add metrics for the write buffer Use ConcurrentHashMap to reduce risks \[cleanup] move the write cache out of the storage impl Bump org.apache.hadoop:hadoop-common from 3.3.6 to 3.4.0 Support write entry for the StorageApi Filter topics with not support schema Optimize logs in compaction service Refactor compaction metrics Optimize compaction service default configurations Make the stream map evict by time. \[cleanup]Remove the unused code in the persistStorageApi Fix the flaky test testOperationRejection Fix wrong metadata updates for first and last entry headers Increase commit runner concurrency Fix som cve feat(distributed-lock): use metadata oxia client as distributed lock client Use concurrent map for the cache strings in storageApi \[cleanup] Move the metrics part out of the implementation Make the maxPendingAddRequest dynamic Correct the dockerfile used file location Make the sync method call the async method to get result Fix the request rejection of SimpleStorageImpl Fix the missed exception handling for getting the next read index Fix prepared publish task may failed issue. Fix the ledger deletion for PersistCache Parquet read perfomance improvement Delay metadata update task for managedLedger Introduce compact cordinator. Fix flaky test `testReadAfterTrimmed` Decrease s3OpsRateLimitPerSecond to 100 Enable the CI on all branches Fix delete topic failed when not load Fix compaction otel paramter