Skip to main content

StreamNative Weekly Release Notes v5.0.0.2

Download

Distributions

Packages

Images

General Changes

Apache Pulsar

(#27117) [cleanup][test] Fix deprecation warnings in tests (#27115) [cleanup][test] Replace the deprecated URL(String) constructor in tests (#27114) [cleanup][build] Put swagger annotations on test compile classpaths to fix RequiredMode javac warnings (#27116) [cleanup][test] Replace Netty’s deprecated NioEventLoopGroup and DefaultEventLoopGroup in tests (#27118) [cleanup][test] Fix unchecked warnings in tests (#27090) [fix][test] Fix Mockito stubbing race in NamespaceServiceSplitFailureTest (#27086) [fix][client] Keep non-serialized builder settings across loadConf (#26257) [cleanup][misc] Fix javac deprecation and unchecked warnings in main sources (#27097) [fix][client] Preserve pooled heap message data across repeated reads (#27092) [improve][test] Reduce SchemaServiceTest wall time from 152.5s to 75.0s (#27095) [fix][build] Upgrade RoaringBitmap to 1.6.25 to fix LongBitmap range contains and previousAbsentValue (#27068) [fix][client] Fix double-counted permits when the receiver queue grows during reconnect (#20453) [improve][pip] PIP-273: Enable hostname verification by default (#16647) [fix][ml] Discard a ledger created after its creation was given up on (#25189) [improve][broker] Add http produce backlog quota check (#27078) [fix][client] Fail transaction coordinator requests after close instead of throwing NullPointerException (#27081) [fix][client] Ignore superseded subscribe and producer attempts (#27083) [fix][ws] Release WebSocket consumer pending slots only for pending message ids (#27087) [fix][client] Flush the pending acks before the zero-queue permits on reconnect (#22323) [improve][fn] Support OAuth2 in Go instance (#22978) [improve][txn] Allow transaction owners to abort their own transactions via the abortTransaction admin API (#27073) [fix][client] Fail pending transaction coordinator requests when TransactionMetaStoreHandler closes (#27082) [fix][txn] Keep plain publishes behind the transactional ones TopicTransactionBuffer holds back (#15363) [fix][broker] Retry the dedup cursor delete after a failure while disabling at topic load (#13280) [fix][client] Fix DLQ sending a released message when poolMessages is enabled (#16502) [fix][broker] Start packages management before the function worker service (#27084) [fix][cli] Make pulsar-perf consume and transaction end and await every transaction they open (#20737) [fix][broker] Close the existing consumer when the same consumer subscribes again (#27079) [fix][client] Keep non-transactional messages out of transactional batches on scalable topics (#27080) [fix][txn] Keep plain publishes to scalable-topic segments behind pending transactional ones (#16133) [Functions] fix upload nar file to start function task failed (#19260) [fix][broker] Keep the bundle affinity broker for every lookup until the bundle is owned (#18937) [fix][admin] Read the error response body once when extracting the server’s reason (#20246) [fix][broker] Don’t share the dispatcher’s FilterContext with analyzeBacklog (#19883) [improve][admin] Include schema ledger ids in topic internal-info (#16325) [fix][broker] Fall back to the broker isolation groups after a namespace bookie affinity group is removed (#16563) [fix][broker] Don’t count the target namespace in its own anti-affinity group (#27077) [fix][cli] Make pulsar-perf produce end and await every transaction it opens (#20032) [fix][client] Fix consumer discarding all messages after redelivering during a reconnection (#27066) [fix][client] Fix pattern consumers that stop updating topics when subscribing to a new topic fails (#27067) [fix][client] Fix stale batch ACKs causing skipped messages after seek (#27062) [fix][client] Flush the pending acks before the flow command when the consumer reconnects (#26087) [improve][admin] Validate dynamic configuration updates against FieldContext constraints (#15144) [fix] [client] NotifyPendingReceivedCallback when zeroConsumerImpl received batch message firstly then close consumer (#27070) [fix][test] Fix flaky V5CumulativeAckTest.testRetryingAnAckThatFailedWhileDisconnectedAdvancesTheCursor (#20053) [fix][txn] Treat a concurrent duplicate transaction status update as success (#27058) [improve][misc] PIP-351: Add options to Pulsar-Test client to support KeyStore based TLS (#17946) [fix][broker] Copy subscriptions to the partitions created by createMissedPartitions (#19679) [improve][fn] PIP-237: Expose PulsarAdmin in SinkContext and SourceContext (#19867) [improve][client] Log the stack trace when MessageCryptoBc fails to load a key or encrypt/decrypt a data key (#19352) [fix][ml] Fix off-by-one backlog of cursors created while an entry is being added (#27061) [fix][client] Stop the partition update timer of a partitioned producer that ended during creation (#21013) [fix][fn] Use TLS urls for the embedded function worker when brokerClientTlsEnabled is set (#27052) [fix][meta] Reload the bookie info on a cache miss in PulsarRegistrationClient (#19602) [fix][client] Make ackTimeoutRedeliveryBackoff work on Exclusive and Failover subscriptions (#27060) [fix][ml] Fix the estimated backlog of lazily recovered cursors (#27055) [fix][proxy] Copy GetTopicsOfNamespace properties before the broker connection completes (#26969) [improve][broker] Index topic watcher notifications by namespace (#27024) [fix][client] Close the producer when its encryption key cannot be loaded (#20824) [fix][client] Read the OAuth2 private key without requiring a content type (#22577) [fix][broker] Reject namespace-level partitioned topic creation on a remote partition conflict (#20814) [fix][ml] Don’t trim ledgers while cursors are lazily recovering (#22695) [fix][broker] Don’t swap the local metadata store’s synchronizer when configurationMetadataSyncEventTopic is updated on a shared store (#21928) [fix][client] Queue acks without grouping while the consumer is reconnecting (#20929) [improve][conf] Add BookKeeper DirectIO entry logger settings to bookkeeper.conf (#22166) [fix][txn] Fix PendingAckHandleImpl potential thread safety issue (#22516) [fix][fn] Send Go function logs to the log topic periodically and on shutdown (#22021) [fix][broker] Fix ‘rewinded’ typos in PersistentDispatcherSingleActiveConsumer logs (#27054) [fix][broker] Serialize outgoing commands before passing them to broker interceptors (#27057) [fix][txn] Fix reuse of recycled pooled objects in transaction code paths (#27056) [fix][ml] Run the add-entry timeout check on the managed ledger executor (#27053) [fix][broker] Reply to failed GetLastMessageId with the original request id (#23495) [fix][broker] Don’t redeliver individually acknowledged messages from the compacted ledger (#23353) [fix][broker] Don’t count unread fixed-delay entries in msgBacklogNoDelayed (#22803) [fix][ml] Move the read position atomically when a replay is at it (#26607) [fix][proxy] Answer failed partition metadata and lookup requests with their own response type (#23488) [fix][broker] Don’t allow creating a non-persistent partitioned topic with ‘-partition-’ in name (#23830) [fix][client] Client interceptors close method not called after producer/consumer/reader close (#24672) [improve][conf] Document managedLedgerForceRecovery and schemaLedgerForceRecovery in broker.conf and standalone.conf (#27050) [fix][ml] Materialize compressed ManagedCursorInfo before releasing the decompressed buffer (#23470) [fix][ml] Fix races in managed ledger initialization timeout handling (#26401) [fix][broker] Manage in-flight bucket work during clear (#20540) [fix][broker] Don’t continue loading a topic after its load has timed out (#24783) [fix][fn] Only load crypto_key_reader for python function when cryptoKeyReaderConfig is provided (#24342) [fix][test] Fix flaky NegativeAcksTest.testNegativeAcksWithBackoff (#26458) [feat][fn] Expose the Prometheus metrics registry to Go functions via FunctionContext (#27037) [fix][broker] Reject replicated status for the compaction subscription (#27041) [improve][test] Add deduplication contract coverage across broker restart (#27039) [improve][ml] Read ahead the next range of a cursor’s read into the entry cache (#26911) [fix][broker] Honor secondary tenant admin roles in HTTP admin requests (#27003) [fix][broker] Fix non-persistent subscribe cleanup during migration (#26631) [fix][broker] Fail compacted reads when the compacted ledger is missing (#26553) [fix][managed-ledger] Honor batch-index cap in isCursorDataFullyPersistable (#26490) [fix][fn] Honour consumerCryptoFailureAction in the Python function runtime (#26305) [fix][ml] Don’t drop a durable cursor reset when an ack is queued behind it (#26277) [improve][broker] improve readability for the class TopicPolicyListenerWrapper (#26953) [fix][broker] Keep the topic-level permissions of a topic migrated to a scalable topic (#26396) [fix][broker] Shared subscription stalls forever on a chunked message (#26542) [improve][broker] Add a random value upto 5% to avoid rollover multiple cursor ledgers (#27040) [fix][test] Fix flaky OffloadPrefixTest.terminateTriggersAutomaticOffloadForLastLedger (#27032) [fix][broker] Ignore replicated subscription updates for the compaction subscription (#27035) [fix][build] Include the java.compiler module in the Pulsar image JVM (#27028) [improve][test] Add IoT telemetry scenarios with partitioned topics to the performance tests (#27009) [improve][broker] Upgrade bookkeeper to 4.18.2 (#27026) [fix][broker] Retry incompatible replication schemas without topic unload (#21767) [fix][client] Fix message leaks and size and permit accounting races when clearing the consumer’s incoming queue (#27025) [fix][test] Stub the cancellable listener in PulsarLedgerCreateRecoveryTest (#27021) [fix][broker] Reset fixed delay detection when the in-memory delayed delivery tracker is cleared (#26972) [fix][broker] Fix managed-ledger scheduler thread stalls when estimating offline backlog (#26956) [improve][broker] Reduce allocations on dispatch and managed-ledger read hot paths (#26960) [improve][broker] Cache schema payloads to stop re-reading BookKeeper history on every explicit-version request (#27001) [fix][client] Skip zero-queue permits for completed async receives (#26998) [fix][ml] Preserve individual batch ACKs when merging cumulative ACKs (#26986) [fix][client] Complete a TableView refresh waiting for a message that could not be applied (#26845) [fix][meta] Fix repeated metadata recovery failures after a ZooKeeper session expires (#26900) [fix][broker] Prevent changing shadow sources through topic properties (#26896) [fix][meta] Recover ambiguous ledger metadata creation using cToken (#26913) [fix][test] Make transaction buffer recovery placement deterministic (#26906) [fix][broker] Reject unsupported subscription-only backlog requests (#26996) [fix][broker] Resume snapshots after re-enabling deduplication (#26889) [fix][client] Release the output buffer when a compression codec fails to decode (#27018) [fix][ml] Don’t fail an add on a terminated managed ledger before the adds in flight ahead of it (#27006) [improve][ml] Read a cursor’s close replay positions with one range read (#27017) [fix][broker] Don’t continue the REST topic listings that waited for a size estimate one after another on a metadata store thread (#17411) [improve][broker] Let authorization providers restrict topic auto-creation (#26988) [improve][doc] Note in CODING.md that a delayedExecutor() task cannot be taken back (#27013) [fix][test] Fix flaky PulsarLedgerUnderreplicationManagerCloseTest.testCloseAggregatesFailuresAndRetriesOnlyUnreleasedLocks (#26979) [improve][broker] Enable Log4j2 async loggers again (#27015) [fix][broker] Fix the broker’s WebSocket service ignoring configurationStoreConfigPath (#20488) [fix][ml] Fix race in ManagedCursorImpl skip and getNthEntry that could skip unacknowledged entries (#26925) [fix][broker] Preserve non-durable cursor ownership during cleanup (#26999) [fix][proxy] Fix Proxy Protocol forwarding for UNKNOWN and IPv6 (#27012) [improve][test] Add TLS and JWT authentication to the performance testing framework (#26825) [improve][broker] Default numIOThreads to half the available processors, at least 8 (#24256) [fix][meta] Unregister metadata store listeners when their owners are closed (#26945) [fix][broker] Fix topic listing memory limit enforcement (#26822) [improve][broker] Record the deduplication’s persisted sequence IDs in place (#27011) [improve][doc] Warn about /tmp cleanup in narExtractionDirectory docs (#26968) [improve][broker] Hash Key_Shared sticky keys without allocation (#26984) [improve][broker] Add scalable topic stats diagnostics (#27007) [fix][broker] Don’t dispatch a replay read that was pending when all consumers of a Shared subscription disconnected (#27008) [improve][bk] Partition index and filter blocks in the entry location RocksDB config (#26267) [fix][broker] Authorize cluster listing requests (#26983) [improve][broker] Add scalable-topic controller and layout metrics (#27004) [improve][bk] Lower the bookies’ read-ahead after a read cache miss from 1,000 to 100 entries (#27005) [improve][ml] Don’t visit a ledger’s other single-entry pending reads for a read of a single entry (#26982) [feat][broker] Add scalable topic consumer coordination metrics (#26981) [improve][broker] Add scalable topic segment-owner metrics (#26980) [feat][broker] Add scalable-topic session and request metrics (#26701) [fix][broker] Fix spurious placement failures when IsolatedBookieEnsemblePlacementPolicy counts unavailable secondary bookies (#26991) [fix][broker] Fence non-persistent subscriptions before asynchronous … (#26995) [improve][test] Add a script that builds and installs hsdis for perfasm (#26990) [fix][client] Fix the embedded native image configuration and make its smoke test run (#26994) [fix][build] Cap test compilation release at the server Java baseline (#26993) [improve][test] Add IoT scenarios for Failover, compaction, retries, delayed delivery, chunking, pattern subscriptions and TTL to the performance tests (#26975) [improve][test] Add Key_Shared replay queue and stuck consumer scenarios to the performance tests (#26989) [fix][ci] Fix flaky native image build caused by a GraalVM class initializer simulation race

Security Fixes

Apache Pulsar

(#27023) [fix][sec] Bump github.com/klauspost/compress from 1.18.0 to 1.18.7 in /pulsar-function-go