What’s Changed
🎉 Major New Features
Advanced Data Lake and Lakehouse Support (Brand New Feature)
- Unity Catalog Integration: Full support for Databricks Unity Catalog with OAuth2 authentication
- Snowflake Open Catalog: Native support for Snowflake open catalog integration
- S3 Table Support: Support for external/managed S3 tables with proper secret management
- Multi-Catalog Support: Support for multiple catalogs including Unity, Tabular, Polaris, and Open Catalog
- Lakehouse Types: Support for Delta, Iceberg, and combined Delta+Iceberg lakehouse types
- Table Modes: Support for both MANAGED and EXTERNAL table modes
- Catalog Configuration: Flexible catalog configuration with URI and secret management
Enhanced Compaction Scheduler (Major Enhancement)
- Azure DFS Support: Added support for Azure Data Lake Storage Gen2 (AzureDfs) backend storage
- Pod Labels Support: Ability to specify custom pod labels for compaction scheduler
- Resource Scaling: Compaction scheduler now scales to half of broker replicas for optimal performance
- S3 Secret Management: Enhanced AWS S3 client secret mounting for compaction service
- Region Unification: Unified region field configuration for Ursa and compaction services
- NPE Fixes: Fixed CompactionScheduler null pointer exceptions
Ursa Cloud Storage Enhancements (Major Improvement)
- Generic Bucket Configuration: Support for generic bucket configurations beyond S3
- Region Field Unification: Unified region field handling across Ursa and compaction
- Secret Path Management: Improved secret mount path handling when Ursa is enabled
- AWS Region Override: Support for overwriting AWS region for Polaris catalog
- Lakehouse Integration: Enabled lakehouse type support on Ursa and broker services
Advanced Reconciliation Controls (New Feature)
- Coordinator Pause: Support for pausing Pulsar coordinator reconciliation with annotations
- Oxia Shard Sorting: Fixed Oxia shard reconciliation with proper sorting
- Namespace Reconciliation: Improved Oxia namespace reconciliation logic
- Cleanup Logic: Enhanced replication cluster cleanup logic for new Pulsar versions
🔧 Feature Improvements
Graceful Rollout Enhancements
- Label Fixes: Fixed graceful rollout label handling
- Function Owner: Fixed function owner assignment during graceful rollout
- Servlet Support: Enhanced servlet support for graceful rollout operations
Toolset and Resource Management
- Memory Limits: Increased default memory limits for toolsets
- Pod Resources: Added pod resource configuration for toolset
- Auto Release: Support for automatically getting latest releases
Configuration Management
- Backend Storage Type: Corrected compaction backend storage type configuration names
- Log4j Sync: Synchronized Pulsar log4j configuration
- Rest Message API: Proper handling when REST message API is explicitly disabled
- Secret Path Updates: Updated secret paths and added S3 region field support
Istio and Networking
- Zone Affinity: Upgraded to topology.kubernetes.io/zone label for better affinity
- HTTPS Protocol: Fixed Istio HTTPS protocol handling
- Multi-Address Support: Support for multi-address configuration for global ZooKeeper
- Resource Conversion: Added Istio v1beta resource conversion tests
🐛 Bug Fixes
Security and CVE Fixes
- Go Version Upgrade: Upgraded Go version to fix CVE vulnerabilities
- Crypto Upgrades: Upgraded crypto libraries to fix CVE-45337 and other security issues
- Golang Module: Upgraded golang.org/x/net module to fix CVE vulnerabilities
Compaction and Storage Fixes
- S3 Secret Mounting: Fixed S3 secret mounting for S3 tables
- Compaction NPE: Fixed CompactionScheduler null pointer exceptions
- Storage Path: Fixed mount path for custom storage configurations
- Backend Storage: Corrected compaction backend storage type configuration names
Reconciliation and Controller Fixes
- Oxia Shards: Fixed Oxia shard reconciliation with proper sorting
- Graceful Rollout: Fixed graceful rollout label handling
- Function Owner: Fixed function owner assignment during graceful rollout
- Replication Cleanup: Disabled reconcile cleanup for replication clusters in new Pulsar versions
Configuration and Deployment Fixes
- Rest Message API: Fixed handling when REST message API is explicitly disabled
- Log4j Config: Synchronized Pulsar log4j configuration
- Secret Management: Improved secret path and region field handling
- Init Container: Removed init container for autorecovery
🚀 Performance Optimizations
Compaction Performance
- Resource Scaling: Compaction scheduler now scales to half of broker replicas
- S3 Integration: Enhanced AWS S3 client secret mounting for better performance
- Region Optimization: Unified region field handling for better resource utilization
Reconciliation Performance
- Shard Sorting: Fixed Oxia shard sorting for better reconciliation performance
- Pause Controls: Added ability to pause coordinator reconciliation when needed
- Cleanup Logic: Improved replication cluster cleanup logic
Resource Management
- Memory Limits: Increased default memory limits for toolsets
- Pod Resources: Better pod resource allocation
- Auto Scaling: Improved auto-scaling capabilities
🔄 Dependencies and Infrastructure
Security Updates
- Go Version: Upgraded Go version to fix CVE vulnerabilities
- Crypto Libraries: Upgraded crypto libraries for security fixes
- Golang Modules: Updated golang.org/x/net module
CI/CD Improvements
- JQ Installation: Fixed JQ installation in CI workflows
- Disk Cleanup: Improved CI disk cleanup processes
- Test Coverage: Added Istio v1beta resource conversion tests
External Dependencies
- Function Mesh: Updated function mesh configurations
- Kafka Connect: Fixed image capabilities for Kafka Connect
- PFSQL: Updated PFSQL integration
📋 Configuration Changes
New Configuration Options
- Lakehouse Type: New lakehouse type configuration for Delta, Iceberg support
- Catalog Configuration: New catalog configuration for Unity, Snowflake, S3 tables
- Table Modes: New table mode configuration (MANAGED/EXTERNAL)
- Region Fields: Unified region field configuration
- Pod Labels: Compaction scheduler pod label configuration
Enhanced Configuration
- Ursa Generic Bucket: Generic bucket configuration support
- S3 Region: Enhanced S3 region field support
- Secret Paths: Updated secret path configurations
- Backend Storage: Corrected backend storage type names
⚠️ Breaking Changes
- Compaction Backend Storage: Backend storage type configuration names have been corrected
- Secret Mounting: S3 secret mounting behavior changed for S3 tables
- Region Fields: Region field handling has been unified across Ursa and compaction
- Graceful Rollout Labels: Graceful rollout label handling has been updated
📚 Documentation and Examples
- New Unity Catalog configuration examples
- Snowflake Open Catalog integration guides
- S3 table configuration documentation
- Lakehouse type configuration examples
- Multi-catalog setup guides