What’s Changed
🚀 New Features
Unity Catalog Support
- Unity Iceberg Catalog Integration: Added comprehensive support for Unity Catalog with Iceberg backend
- OAuth2 Authentication: Support for OAuth2 server URI configuration for Unity Iceberg catalog
- Catalog Configuration: Enhanced catalog management with support for catalog name, schema name, and user agent configuration
- Multi-Catalog Support: Support for Unity, Tabular, and Polaris catalog backends
Agent Function Support
- Agent Function Controller: New controller for managing agent functions with full lifecycle support
- Feature Gate: Added AGENT_FUNCTION_ENABLE feature gate (enabled by default)
- StatefulSet Management: Complete StatefulSet generation and management for agent functions
Enhanced Compaction Service
- Compaction Scheduler: Added data source configuration for compaction scheduler
- Ursa Lakehouse Integration: Support for Ursa Lakehouse in tiered storage configuration
- Token Refresh: Enhanced token refresh capabilities for compaction scheduler
- Memory Optimization: Fixed heap and direct memory configuration for compaction service
Kafka Connect Improvements
- RBAC Support: Enabled Kafka RBAC for versions > 4.0.5.0
- SASL Mechanisms: Support for SCRAM-SHA-256 and SCRAM-SHA-512 authentication
- Servlet Configuration: Enhanced ksn-rest-servlet support for versions >= 4.0.4.4
- OAuth2 Configuration: Fixed KOP OAuth2 configuration errors with comprehensive e2e tests
🔧 Enhancements
Pulsar Components
- Active Consumer Failover: Enabled activeConsumerFailoverConsistentHashing by default
- Parallel PMP: Use Parallel PMP during broker initialization for improved performance
- Dynamic Min Ready Seconds: Support for dynamic minimum ready seconds configuration
- Broker Cleanup: Improved cleanup of Broker pods when StatefulSet gets deleted
MQTT Proxy
- Virtual Service Port: Fixed virtual service port configuration for MQTT proxy
- ConfigMap Improvements: Enhanced MQTT configuration management
- Address Configuration: Removed MQTT advertised address for better networking
Monitoring & Observability
- License Metrics: Added comprehensive license metrics for SN operator
- Cloud Telemetry: Fixed cloud telemetry data event time synchronization
- Coordinator Logging: Enhanced coordinator logging when updating components
Resource Management
- Resource Limits: Added CU/SU resource limit enforcement in pre-reconciler
- License Claims: Added enforce_resource_limits in license claims
- Feature Gates: Improved feature gate management and configuration
🔒 Security Updates
Dependency Updates
- Go Version: Upgraded to Go 1.23.12 to address security vulnerabilities
- OAuth2 Library: Updated golang.org/x/oauth2 to v0.27.0 to fix CVE-2025-22868
- Pulsar Client: Upgraded pulsar-client-go to fix Go client panic issues
- Kubernetes: Updated to Kubernetes v0.30.9 for improved security and stability
Authentication & Authorization
- Token Authentication: Enhanced token authentication for graceful rollout
- JWT Support: Improved JWT token handling and validation
- RBAC Enhancements: Strengthened role-based access control across components
🐛 Bug Fixes
Core Fixes
- Virtual Hosts Order: Fixed virtual hosts order in Envoy filters
- Token Parameters: Fixed token parameters for graceful rollout
- Compaction Cache: Fixed crash when enabling compaction cache
- Destination Rule: Fixed label configuration for destination rule
Component-Specific Fixes
- Kafka Connect Servlet: Fixed servlet handling for upstream removal
- Pulsar Rollout: Fixed missing pulsar-rollout servlet
- Ursa Configuration: Updated Ursa flag check to avoid NPE
- Classic Engine: Renamed classic to pulsar for better clarity
Infrastructure Fixes
- CI/CD: Removed SSH access from CI and improved test stability
- Image Management: Fixed image tag usage in CI instead of using ‘latest’
- Test Improvements: Upgraded Kafka version to 7.9.1 for better test reliability
📦 Dependencies
Updated Dependencies
- Istio: Updated to Istio v1.24.2 for improved service mesh capabilities
- Prometheus: Updated Prometheus operator to v0.74.0
- Controller Runtime: Updated to v0.18.6
- Function Mesh: Updated to v0.21.8
Go Module Updates
- OpenTelemetry: Updated to v1.27.0 for enhanced observability
- Kubernetes Libraries: Updated all Kubernetes-related libraries to v0.30.9
- Protobuf: Updated to v1.34.1
🚨 Breaking Changes
Configuration Changes
- Classic to Pulsar: Renamed “classic” references to “pulsar” throughout the codebase
- Unity Type Removal: Removed unity type and added catalog name configuration
- Compaction Service: Removed unused configuration for compaction service
API Changes
- Catalog Configuration: Updated catalog configuration structure for Unity support
- Feature Gates: Some feature gates have been reorganized and renamed
🔄 Migration Notes
From v0.11.7 to v0.12.9
- Update Configuration: Review and update any “classic” references to “pulsar”
- Feature Gates: Check feature gate configurations, especially for new Agent Function support
- Catalog Configuration: Update catalog configurations if using Unity Catalog features
- Dependencies: Ensure Kubernetes cluster supports v0.30.9 APIs
📋 Upgrade Instructions
- Backup: Create a backup of your current configuration
- Update CRDs: Apply the new CRD definitions
- Update Operator: Deploy the new operator version
- Verify: Check that all components are running correctly
- Test: Validate new features in a test environment before production use
🎯 Performance Improvements
- Faster Scaling: Improved scaling performance with FastScalingUp feature gate
- Memory Optimization: Better memory management for compaction services
- Parallel Processing: Enhanced parallel processing during broker initialization
- Resource Efficiency: Improved resource utilization across all components
📚 Documentation
- Updated API documentation for new Unity Catalog features
- Enhanced configuration examples for Agent Functions
- Improved troubleshooting guides for common issues
- Added migration guides for breaking changes