Kafka cluster health monitoring, consumer lag detection, broker metrics and anomaly intelligence agent
Consumer lag, broker metrics (CPU, heap, GC), topic throughput (msg/sec), partition counts, and replica sync status.
Synthetic, Real Kafka (bootstrap), Prometheus JMX, Schema Registry, ZooKeeper, KafkaConnect, MirrorMaker replication tracking.
cluster_overview, consumer_lag, broker_metrics, topic_metrics, detect_anomalies with Claude tool-use loop.
Cluster health score, broker status summary, URP count, active anomalies, consumer lag trend chart.
All groups sorted by lag (worst-first), lag trend (growing/stable/recovering), messages/min, group state.
Topic throughput (msg/sec, bytes in/out), retention usage %, partition count, replication factor, history.
Per-broker CPU%, heap%, GC pause ms, request handler idle%, URP count, disk%, metrics trend.
Connector status (running/failed/paused), failed task count, connector type (source/sink), state history.
Schema versions, compatibility levels per subject, schema evolution tracking, subject listing.
ZooKeeper node health, leader election status, broker registration state, config changes.
Cross-cluster replication lag, checkpoint offsets, mirror mode (active/passive), replication rate.
Anomaly detection (critical/warning severity), remediation steps, chat Q&A with tool-use, pattern learning.
| Feature | Status | Notes |
|---|---|---|
| Synthetic data generation | ✓ Working | Demo/testing mode with realistic broker, consumer, topic metrics |
| Real Kafka cluster connection | ✓ Working | Bootstrap servers, SASL/TLS auth, dynamic broker discovery |
| Consumer lag monitoring | ✓ Working | Real-time lag, trend detection (growing/stable/recovering) |
| Broker metrics collection | ✓ Working | CPU, heap, GC pause, URP, request handler idle%, disk% |
| Topic throughput monitoring | ✓ Working | Messages/sec, bytes in/out, retention %, partition/replication tracking |
| Anomaly detection engine | ✓ Working | High GC pause, URP spikes, consumer lag breach, broker degradation |
| KafkaConnect status | ✓ Working | Connector state, failed task tracking, task count per connector |
| Schema Registry integration | ✓ Working | Schema versions, compatibility modes, subject tracking |
| ZooKeeper health monitoring | ✓ Working | Node health, leader election, broker registration |
| MirrorMaker replication lag | ✓ Working | Checkpoint offset tracking, replication rate, lag trend |
| AI tool-use loop (5 functions) | ✓ Working | cluster_overview, consumer_lag, broker_metrics, topic_metrics, detect_anomalies |
| PostgreSQL metrics history | ✓ Working | Time-series storage for trend analysis beyond real-time |
| Prometheus JMX integration | ⏳ Planned | Collector exists, wiring to kafka_store pending |
| Vector/RAG knowledge base | ⏳ Planned | Embeddings for anomaly pattern matching, historical context |
| Graph topology (Neo4j) | ⏳ Planned | Broker → topic → consumer group DAG visualization |
| Automated remediation (Action Agent) | ⏳ Planned | Trigger remediation steps (rebalance, pause connector, etc.) |
| Multi-cluster comparison | ⏳ Planned | Side-by-side lag/throughput comparison across clusters |
| Alert escalation to Teams | ⏳ Planned | Via shared escalation notifier (incident_management integration) |