Monitoring Kafka cluster health means tracking broker, topic, and client-side metrics that reveal throughput, latency, and resource pressure so problems can be caught before they cause outages or data loss.
Key Points: • Consumer lag (records-lag-max or per-partition lag) shows whether consumers are keeping up with producers. • Producer request latency and error rate reveal whether writes are slow or failing. • Broker CPU, memory (especially page cache and heap/GC), and disk I/O indicate resource saturation. • Under-replicated and offline partitions signal replication problems or broker outages. • Network throughput (bytes-in/bytes-out per second) shows whether the cluster is approaching bandwidth limits.
Example: If dashboards show consumer lag climbing while broker disk I/O is spiking, the likely cause is disk contention slowing down log writes/reads, so the fix might be adding faster disks or spreading partitions across more brokers rather than just adding consumers.
Interview Tip: A concise interview answer is:
"I watch consumer lag, producer latency, broker CPU/memory/disk I/O, network throughput, and under-replicated partitions, and when I see a throughput drop I correlate it against those metrics — usually it traces back to a disk bottleneck, network saturation, or a hot partition from poor key distribution."