Message duplication in Kafka typically comes from producer retries creating duplicate writes or consumers reprocessing messages after a rebalance, and both are addressed with idempotence and careful offset handling.
Key Points: • Enable idempotent producers (enable.idempotence=true) so retried sends are deduplicated by the broker using sequence numbers, preventing duplicate writes from retries. • Use Kafka's transactional API or Kafka Streams' exactly-once semantics (processing.guarantee=exactly_once_v2) to atomically produce results and commit offsets together. • Disable auto-commit (enable.auto.commit=false) and commit offsets manually only after processing completes, so a rebalance doesn't cause reprocessing of already-handled messages. • Where true exactly-once isn't feasible end-to-end (e.g. writing to a non-transactional external system), make the downstream write idempotent instead, such as using an upsert keyed by message ID.
Example: A payment service enabling idempotent producers and switching to manual offset commits after each transaction is durably written avoids charging a customer twice even if a producer retry or a mid-batch consumer rebalance occurs.
Interview Tip: A concise interview answer is:
"I'd enable idempotent producers to eliminate retry-caused duplicates, use transactional or exactly-once semantics where the processing needs it, and switch consumers to manual offset commits after processing completes so a rebalance can't cause messages to be reprocessed."