Log compaction is a Kafka retention policy that keeps only the latest record for each key instead of deleting records purely by age, making compacted topics suitable for representing current state rather than a full event history.
Key Points: • Use a compacted topic (cleanup.policy=compact) when you only care about the latest value per key, such as a changelog of user profile fields or the current state of an entity. • Compaction periodically removes older records for a key, keeping the topic size bounded even as the same keys are updated indefinitely over time. • The trade-off is that historical intermediate values are eventually discarded, so compacted topics are unsuitable when you need to replay the full sequence of events that led to the current state. • Regular (delete-based) topics retain the complete ordered event history for a configured time or size window, which suits event sourcing or audit use cases. • Kafka Streams' internal changelog topics for state stores are a common real-world example of compacted topics.
Example: A "user-profile" topic keyed by user ID and configured with compaction only needs to retain each user's latest profile snapshot, whereas an "order-events" topic recording every step of an order's lifecycle should stay a regular topic so the full history can be replayed.
Interview Tip: A concise interview answer is:
"I'd reach for a compacted topic whenever I only need the latest value per key — like a changelog or current-state table — because it keeps the topic size bounded, but I'd avoid it whenever the full history of events matters, since compaction discards intermediate values over time."