Apache Kafka How to delete data from Kafka topic? – you probably didn’t know these 2 cool methods!

You are currently viewing Apache Kafka How to delete data from Kafka topic? – you probably didn’t know these 2 cool methods!
Photo by John Schnobrich on Unsplash
Share This Post, Help Others, And Earn My Heartfelt Appreciation! :)
4.9
(1607)

When working with Apache Kafka, there may be a situation when [ Apache Kafka How to delete data from Kafka topic ] we need to delete data from topic, because e.g. during testing junk data was sent, and we have not yet implemented support for such errors, resulting in the so-called “poison pill” – that is, a record (s) that each time we try to consume from Kafka cause that our processing fails.

Apache Kafka How To Delete Data From Kafka Topic?

We can simply delete the topic and create it again. Personally, I think it is better to use the second method, i.e.

#2 Method: Retention Change

The second way is to change the data retention on the topic to some low value, e.g. 1 second. The data will be automatically deleted by Kafka’s internal processes. We don’t have to worry about anything.

First, let’s check the current configuration of the topic: retention.ms=86400000 (7 days) Apache Kafka How to delete data from Kafka topic)

kafka-topics --zookeeper kafka:2181 --topic bigdata-etl-file-source -describe

Topic:bigdata-etl-file-source	PartitionCount:1	ReplicationFactor:1	Configs:retention.ms=86400000
	Topic: bigdata-etl-file-source	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

Retention SET To 1 Second

kafka-configs --zookeeper <zookeeper>:2181 --entity-type topics --alter --entity-name bigdata-etl-file-source --add-config retention.ms=1000

Check configuration:

kafka-configs --zookeeper <zookeeper>:2181 --entity-type topics --alter --entity-name bigdata-etl-file-source --add-config retention.ms=1000

kafka-configs --zookeeper kafka:2181 --entity-type topics --alter --entity-name bigdata-etl-file-source --add-config retention.ms=1000

Remember to wait for a while (about 1 minute) for the data to be deleted.

After we verify that the data has already been removed from the topic, we can restore the previous settings. (Apache Kafka How to delete data from Kafka topic)

Removing Messages from a Kafka Topic, How to delete records from a Kafka topic, How to delete a Kafka Topic

If you enjoyed this post please add the comment below and share this post on your Facebook, Twitter, LinkedIn or another social media webpage.
Thanks in advanced!

How useful was this post?

Click on a star to rate it!

Average rating 4.9 / 5. Vote count: 1607

No votes so far! Be the first to rate this post.

Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments