Apache Kafka How to delete data from Kafka topic? – you probably didn’t know these 2 cool methods!

When working with Apache Kafka, there may be a situation when [ Apache Kafka How to delete data from Kafka" topic ] we need to delete data from topic, because e.g. during testing junk data was sent, and we have not yet implemented support for such errors, resulting in the so-called “poison pill” – that is, a record (s) that each time we try to consume from Kafka cause that our processing fails.

Introduction

In Apache Kafka, deleting messages from a topic is not a straightforward process because Kafka is designed to retain messages for a certain amount of time. However, there are a few options you can consider to effectively remove messages from a Kafka topic:

1. Retention policy: You can set a retention policy on a topic to specify how long Kafka should keep the messages for that topic. When the retention period for a message expires, Kafka will automatically delete it. To set a retention policy, you can use the delete.retention.ms and retention.bytes configuration parameters when you create a topic.
2. Log compaction: You can enable log compaction on a topic to retain only the latest value for each key in the topic. When log compaction is enabled, Kafka will periodically remove any older duplicate messages for a key and only keep the latest value. This can be useful if you want to keep a compact history of the changes to a key over time.
3. Custom consumer: You can write a custom consumer application that reads messages from a topic, processes them, and then discards them. This approach allows you to selectively delete messages based on your own business logic.
4. Topic deletion: As a last resort, you can delete the entire topic and recreate it. This will permanently delete all messages in the topic and is not reversible.

Keep in mind that deleting messages from a Kafka topic can have unintended consequences, such as affecting downstream consumers that rely on those messages. It’s important to carefully consider the implications of deleting messages before doing so.

Kafka Topic And Partitions

The following diagram presents how data is stored in Kafka Topic.

Kafka Topic consists of partitions, which amount can be >=1. The following example shows topic which has 4 partitions. On each partitions the data is stored (each message is presented as single rectangle). Also the each message has the offset – the order id which determines the order in partitions. When consumer reads the data from the messages are read using this offset to keep the order of messages to be read.

Kafka Topic

In Apache Kafka, a topic is a logical category or feed name to which records are published. Producers write data to topics and consumers read from topics.

Topics are used to organize the streams of data in Kafka. Each record in a Kafka topic consists of a key, a value, and a timestamp. The key and value are arbitrary byte arrays and can be used to store any type of data. The timestamp is the time at which the record was written to the topic.

Topics are partitioned, which means that the data in a topic is distributed across multiple servers in the Kafka cluster. Each partition is an ordered, immutable sequence of records that is continuously appended to.

Topics can have multiple consumers, and each consumer can read from a different offset in the topic, allowing for parallel processing of the data.

In Kafka, topics play a central role in the architecture and are used to publish and subscribe to streams of data. They are a key concept in Kafka and are used in a wide variety of data processing and streaming applications.

Topic Partitions

In Apache Kafka, a topic is divided into one or more partitions, which are distributed across the Kafka cluster. Each partition is an ordered, immutable sequence of records that is continuously appended to.

The number of partitions in a topic can be specified when the topic is created, and it can be increased or decreased later as needed. Increasing the number of partitions allows the topic to scale horizontally and handle more data, but it also increases the complexity of the system.

Each partition has a unique ID, and the records in a partition are assigned an offset, which is a sequential ID for the record within the partition. The offset is used to uniquely identify each record in the partition and to maintain the order of the records.

Partitions are used to scale the data in a topic and to allow multiple consumers to read from a topic in parallel. Each consumer can read from a different partition, allowing for parallel processing of the data.

In Kafka, partitions play a crucial role in the architecture and are used to distribute the data in a topic across the Kafka cluster. They are a key concept in Kafka and are used to scale the data and enable parallel processing of the data in a topic.

Apache Kafka How To Delete Data From Kafka Topic?

In Apache Kafka, data in a topic is typically retained for a configurable amount of time, after which it is automatically deleted by the Kafka cluster. By default", data is retained for a week, but this retention period can be modified using the log.retention.hours configuration property.

However, it is also possible to delete data from a Kafka topic manually using the delete.topic.enable configuration property and the kafka-topics command-line tool.

#1 Method

To delete data from a Kafka topic, you can follow these steps:

1. Set the delete.topic.enable configuration property to true. This will allow you to delete topics from the Kafka cluster.
2. Use the kafka-topics tool to delete the topic. The syntax for deleting a topic is as follows:

kafka-topics --bootstrap-server <bootstrap-server> --delete --topic <topic-name>

Replace <bootstrap-server> with the host and port of the Kafka broker, and <topic-name> with the name of the topic you want to delete.

3. Verify that the topic has been deleted by using the kafka-topics tool to list the topics in the Kafka cluster. The deleted topic should not be listed.

Keep in mind that deleting a topic is a permanent operation and cannot be undone. It is also worth noting that deleting a topic will not delete the data stored in the topic’s partitions. Instead, it will mark the topic for deletion and the data will be deleted when the retention period expires.

It is generally not recommended to delete topics unless you are sure you no longer need the data stored in the topic.

#3 Method (Not Recommended)

We can simply delete the topic and create it again. Personally, I think it is better to use the second method, i.e.

#3 Method

The second way is to change the data retention on the topic to some low value, e.g. 1 second. The data will be automatically deleted by Kafka’s internal processes. We don’t have to worry about anything.

First, let’s check the current configuration of the topic: retention.ms=86400000 (7 days) Apache Kafka How to delete data from Kafka" topic)

kafka-topics --zookeeper kafka:2181 --topic bigdata-etl-file-source -describe

Topic:bigdata-etl-file-source	PartitionCount:1	ReplicationFactor:1	Configs:retention.ms=86400000
	Topic: bigdata-etl-file-source	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

Retention SET To 1 Second

kafka-configs --zookeeper <zookeeper>:2181 --entity-type topics --alter --entity-name bigdata-etl-file-source --add-config retention.ms=1000

Check configuration:

kafka-configs --zookeeper <zookeeper>:2181 --entity-type topics --alter --entity-name bigdata-etl-file-source --add-config retention.ms=1000

kafka-configs --zookeeper kafka:2181 --entity-type topics --alter --entity-name bigdata-etl-file-source --add-config retention.ms=1000

Remember to wait for a while (about 1 minute) for the data to be deleted.

After we verify that the data has already been removed from the topic, we can restore the previous settings. (Apache Kafka How to delete data from Kafka" topic)

Removing Messages from a Kafka Topic, How to delete records from a Kafka topic, How to delete a Kafka Topic

Could You Please Share This Post? 
I appreciate It And Thank YOU! :)
Have A Nice Day!