CCAAK · Question #65
Kafka Connect is running on a two node cluster in distributed mode. The connector is a source connector that pulls data from Postgres tables (users/payment/orders), writes to topics with two…
The correct answer is B. Increase the number of Connect Tasks (tasks max value). The connector is currently configured with "tasks.max": "1", which means only one task is handling all tables (users, payment, orders). This can create a bottleneck and lead to lag. Increasing tasks.max allows Kafka Connect to parallelize work across multiple tasks, which can…
Question
Kafka Connect is running on a two node cluster in distributed mode. The connector is a source connector that pulls data from Postgres tables (users/payment/orders), writes to topics with two partitions, and with replication factor two. The development team notices that the data is lagging behind. What should be done to reduce the data lag*? The Connector definition is listed below:
{ "name": "confluent-postgresql-source", "connector class": "PostgresSource", "topic.prefix": "postgresql_", & nbsp;& nbsp;& nbsp;… "db.name": "postgres", "table.whitelist": "users.payment.orders”, "timestamp.column.name": "created_at", "output.data format": "JSON", "db.timezone": "UTC", "tasks.max": "1" }
Options
- AIncrease the number of Connect Nodes.
- BIncrease the number of Connect Tasks (tasks max value).
- CIncrease the number of partitions.
- DIncrease the replication factor and increase the number of Connect Tasks.
How the community answered
(18 responses)- B83% (15)
- C11% (2)
- D6% (1)
Explanation
The connector is currently configured with "tasks.max": "1", which means only one task is handling all tables (users, payment, orders). This can create a bottleneck and lead to lag. Increasing tasks.max allows Kafka Connect to parallelize work across multiple tasks, which can pull data from different tables concurrently and reduce lag.
Topics
Community Discussion
No community discussion yet for this question.