CCDAK Sample Questions Answers

Table→ Product, Customers

Stream→ Orders_Placed, Shipment_Of_Orders

Tablesrepresent slowly changing,reference data— e.g., customer profiles, product info.

Streamsrepresentreal-time event data— e.g., orders placed, shipments processed.

FromKafka Streams Documentation:

“Use KTable for reference datasets, and KStream for event-based processing such as orders or logs.”

The official CLI utility for consuming messages from Kafka topics is kafka-console-consumer.sh. It connects to the broker, consumes messages, and prints them to standard output.

FromKafka CLI Tools Documentation:

"kafka-console-consumer.sh is used to read data from a Kafka topic and write it to standard output."

The other options are not valid Kafka CLI tools.

This syntax is a clear match toProtocol Buffers (Protobuf). It defines a schema with fields, types, and tags, which is a format supported by Kafka when usingProtobuf-based schema registry serialization.

FromConfluent Schema Registry Docs:

“Kafka supports Protobuf serialization, where schemas are written in .proto files and include fields with tags.”

int32, double, and string are standard Protobuf types.

Avro uses JSON-style schema.

JSON Schema uses JSON object structure, not .proto.

YAML is unrelated.

If the consumer fails to send heartbeats in time (due to being slow under load), it is considereddead, causing a rebalance. Increasing session.timeout.ms gives the consumermore time to process messages and send heartbeats, preventing premature removal.

FromKafka Docs > Heartbeats and Failures:

“If the consumer does not send a heartbeat in session.timeout.ms, it is considered dead and the group coordinator will trigger a rebalance.”

Reducing max.poll.records mightslow consumption further. More consumers won’t help since each partition already has a dedicated consumer.

Kafka Connect can run instandalone modeordistributed mode. For simple tasks likereading logs from a file on a single host,standalone modeis recommended.

FromKafka Connect User Guide:

“Standalone mode is useful when running connectors on a single machine (e.g., for development or simple deployments like log collection from a local file).”

Distributed mode is preferred for scalability and fault tolerance but overkill for this use case.

A:Kafka batches storecompression metadata in their headers, indicating how the batch was compressed.

B:Compression is controlled via theproducer config compression.type, which defaults to none. Producers explicitly set Gzip or other formats.

FromKafka Producer Docs:

“Compression is done by the producer, and the type is stored in the message batch metadata. The broker does not recompress.”

C is false: messages are serializedbeforecompression.

D is false: messages without keys are distributed acrossmultiple partitions.

In Kafka Streams,the number of stream tasks = number of input partitions × num.stream.threads × number of instances,but only as many as the number of partitions can actively consume at once.

However, in this case,Kafka Streams assigns one task per partition, and because there are 5 partitions and 15 threads (3 instances × 5 threads),15 tasks are created, andall 15 can be activedepending on processing topology.

FromKafka Streams Developer Guide:

"Kafka Streams creates one task per input partition. If you increase the number of stream threads, it runs multiple tasks in parallel within a single instance."

So,15 stream tasks will be created and 15 will be actively consuming.

Increasing partitions increasesparallelism, but also means:

Consumers in a group may have to handlemore partitions, especially if the number of consumers is lower than the number of partitions.

This can result inincreased lag, especially under high load.

FromKafka Topic Management Docs:

“Increasing the number of partitions increases consumer work, and if consumers can’t keep up, lag can accumulate.”

A is false:existing data is not redistributed.

B is false:records with the same key always map to the same partitionbased on hash.

D is not directly impacted by the partition count.

TheStickyAssignortries to minimize partition movement bypreserving existing assignmentsas much as possible while still achieving a balanced assignment. This improvesconsumer stabilityand reduces rebalances.

From theKafka Consumer Assignor Documentation:

“The StickyAssignor attempts topreserve as many existing assignments as possible, which helps minimize partition movement between rebalances.”

RoundRobinAssignor focuses on even distribution, not stability.

RangeAssignor groups partitions by topic and assigns them consecutively, but can lead to imbalances.

PartitionAssignor is an abstract base class, not an assignor used directly.

With acks=all, the leader waits formin.insync.replicasto acknowledge the message. Since min.insync.replicas=3, Kafka will only commit the messageonce 3 brokers (leader + 2 followers)confirm they have the message.

FromKafka Documentation > Acks and Durability:

“If acks=all is specified, the producer will wait until the full set of in-sync replicas has acknowledged the record. The minimum number of in-sync replicas is controlled by min.insync.replicas.”

Even though the replication factor is 4, only3 acknowledgments are needed, as defined by min.insync.replicas.

Unit Testing→ MockProducer

Integration Testing→ Testcontainers

Performance Testing→ Trogdor

Mock Data Generation→ Connect Datagen

MockProducer: Simulates a Kafka producer in unit tests (no real broker interaction).

Testcontainers: Spawns Kafka in Docker forreal environment testing.

Trogdor: Kafka’s built-inperformance load testingframework.

Connect Datagen: Createssample source recordsfor test and demo purposes.

FromKafka Developer Tools Guide:

“Kafka developers commonly use MockProducer for unit tests, Testcontainers for integration, and Trogdor for performance tests.”

To increase consumerthroughput, one effective strategy is toincrease the amount of datafetched in each poll by raisingfetch.max.bytesormax.partition.fetch.bytes. This allows each poll to retrieve more records per request, improving processing efficiency.

FromKafka Consumer Config Docs:

“Increasing fetch size allows consumers to retrieve larger batches of messages, improving throughput and reducing request overhead.”

Removing consumers (A) may reduce parallelism.

Manual commit (C) adds complexity, not throughput.

Decreasing session timeout (D) risks unnecessary rebalances.

unclean.leader.election.enable=false→ Prevents data loss by only considering in-sync replicas when rebalancing.

replication.factor→ Specifies how many redundant copies of partitions are distributed across brokers.

min.insync.replicas=2→ Sets the standard for the number of partition instances that must keep up with the latest committed message.

unclean.leader.election.enable=false ensures that onlyin-sync replicascan be elected as leaders. If disabled, an out-of-sync replica may become leader, potentially leading to data loss.

replication.factor defineshow many brokerswill maintain copies of each partition, directly impacting durability and availability.

min.insync.replicas determineshow many replicas must acknowledgea write when acks=all is used, enforcing write durability.

The default maximum message size that a Kafka broker accepts is1MB (1,048,576 bytes), controlled by the config propertymessage.max.bytes.

FromKafka Broker Configuration Docs:

“The default maximum message size is 1MB. To accept larger messages, configure message.max.bytes and the producer’s max.request.size.”

Producers also have a matching limit via max.request.size, and consumers via fetch.message.max.bytes.

SMTs (Single Message Transforms) arelightweight transformationsapplied to individual messages as they pass through Kafka Connect.

Chaining SMTs: You can applymultiple SMTsin sequence by defining them in order in the connector config.

Masking or modifying fieldsis acommon use case(e.g., redacting sensitive data).

FromKafka Connect Documentation:

“Single Message Transforms (SMTs) are applied to individual messages. You can chain multiple SMTs together.”

SMTsdo not perform joins(B is incorrect).

Converters and SMTs areseparate concerns(D is incorrect).