New to flume and I'm trying to relay log messages received over netcat source to Kafka sink.
Everything seems to be fine, except that Flume is acting like it IS assigning a partition key to the produced messages though none is assigned. I'd like the messages to be assigned to a random partition, so that consumers are load balanced.
* Flume 1.6.0
* Kafka 0.9.0.1
Kafka topic config: kafka-topics --zookeeper localhost/kafka --create --topic activity.history --partitions 20 --replication-factor 1
Flume agent listens on 3132tcp for connections, and publishes messages received to the Kafka activity.history topic. I'm running two instances of the Python consumer.
What happens however, is all logs messages get sent to a single Kafka consumer...if I restart Flume (leave consumers running) and re-run the test, all messages get published to the other consumer. So it feels like Flume is assigning a permanent partition key even though one is not defined (and should therefore be random).
Any advice is greatly appreciated.