Hi All,

Here's my current flume setup for a hadoop cluster to collect service logs

- Run flume agent in each of the nodes
- Configure flume sink to write to hdfs and the files end up in this way



But I want to be able to write all the logs from multiple agents to a single file in hdfs . How can I achieve this and what would the topology look like. 
can this be done via collector ? If yes, where can I run the collector and how will this scale for a 1000+ node  cluster.