I have 2 flume instances running (1.4 version), each instance has a configuration to rotate HDFS sink file per hour. I get 24 HDFS sink files created / day from each instance and 48 from 2 instances.
Instead these 48 files, I want only 24 files created, so I tried these below option:
In both flume instances I configured only source and channel, so that both the instances will write to same channel then, I created a one more instance that has only channel and sink which suppose to read all the data from channel and write it into sink, but this did not work. This new sink channel was able to read the existing data files from channel but did not pick any of the new data files created from other flume instances.
Can someone please help on how to fix this issue?