flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eilander, J. (Jorn)" <Jorn.Eilan...@ing.nl>
Subject Persistency/Reliability one source - multiple channels
Date Thu, 25 Feb 2016 13:35:08 GMT

I have the following use case:

1 Tibco source with events that need to be written to two different sinks, one custom Oracle
sink and one HDFS (or File) sink. In the docs I've read that channels are quite good at being
persistent, especially in a 1:1 relation with a sink. So my initial setup would contain one
source, two channels, and each channel one sink.

But let's say that one of my channels fails for some reason; What happens then.

-          Does the source 'stop' until it can reach the channel again?

-          Does the source continue to the working channel but remembers where the failing
channel left, and once it starts it resends the events?

-          Does the source keep sending to the working channel and forgets about the failing

The issue here is that both sinks _MUST_ be in sync (pun intended), and duplicates are not
allowed to happen in either sink.

I will see if I can get some test cases up and running tomorrow, but I can't be the only one
having to figure this out. Any help and/or ideas are more than welcome.

With kind regards,

Jorn Eilander

The information in this e-mail is confidential and only meant for the intended recipient.
If you are not the intended recipient , don't use or disclose it in anyway. Please let the
sender know and delete the message immediately.

View raw message