flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Praveen Ramachandra <praveen_ramachan...@yahoo.com>
Subject Re: Flume-NG Channels
Date Tue, 10 Jan 2012 05:36:48 GMT
They are in low 100's in the best case scenario, and could be in 1000 in the worst case scenario.

I believe this aspect can be pretty much shielded from application if the underlying platform
has the right set of responsibilities.

Praveen Ramachandra

 From: Ralph Goers <ralph.goers@dslextreme.com>
To: flume-user@incubator.apache.org 
Sent: Monday, January 9, 2012 6:53 PM
Subject: Re: Flume-NG Channels

On Jan 9, 2012, at 2:28 AM, Praveen Ramachandra wrote:

>We were trying to design a multi-tenanted system using flume-ng, where each logically
independent data set is modelled through a channel going through the system of collectors,
aggregators and delivery agents (to end destination). Each channel will carry data that logically
belong together. The requirement is that we should be able to bring up and tear down a channel
with ease.
>When we completed the exercise, it turned out that we have to run a separate Source/Sink,
at a designated host/port combination for each channel. The issue with this is that, it is
an operational overhead that we have work with net-ops to punch holes in the firewall to let
tcp traffic flow on non-standard ports. I would imagine that it would be the case in many
organizations as well.
>Two questions.
>1. Did I get the modeling right with flume-ng
>2. Is there a better way to do it at a platform level
>            2.1 I know if I can write a bunch of custom sinks/sources and embed
a notion of channel to which each events belong to in the message, I can effectively mux and
demux the events at either ends.
>            2.2 Which means the default support for channel is also not of much

What is your target destination(s) for the tenants?  Can they all flow through a single channel
in Flume and then be delivered to the correct destination by a smarter sink at the end?

View raw message