flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From iain wright <iainw...@gmail.com>
Subject Re: Flume Cluster?
Date Fri, 07 Apr 2017 23:06:44 GMT
It depends how you are defining "Cluster".

I generally think of it as mutliple processes that are coordinating and
using some kind of consesous algorithm. (IE Zookeeper). Perhaps my
definition is wrong, or differs from yours.

You can ofcourse achieve durability, high availability, and scale using
flume

Consider:
100 application servers with embedded flume agents
10 downstream flume servers performing aggregations and writes to hdfs
HDFS

Events first come to fruition at the webserver, and are ingested by the
flume embedded agent local to each appserver
appserver flume sends events to the downstream flume aggregate tier, if one
of the 10 servers are unvailable, it trys the next
Finally events are written to HDFS by the aggregate tier

The agents that are aggreating events can be in different availability
zones/datacenters/etc.

You can also replicate events and write to multiple downstream consumers,
route based on contents of the event, etc.

Is there something in particular you are hoping to accomplish? I think with
a more specific question, you will get a better answer




-- 
Iain Wright

This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.

On Fri, Apr 7, 2017 at 2:50 PM, jeff saremi <jeffsaremi@hotmail.com> wrote:

> Saikat, how up to date is this material? Considering what Iain Wright just
> said about lack of such a thing as a cluster?
>
> thanks
>
> JEff
> ------------------------------
> *From:* Saikat Kanjilal <sxk1969@hotmail.com>
> *Sent:* Friday, April 7, 2017 1:24:52 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Flume Cluster?
>
>
> http://ankitasblogger.blogspot.com/2011/05/installing-flume-in-cluster-
> complete.html
>
> <http://ankitasblogger.blogspot.com/2011/05/installing-flume-in-cluster-complete.html>
> Installing Flume in the cluster - A complete step by step ...
> <http://ankitasblogger.blogspot.com/2011/05/installing-flume-in-cluster-complete.html>
> ankitasblogger.blogspot.com
> Before we start configure flume, you need to have a running Hadoop
> cluster, which will be the centralize storage for flume. Please refer to
> Installing ...
>
>
> Hoping this helps
>
>
>
> ------------------------------
> *From:* jeff saremi <jeffsaremi@hotmail.com>
> *Sent:* Friday, April 7, 2017 1:15 PM
> *To:* user@flume.apache.org
> *Subject:* Flume Cluster?
>
>
> Is there such a thing as a flume cluster? If there is I can't find any
> documentation for it.
> If there is not, how does flume deal with scale and availability?
>
> thanks
>
> Jeff
>
>
>

Mime
View raw message