flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Lord <jl...@cloudera.com>
Subject Re: Spreading multiple file-channels over distinct disks
Date Wed, 18 Feb 2015 18:15:53 GMT
+1 to a single flume instance/jvm per machine
You can leverage multiple spindles/drives by dedicating each to a separate
file channel. Ideally you would have a separate spindle for the logs as
well as the file channel itself. In order to spread the load across these
channels you would need to do one of three things (that i can think of)

1. Use multiple sources; so 1 source -> 1 channel -> 1 or more sinks
2. Use the mutiplexing channel selector in combination with an interceptor
that labels event headers and does some form of contextual routing.
3. Create a custom channel selector. The ootb channel selectors are
replicating (default) and multiplex

https://flume.apache.org/FlumeUserGuide.html#fan-out-flow

On Wed, Feb 18, 2015 at 9:00 AM, Smaine Kahlouch <
smaine.kahlouch@smartjog.com> wrote:

>  Hello,
>
> I'm not sure how to perform this load balancing, could you please give me
> an example ?
>
> Regards,
> Smaine
>
>
> On 02/18/15 17:39, Smaine KAHLOUCH wrote:
>
>  Thank you Ahmed, i'll try that
>
>
>  Regards,
>
> Smaine
>  ------------------------------
> *From:* Ahmed Vila <avila@devlogic.eu> <avila@devlogic.eu>
> *Sent:* Wednesday, February 18, 2015 5:33 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Spreading multiple file-channels over distinct disks
>
>  Hi Smaine,
>
>  There is always an overhead of running multiple flume instances, i.e.
> multiple Java VMs. Thus, I wouldn't recommend running more of them if
> that's not due to distribution across servers.
>
>  But, you are able to do round-robin events from source into multiple
> channels and those channels could have one disk per channel.
> However, in that case you'll also need at least one sink per channel.
> Guess that's even desirable if you're sinking your data into the HDFS.
>
>  Regards
>
> On Wed, Feb 18, 2015 at 10:14 AM, Smaine Kahlouch <
> smaine.kahlouch@smartjog.com> wrote:
>
>> Hi all!
>>
>> I would like to know what is the best way to spread the i/o on my server.
>> Actually it has several disks and i was wondering how to spread Flume's
>> work over them :
>> * One Flume instance with multiple working directories (dataDirs), each
>> working directory on a single disk
>> * Multiple flume instances with a file channel on a distinct disk (using
>> containers would be the best way to do so)
>>
>> Could you please helpe me to find out the best architecture which allows
>> us to keep file-channel for data persistency ?
>>
>> Regards,
>>
>> --
>> Smaine Kahlouch - Engineer, Research & Engineering
>> Arkena | T: +33 1 5868 6196
>> 27 Blvd Hippolyte Marquès, 94200 Ivry-sur-Seine, France
>> arkena.com
>>
>>
>
>
>  --
>
> Best regards,
> Ahmed Vila | Senior software developer
> DevLogic | Sarajevo | Bosnia and Herzegovina
>
>  Office : +387 33 942 123
> Mobile: +387 62 139 348
>
>  Website: www.devlogic.eu
> E-mail   : avila@devlogic.eu
>   ---------------------------------------------------------------------
> This e-mail and any attachment is for authorised use by the intended
> recipient(s) only. This email contains confidential information. It should
> not be copied, disclosed to, retained or used by, any party other than the
> intended recipient. Any unauthorised distribution, dissemination or copying
> of this E-mail or its attachments, and/or any use of any information
> contained in them, is strictly prohibited and may be illegal. If you are
> not an intended recipient then please promptly delete this e-mail and any
> attachment and all copies and inform the sender directly via email. Any
> emails that you send to us may be monitored by systems or persons other
> than the named communicant for the purposes of ascertaining whether the
> communication complies with the law and company policies.
>
> ---------------------------------------------------------------------
> This e-mail and any attachment is for authorised use by the intended
> recipient(s) only. This email contains confidential information. It should
> not be copied, disclosed to, retained or used by, any party other than the
> intended recipient. Any unauthorised distribution, dissemination or copying
> of this E-mail or its attachments, and/or any use of any information
> contained in them, is strictly prohibited and may be illegal. If you are
> not an intended recipient then please promptly delete this e-mail and any
> attachment and all copies and inform the sender directly via email. Any
> emails that you send to us may be monitored by systems or persons other
> than the named communicant for the purposes of ascertaining whether the
> communication complies with the law and company policies.
>
>
> --
> Smaine Kahlouch - Engineer, Research & Engineering
> Arkena | T: +33 1 5868 6196
> 27 Blvd Hippolyte Marquès, 94200 Ivry-sur-Seine, Francearkena.com
>
>

Mime
View raw message