flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Smaine Kahlouch <smaine.kahlo...@smartjog.com>
Subject Re: Spreading multiple file-channels over distinct disks
Date Wed, 18 Feb 2015 17:00:09 GMT
Hello,

I'm not sure how to perform this load balancing, could you please give 
me an example ?

Regards,
Smaine

On 02/18/15 17:39, Smaine KAHLOUCH wrote:
>
> Thank you Ahmed, i'll try that
>
>
> Regards,
>
> Smaine
>
> ------------------------------------------------------------------------
> *From:* Ahmed Vila <avila@devlogic.eu>
> *Sent:* Wednesday, February 18, 2015 5:33 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Spreading multiple file-channels over distinct disks
> Hi Smaine,
>
> There is always an overhead of running multiple flume instances, i.e. 
> multiple Java VMs. Thus, I wouldn't recommend running more of them if 
> that's not due to distribution across servers.
>
> But, you are able to do round-robin events from source into multiple 
> channels and those channels could have one disk per channel.
> However, in that case you'll also need at least one sink per channel. 
> Guess that's even desirable if you're sinking your data into the HDFS.
>
> Regards
>
> On Wed, Feb 18, 2015 at 10:14 AM, Smaine Kahlouch 
> <smaine.kahlouch@smartjog.com <mailto:smaine.kahlouch@smartjog.com>> 
> wrote:
>
>     Hi all!
>
>     I would like to know what is the best way to spread the i/o on my
>     server.
>     Actually it has several disks and i was wondering how to spread
>     Flume's work over them :
>     * One Flume instance with multiple working directories (dataDirs),
>     each working directory on a single disk
>     * Multiple flume instances with a file channel on a distinct disk
>     (using containers would be the best way to do so)
>
>     Could you please helpe me to find out the best architecture which
>     allows us to keep file-channel for data persistency ?
>
>     Regards,
>
>     -- 
>     Smaine Kahlouch - Engineer, Research & Engineering
>     Arkena | T: +33 1 5868 6196 <tel:%2B33%201%205868%206196>
>     27 Blvd Hippolyte Marquès, 94200 Ivry-sur-Seine, France
>     arkena.com <http://arkena.com>
>
>
>
>
> -- 
>
> Best regards,
>
> Ahmed Vila | Senior software developer
> DevLogic | Sarajevo | Bosnia and Herzegovina
>
> Office : +387 33 942 123
> Mobile: +387 62 139 348
>
> Website: www.devlogic.eu <http://www.devlogic.eu>
> E-mail   : avila@devlogic.eu <mailto:avila@devlogic.eu>
> ---------------------------------------------------------------------
> This e-mail and any attachment is for authorised use by the intended 
> recipient(s) only. This email contains confidential information. It 
> should not be copied, disclosed to, retained or used by, any party 
> other than the intended recipient. Any unauthorised distribution, 
> dissemination or copying of this E-mail or its attachments, and/or any 
> use of any information contained in them, is strictly prohibited and 
> may be illegal. If you are not an intended recipient then please 
> promptly delete this e-mail and any attachment and all copies and 
> inform the sender directly via email. Any emails that you send to us 
> may be monitored by systems or persons other than the named 
> communicant for the purposes of ascertaining whether the communication 
> complies with the law and company policies.
>
> ---------------------------------------------------------------------
> This e-mail and any attachment is for authorised use by the intended 
> recipient(s) only. This email contains confidential information. It 
> should not be copied, disclosed to, retained or used by, any party 
> other than the intended recipient. Any unauthorised distribution, 
> dissemination or copying of this E-mail or its attachments, and/or any 
> use of any information contained in them, is strictly prohibited and 
> may be illegal. If you are not an intended recipient then please 
> promptly delete this e-mail and any attachment and all copies and 
> inform the sender directly via email. Any emails that you send to us 
> may be monitored by systems or persons other than the named 
> communicant for the purposes of ascertaining whether the communication 
> complies with the law and company policies.

-- 
Smaine Kahlouch - Engineer, Research & Engineering
Arkena | T: +33 1 5868 6196
27 Blvd Hippolyte Marquès, 94200 Ivry-sur-Seine, France
arkena.com


Mime
View raw message