flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bessenyei Balázs Donát <bes...@apache.org>
Subject Re: Use case for Flume
Date Tue, 05 Sep 2017 15:42:56 GMT
Hi GIntas,

Do you happen to have a config file we could examine?

Based on the scenario you have described, I'm thinking of a
"multi-agent flow" (
http://flume.apache.org/FlumeUserGuide.html#setting-multi-agent-flow )
with a Kafka source at the "start" of the first agent and a HDFS sink
at the "end" of the last agent. You can scale this system as you like
(ie. single-node to a whole cluster).
The interceptor approach sounds also possible.

Do you have any performance concerns or are you just looking for an
elegant way to implement a solution?


Thanks,

Donat


2017-09-05 14:00 GMT+02:00 Gintautas Sulskus <gintautas.sulskus@gmail.com>:
> Hi,
>
> I have a question regarding Flume suitability for a particular use case.
>
> Task: There is an incoming constant stream of links that point to files.
> Those files to be fetched and stored in HDFS.
>
> Desired implementation:
>
> 1. Each link to a file is stored in Kafka queue Q1.
> 2. Flume A1.source monitors Q1 for new links.
> 3. Upon retrieving a link from Q1, A1.source fetches the file. The file
> eventually is stored in HDFS by A1.sink
>
> My concern here is a seemingly overloaded functionality of A1.source. The
> A1.source would have to perform two activities: 1.) to periodically poll
> queue Q1 for new links to files and then  2.) fetch those files.
>
> What do you think? Is there a cleaner way to achieve this, e.g. by using an
> interceptor to fetch files? Would this be appropriate?
>
> Best,
> GIntas

Mime
View raw message