flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Lord <jl...@cloudera.com>
Subject Re: Can a Flume agent listen for a JSON/CSV file on multiple ports?
Date Wed, 11 Feb 2015 19:37:56 GMT
On Wed, Feb 11, 2015 at 10:29 AM, David Novogrodsky <
david.novogrodsky@gmail.com> wrote:

> All,
>
> I have a general question regarding Flume.
>
> We have some agents that write and send files from several host
> machines.  The agent will be installed on several hundred machines.
> The file will contain host information in a header in JSON fomat.  The
> detail records, the rest of the file, will be in CSV format.  Each
> host will be communicating through its own port.
>
> 1. Can a Flume agent listen on multiple ports?  We do not want a Flume
> agent for each port.
>

A flume agent can have multiple sources, each listening on a distinct port.
I don't think you want a 1-1 mapping here though from your application to
flume sources.



> 2. Is there an out-of-the-box Flume agent that can handle JSON and CSV
> fformat?
>

No


> 3. We can change the format of the incoming file.  Would it be simpler
> if each record was atomic and did not need header files??
>

Possibly.

Flume was designed to stream events, not necessarily files.
Does that make sense? Have you looked at the rpc client? Is that possibly
something you could leverage in your agent code? That would be the easiest
path, use the flume rpc client to generate events and pass those on to the
flume agent.

https://flume.apache.org/FlumeDeveloperGuide.html#rpc-clients-avro-and-thrift






>
> David Novogrodsky
> david.novogrodsky@gmail.com
> http://www.linkedin.com/in/davidnovogrodsky
>

Mime
View raw message