flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ehsan ul Haq <m.ehsan....@gmail.com>
Subject Re: Is it possible for Flume-NG to transfer binary files?
Date Fri, 11 Apr 2014 08:27:24 GMT
Yes it is possible to transfer binary files into Hadoop (HDFS/Hbase etc)
using flume.
If your binary files are already being stored in some local file system in
some folders then you can use SpoolingDirectory source "
https://flume.apache.org/FlumeUserGuide.html#spooling-directory-source".
The default deserializer is a LINE based text deserializer which is not for
binary data. You can use BlobDeserializer "
https://flume.apache.org/FlumeUserGuide.html#blobdeserializer" which can
handle binary data.

Cheers
Ehsan


On Fri, Apr 11, 2014 at 9:58 AM, R W <aijwjia@gmail.com> wrote:

> Hi All
>
> We have an App will generate huge amount of binary based file, it's not
> like text file which we may have new line separator for it, we want to
> collect these binary files and store them into a Hadoop cluster, I'm new to
> Flume, so could anyone share some ideas here on how to do it with Flume,
> thanks in advance.
>
> Cheers
> aij
>

Mime
View raw message