flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ehsan ul Haq <m.ehsan....@gmail.com>
Subject Re: Is it possible for Flume-NG to transfer binary files?
Date Fri, 11 Apr 2014 10:17:22 GMT
What is the use case? Where in hadoop you want to store? Where is the
source coming from is it in files?

What is the issue with BlobDeserializer?

Ehsan
On 11 Apr 2014 12:07, "R W" <aijwjia@gmail.com> wrote:

> Hi Ehsan
>
> Thanks for your quick responding, is it BlobDeserializer the only way to
> go? Thanks.
>
> Cheers
> aij
>
>
> On Fri, Apr 11, 2014 at 4:27 PM, Ehsan ul Haq <m.ehsan.haq@gmail.com>wrote:
>
>> Yes it is possible to transfer binary files into Hadoop (HDFS/Hbase etc)
>> using flume.
>> If your binary files are already being stored in some local file system
>> in some folders then you can use SpoolingDirectory source "
>> https://flume.apache.org/FlumeUserGuide.html#spooling-directory-source".
>> The default deserializer is a LINE based text deserializer which is not for
>> binary data. You can use BlobDeserializer "
>> https://flume.apache.org/FlumeUserGuide.html#blobdeserializer" which can
>> handle binary data.
>>
>> Cheers
>> Ehsan
>>
>>
>> On Fri, Apr 11, 2014 at 9:58 AM, R W <aijwjia@gmail.com> wrote:
>>
>>> Hi All
>>>
>>> We have an App will generate huge amount of binary based file, it's not
>>> like text file which we may have new line separator for it, we want to
>>> collect these binary files and store them into a Hadoop cluster, I'm new to
>>> Flume, so could anyone share some ideas here on how to do it with Flume,
>>> thanks in advance.
>>>
>>> Cheers
>>> aij
>>>
>>
>>
>

Mime
View raw message