flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gabriel Eisbruch <gabrieleisbr...@gmail.com>
Subject Re: Hive Loader perfectly to use together with flume
Date Fri, 04 Nov 2011 01:21:59 GMT
Hi Migjie,
    In hive you doesn't need move the data to another site however if ypu
use hive reading a simple folder, it's imposible run querys with using only
a little part of data (for ejample if you have server logs, read only one
day or one month)  so you can use "partition" to generate "buckets". The
problem it's that flume doesn't have a "trigger" or similar to create new
partitions in hive when a file is finished (this problem is solved by my
little program).


2011/11/3 Mingjie Lai <mjlai09@gmail.com>

> Sounds interesting.
> My stupid question, do you mean loading data from hive to somewhere else?
> (I don't know much detail about hive)
> On 11/02/2011 01:42 PM, Gabriel Eisbruch wrote:
>> Hello every body,
>>     I have developed a hive loder that enable us to load hive
>> partitions base on the file name into a directoy It give us the
>> posibility of get hive date partition or other partitions kind using
>> flume (while wait for the version 9.5).
>> The project: https://github.com/geisbruch/**HiveLoader<https://github.com/geisbruch/HiveLoader>
>> Thanks,
>> Gabriel.

View raw message