flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henry Ma <henry.ma.1...@gmail.com>
Subject Can we treat a whole file as a Flume event?
Date Tue, 22 Jan 2013 01:45:48 GMT

When using Flume to collect log files, we want to just COPY the original
files from several servers to a central storage (unix file system), not to
roll up to a big file. Because we must record some messages of the original
file such as name, host, path, timestamp, etc. Besides, we want to
guarantee total reliability: no file miss, no file reduplicated.

It seems that, in Source, we must put a whole file (size may be between
100KB and 100MB) into a Flume event; and in Sink, we must write each event
to a single file.

Is it practicable? Thanks!

Best Regards,
Henry Ma

View raw message