flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Matt.Elli...@gdc4s.com>
Subject RE: Preventing Data Loss during Restart
Date Tue, 12 Feb 2013 20:19:13 GMT
Yeah I’m starting to answer my own question. We’re using 1.3 so we do have Avro. We were
trying to avoid installing anything on our client (Source) machines so that we could avoid
installing Java on machines we didn’t need it on.


From: Hari Shreedharan [mailto:hshreedharan@cloudera.com] 
Sent: Tuesday, February 12, 2013 2:42 PM
To: user@flume.apache.org
Subject: Re: Preventing Data Loss during Restart




What version of Flume are you using? Also note that Syslog is a fire and forget protocol,
so when you reconfigure, any events not persisted to the file channel would be lost. Since
there is no way of informing the data source that the data was not written to disk, this data
could in fact be lost. We recommend using a source which actually does report failure, like
Avro/Thrift (available on trunk, not in any release yet) or HTTP. This will allow you to retry
if Flume reports failure.






Hari Shreedharan


On Tuesday, February 12, 2013 at 11:24 AM, Matt.Elliott@gdc4s.com wrote:

	I’ve seen some threads on this online in the past but I can’t seem to find a distinct
answer. We’re deploying Flume in a production environment where we’re going to be grabbing
log data from syslog and other sources. While Flume supports run time configuration changes
we are still noticing data loss during testing even with a file channel. Now, this is a single
channel, source, and sink set up, no redundancy. Does anyone know of a clean way to support
guaranteed delivery without redundancy?





	This message and/or attachments may include information subject to GDC4S S.P. 1.8.6 and GD
Corporate Policy 07-105 and are intended to be accessed only by authorized recipients.  Use,
storage and transmission are governed by General Dynamics and its policies. Contractual restrictions
apply to third parties.  Recipients should refer to the policies or contract to determine
proper handling.  Unauthorized review, use, disclosure or distribution is prohibited.  If
you are not an intended recipient, please contact the sender and destroy all copies of the
original message.



View raw message