flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ZORAIDA HIDALGO SANCHEZ <zora...@tid.es>
Subject Re: Spooling fileSuffix attribute ignored
Date Wed, 22 May 2013 07:43:19 GMT
Oh! I see, then was a miss understanding. Ok, so we will need to find a workaround. Thanks
a lot.

De: Mike Percy <mpercy@apache.org<mailto:mpercy@apache.org>>
Responder a: Flume User List <user@flume.apache.org<mailto:user@flume.apache.org>>
Fecha: miércoles, 22 de mayo de 2013 09:35
Para: Flume User List <user@flume.apache.org<mailto:user@flume.apache.org>>
Asunto: Re: Spooling fileSuffix attribute ignored

Hi Phil,
Nice approach. How is the spooling directory source working for you? Any thoughts on how it
could be improved?

Mike


On Tue, May 21, 2013 at 8:17 AM, Phil Scala <Phil.Scala@globalrelay.net<mailto:Phil.Scala@globalrelay.net>>
wrote:
Hi,

Based on my use and understanding that setting “fileSuffix” is simpy the extension to
the file to be added once the file was consumed and placed onto the channel.  I don’t think
it was intended to be used to indicate a completely uploaded file.    In the dev newsgroup
there was a discussion about having the spooler ”wait” for a little while before ingesting
the file.  An ignore pattern may also be a good idea.

Currently I am using an upload directory that I monitor and when lsof reports a file is not
in use/open I then move it to my spool directory.

HTH
Phil



Phil Scala
Software Developer / Architect
Global Relay

phil.scala@globalrelay.net<mailto:phil.scala@globalrelay.net>

866.484.6630<tel:866.484.6630>  |  info@globalrelay.net<https://ex1.globalrelay.net/owa/redir.aspx?C=01df0579a4cc42feb9f7e2801368f729&URL=mailto%3ainfo%40globalrelay.net>
 |  globalrelay.com<https://ex1.globalrelay.net/owa/redir.aspx?C=01df0579a4cc42feb9f7e2801368f729&URL=http%3a%2f%2fwww.globalrelay.com%2f>

From: ZORAIDA HIDALGO SANCHEZ [mailto:zoraida@tid.es<mailto:zoraida@tid.es>]
Sent: Tuesday, May 21, 2013 9:21 AM
To: Flume User List
Subject: Spooling fileSuffix attribute ignored

Dear all,
I am using the Spooling attribute "fileSuffix" to ensure that only when a file has been completely
upload into the Spooling dir,  is consumed by the source. However, files with no suffix are
also appended to the channel and then processed by the sink.
My configuration:

tier1.sources  = s1
tier1.channels = c1
tier1.sinks    = s1

# For each source, channel, and sink, set
# standard properties.
tier1.sources.s1.type     = spooldir
tier1.sources.s1.spoolDir = /home/user/flume/data
tier1.sources.s1.deletePolicy = immediate
tier1.sources.s1.batchSize = 1000
tier1.sources.s1.bufferMaxLines = 3000
tier1.sources.s1.fileHeader = true
tier1.sources.s1.fileSuffix=.COMPLETED

Is that ok?

Thanks.

________________________________

Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política
de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on
the basis of the terms set out at:
http://www.tid.es/ES/PAGINAS/disclaimer.aspx


________________________________

Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política
de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on
the basis of the terms set out at:
http://www.tid.es/ES/PAGINAS/disclaimer.aspx

Mime
View raw message