flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ZORAIDA HIDALGO SANCHEZ <zora...@tid.es>
Subject Re: Spooling and duplicated files
Date Wed, 22 May 2013 07:47:21 GMT
One after the other but you are right, I am using  deletePolicy=immediate but I thought Flume
would keep track in memory of the processed files. Thanks.

De: Israel Ekpo <israel@aicer.org<mailto:israel@aicer.org>>
Responder a: Flume User List <user@flume.apache.org<mailto:user@flume.apache.org>>
Fecha: miércoles, 22 de mayo de 2013 08:33
Para: Flume User List <user@flume.apache.org<mailto:user@flume.apache.org>>
Asunto: Re: Spooling and duplicated files

When were these two files added?

This recommendation is made so that Flume does not attempt to process files that are currently
being written to.

It could be possible that the original file was already processed and renamed (with the suffix
added to the original name) and then a new file is added to the spooling directory after this
change.

If Flume is currently working on a file and you write to this file while things are in progress,
you will encounter an exception that will stop the agent.



On 21 May 2013 09:11, ZORAIDA HIDALGO SANCHEZ <zoraida@tid.es<mailto:zoraida@tid.es>>
wrote:
>From User Guide I read:

"This channel expects that only immutable, uniquely named files are dropped in the spooling
directory. If duplicate names are used, or files are modified while being read, the source
will fail with an error message. For some use cases this may require adding unique identifiers
(such as a timestamp) to log file names when they are copied into the spooling directory."

But I added two files with the same name and both were processed correctly. Is that ok? What
am I miss understanding?

Thanks.

________________________________

Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política
de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on
the basis of the terms set out at:
http://www.tid.es/ES/PAGINAS/disclaimer.aspx


________________________________

Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política
de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on
the basis of the terms set out at:
http://www.tid.es/ES/PAGINAS/disclaimer.aspx

Mime
View raw message