flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From NGuyen thi Kim Tuyen <tuyen03a...@gmail.com>
Subject Re: Flume-NG 1.3.1 : Spooling dir source : java.io.IOException: Stream closed
Date Mon, 28 Jan 2013 08:02:18 GMT
Thank for your reply .

If spooling source only works on "done" , immutable files , it's not
suitable my problem . I think I 'll use exec tail command instead . But
warning from http://flume.apache.org/FlumeUserGuide.html#exec-source  : The
problem with ExecSource and other asynchronous sources is that the source
can not guarantee that if there is a failure to put the event into the
Channel the client knows about it. ..... For stronger reliability
guarantees, consider the Spooling Directory Source or direct integration
with Flume via the SDK.

I'm still considering between ExecSource and Log4jAppender .
http://www.slideshare.net/sematext/search-analytics-with-flume-and-hbase


Could you share me your opinion ?

On Mon, Jan 28, 2013 at 2:29 PM, Mike Percy <mpercy@cloudera.com> wrote:

> Hi Nguyễn,
> The spooling source only works on "done", immutable files. So they have to
> be atomically moved and they cannot be modified after being placed into the
> spooling directory.
>
> Regards,
> Mike
>
>
> On Sun, Jan 27, 2013 at 11:14 PM, NGuyen thi Kim Tuyen <
> tuyen03a128@gmail.com> wrote:
>
>> Hi ,
>>
>> Please help me .
>>
>> I want to use Flume in the following case :
>> Spooling directory source --> FileChannel --> HBase sink . But I have
>> some problems with Spooling directory source :
>>
>> Here is my test flume.conf :
>> t-game-db194.sources = test-hbase
>>
>> t-game-db194.sinks = sink-hbase
>>
>> t-game-db194.channels = hbase-channel
>>
>> #source spoolDir
>> t-game-db194.sources.test-hbase.type = spooldir
>>
>> t-game-db194.sources.test-hbase.spoolDir =/var/log/testhbase
>>
>> t-game-db194.sources.test-hbase.fileHeader = true
>>
>> t-game-db194.sources.test-hbase.channels = hbase-channel
>>
>> #file Channel
>> t-game-db194.channels.hbase-channel.type = file
>>
>> t-game-db194.channels.hbase-channel.checkpointDir =
>> /var/log/flume-ng/checkpoint
>>
>> t-game-db194.channels.hbase-channel.dataDir = /var/log/flume-ng/filedata
>>
>>
>> #sink
>> t-game-db194.sinks.sink-hbase.type = logger
>>
>> t-game-db194.sinks.sink-hbase.channel = hbase-channel
>>
>> And I tested : echo "tuyen ssssssssss " >>
>> "/var/log/testhbase/hbase_1.log" . The first event is OK , but the next
>> events are not work . Here is flume.log
>>
>> 28 Jan 2013 13:16:47,424 INFO  [lifecycleSupervisor-1-0]
>> (org.apache.flume.source.SpoolDirectorySource.start:64)  -
>> SpoolDirectorySource source starting with directory:/var/log/testhbase
>> 28 Jan 2013 13:16:47,732 INFO  [pool-7-thread-1]
>> (org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile:229)
>>  - Preparing to move file /var/log/testhbase/hbase_1.log to
>> /var/log/testhbase/hbase_1.log.COMPLETED
>> 28 Jan 2013 13:16:48,436 INFO
>>  [SinkRunner-PollingRunner-DefaultSinkProcessor]
>> (org.apache.flume.sink.LoggerSink.process:70)  - Event: {
>> headers:{file=/var/log/testhbase/hbase_1.log} body: 74 75 79 65 6E 20 73 73
>> 73 73 73 73 73 73 73 73 tuyen ssssssssss }
>>
>> 28 Jan 2013 13:17:08,836 INFO  [pool-7-thread-1]
>> (org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile:229)
>>  - Preparing to move file /var/log/testhbase/hbase_1.log to
>> /var/log/testhbase/hbase_1.log.COMPLETED
>> 28 Jan 2013 13:17:08,837 ERROR [pool-7-thread-1]
>> (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148)
>>  - Uncaught exception in Runnable
>> java.lang.IllegalStateException: File name has been re-used with
>> different files. Spooling assumption violated for
>> /var/log/testhbase/hbase_1.log.COMPLETED
>>  at
>> org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile(SpoolingFileLineReader.java:272)
>>  at
>> org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:185)
>>  at
>> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
>>  at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>  at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>  at java.lang.Thread.run(Thread.java:662)
>> 28 Jan 2013 13:17:09,340 ERROR [pool-7-thread-1]
>> (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148)
>>  - Uncaught exception in Runnable
>> java.io.IOException: Stream closed
>> at java.io.BufferedReader.ensureOpen(BufferedReader.java:97)
>>  at java.io.BufferedReader.readLine(BufferedReader.java:292)
>>  at java.io.BufferedReader.readLine(BufferedReader.java:362)
>>  at
>> org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:180)
>>  at
>> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
>>  at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>  at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>  at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>
>>
>> Are there more documents about Flume-ng spooling source  , beside
>> http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source ?
>>
>> Could you please give me some advice ?
>>
>> --
>> Nguyễn Thị Kim Tuyên
>> Computer Science Engineering
>> HCMC University Of Technology.
>
>
>


-- 
Nguyễn Thị Kim Tuyên
Computer Science Engineering
HCMC University Of Technology.

Mime
View raw message