flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: .SpoolingFileLineReader warning....
Date Sat, 17 Nov 2012 16:15:10 GMT
Ok, do you mind sharing your log rotate config to see if we can reproduce? 

-- 
Brock Noland
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)


On Saturday, November 17, 2012 at 10:01 AM, Dan Young wrote:

> Hey Brock,
> No I have not modified the conf while the agent was running. 
> /mnt/flume is local. Note that this is running on an ec2 instance and the disk is the
ephemeral drive, not EBS.
> Regards ,
> Dano
> On Nov 17, 2012 8:58 AM, "Brock Noland" <brock@cloudera.com (mailto:brock@cloudera.com)>
wrote:
> > Hi,
> > 
> > I highly doubt it's related to
> > (https://issues.apache.org/jira/browse/FLUME-1721) but have you
> > modified the configuration file since starting the agent?  If so, can
> > you restart the agent and see if the error continues?
> > 
> > Also, is /mnt/flume local disk or NAS?
> > 
> > Brock
> > 
> > On Sat, Nov 17, 2012 at 9:02 AM, Dan Young <danoyoung@gmail.com (mailto:danoyoung@gmail.com)>
wrote:
> > > First a bit of context, I'm using logrotate to monitor and copy (cp -p) log
> > > files to a flume spooling directory source.  So every hour, logrotate checks
> > > for and copies a file from the source to the flume destination. I see the
> > > following warning message in the flume logs.
> > >
> > >
> > > 17 Nov 2012 14:47:07,682 WARN  [pool-10-thread-1]
> > > (org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile:328)  -
> > > Could not find file:
> > > /mnt/flume/clickstream/clickstream.log-2012-11-17-1353163623
> > > java.io.FileNotFoundException:
> > > /mnt/flume/clickstream/clickstream.log-2012-11-17-1353163623 (Permission
> > > denied)
> > > at java.io.FileInputStream.open(Native Method)
> > > at java.io.FileInputStream.<init>(FileInputStream.java:138)
> > > at java.io.FileReader.<init>(FileReader.java:72)
> > > at
> > > org.apache.flume.client.avro.SpoolingFileLineReader.getNextFile(SpoolingFileLineReader.java:322)
> > > at
> > > org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:172)
> > > at
> > > org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
> > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > at
> > > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> > > at
> > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > > at
> > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> > > at java.lang.Thread.run(Thread.java:722)
> > >
> > >
> > > Although it appears that Flume processes the log, I'm curious why I''m
> > > seeing this and if I have anything with permissions incorrect?
> > >
> > >
> > >
> > > Here's the permissions:
> > >
> > > source log directory under /var/log:
> > > drwxrwxr-x 2 ubuntu    ubuntu   4096 Nov 17 14:47 clickstream
> > >
> > > source files:
> > > -rw-rw-r-- 1 ubuntu ubuntu   9055750 Nov 17 13:29
> > > clickstream.log-2012-11-17-1353158953.gz
> > > -rw-rw-r-- 1 ubuntu ubuntu  13583565 Nov 17 14:17
> > > clickstream.log-2012-11-17-1353161821.gz
> > > -rw-rw-r-- 1 ubuntu ubuntu 131296672 Nov 17 14:47
> > > clickstream.log-2012-11-17-1353163623
> > > -rw-rw-r-- 1 ubuntu ubuntu  65648336 Nov 17 14:52 clickstream.log
> > >
> > > flume source directory under /mnt/flume:
> > > drwxrwxr-x 2 ubuntu ubuntu 4096 Nov 17 14:48 clickstream
> > >
> > > flume source files:
> > > -rw-rw-r-- 1 ubuntu ubuntu 131296672 Nov 17 13:29
> > > clickstream.log-2012-11-17-1353158953.COMPLETED
> > > -rw-rw-r-- 1 ubuntu ubuntu 196945008 Nov 17 14:17
> > > clickstream.log-2012-11-17-1353161821.COMPLETED
> > > -rw-rw-r-- 1 ubuntu ubuntu 131296672 Nov 17 14:47
> > > clickstream.log-2012-11-17-1353163623.COMPLETED
> > >
> > > Any insight would be appreciated.
> > >
> > > Regards,
> > >
> > > Dan
> > 
> > 
> > 
> > --
> > Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/


Mime
View raw message