Ah, yes, I remember that patch. It basically copied the Writables interfaces/classes I think. Yeah, copying those would be a nice workaround (I think those dont have any implementation anyway).


Hari

-- 
Hari Shreedharan

On Wednesday, April 10, 2013 at 1:44 PM, Brock Noland wrote:

I think we just need to copy 4-5 classes over from hadoop. Someone uploaded a patch I just haven't had time to work with it.


On Wed, Apr 10, 2013 at 3:41 PM, Hari Shreedharan <hshreedharan@cloudera.com> wrote:
Yes. We will need to remove the format upgrade code and the old format code from the file channel as well. I think it should be possible to do it, but I suspect it is just that no one had the time to look at this as yet.

-- 
Hari Shreedharan

On Wednesday, April 10, 2013 at 1:38 PM, Roshan Naik wrote:

Hari,
  i recall some discussion around the desire to get rid of this dependency. i guess it means removing that piece of code that supports the format upgrade ? are there any plans to do that ?
-roshan


On Wed, Apr 10, 2013 at 1:25 PM, Paul Chavez <pchavez@verticalsearchworks.com> wrote:
I did not realize I could use pre-built binaries on Windows. Thank you, I used this to grab the hadoop-common.jar from our existing Hadoop cluster and there are no startup errors now.
 
Thanks to you, Brock and Hari.
 
-Paul


From: Israel Ekpo [mailto:israel@aicer.org]
Sent: Wednesday, April 10, 2013 12:07 PM
To: user@flume.apache.org
Subject: Re: FileChannel on Windows

Paul,

If Flume 1.3.1 is what you are looking for, you don't have to build it from source.

You can just download it directly from the site. Its already released.

Paul Chavez <pchavez@verticalsearchworks.com> wrote:

Hello,
 
I've been using flume on Windows to send logs to an avro source for awhile now, using the AvroClient class. I'm now trying to get a full agent running to hopefully leverage the spoolDir and httpSource functionality. Part of my requirements is a persistent channel so I'm attempting to get a config running the utilizes the fileChannel.
 
I followed the instructions at http://mapredit.blogspot.com/2012/07/run-flume-13x-on-windows.html to build flume, same as I did for the AvroClient. However it seems I need some HDFS libraries. Can anyone point me to instructions for building the JAR(s) I need to get this working?
 
Currently I'm running into the following exception:
10 Apr 2013 11:50:29,317 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.start:285)  - Starting FileChannel fc_Default { dataDirs: [c:\flume_data\log] }...
10 Apr 2013 11:50:29,317 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.start:314)  - Failed to start the file channel [channel=fc_Default]
java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
 at java.lang.ClassLoader.defineClass1(Native Method)
 at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
 at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
 at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
 at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
 at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
 at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
 at org.apache.flume.channel.file.Log$Builder.build(Log.java:212)
 at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
 at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
 at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable
 at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
 ... 24 more
10 Apr 2013 11:50:29,317 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:238)  - Unable to start FileChannel fc_Default { dataDirs: [c:\flume_data\log] } - Exception follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
 at java.lang.ClassLoader.defineClass1(Native Method)
 at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
 at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
 at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
 at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
 at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
 at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
 at org.apache.flume.channel.file.Log$Builder.build(Log.java:212)
 at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
 at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
 at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable
 at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
 ... 24 more
10 Apr 2013 11:50:29,317 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.stop:329)  - Stopping FileChannel fc_Default { dataDirs: [c:\flume_data\log] }...
 
 
 
Thank you,
Paul Chavez





--
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org