flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: Unable to activate flume-ng-cassandra-sink
Date Tue, 15 Jan 2013 17:21:01 GMT
Hi,

We haven't documented it yet as it's not in any release. I've been
meaning to do that! :)

I just created https://issues.apache.org/jira/browse/FLUME-1845 for this.

Brock

On Tue, Jan 15, 2013 at 9:12 AM, Jeff Bachtel <jeff.bachtel@gmail.com> wrote:
> No, I hadn't separated the jarfiles out in that manner (I didn't realize the
> order they appeared in the CLASSPATH mattered). I'll certainly go change
> that now.
>
> Is there documentation on the plugins.d directory anywhere? I discovered it
> in the code by accident, and it met my needs while using this custom sink.
>
> Jeff
>
>
> On Mon, Jan 14, 2013 at 9:23 PM, Brock Noland <brock@cloudera.com> wrote:
>>
>> Great to hear the plugin directory is being used. Just to confirm, do
>> you have the following:
>>
>> /usr/local/flume/plugins.d/flume-ng-cassandra-sink/lib/
>>
>> correct?
>>
>> Generally the plugin.d was created so this:
>>
>> /usr/local/flume/plugins.d/plugin/lib/
>>
>> would have the plugin jar and this:
>>
>> /usr/local/flume/plugins.d/plugin/libext/
>>
>> would have any jars the plugin depended on.
>>
>> Brock
>>
>> On Mon, Jan 14, 2013 at 6:17 PM, Connor Woodson <cwoodson.dev@gmail.com>
>> wrote:
>> > The reason I don't recommend using it is that I didn't know it existed
>> > :P
>> >
>> > That seems like a great place to put it. I've done quite a bit of
>> > customization, so for me it's easier to just point Flume to the ./target
>> > directory of my java projects. But the plugins.d seems like a great
>> > place
>> > for third-party libraries.
>> >
>> > Glad I could help!
>> >
>> > - Connor
>> >
>> >
>> > On Mon, Jan 14, 2013 at 6:12 PM, Jeff Bachtel <jeff.bachtel@gmail.com>
>> > wrote:
>> >>
>> >> Awesome, thanks; it was that typo.
>> >>
>> >> The plugins.d directory seems to be working, is there a particular
>> >> reason
>> >> you don't recommend using? I'm brand-spanking new to Flume, so I'm
>> >> curious.
>> >>
>> >> Thanks again,
>> >>
>> >> Jeff
>> >>
>> >>
>> >> On Mon, Jan 14, 2013 at 9:06 PM, Connor Woodson
>> >> <cwoodson.dev@gmail.com>
>> >> wrote:
>> >>>
>> >>> Leave the Cassandra sink jar with its other jars in a folder some
>> >>> where.
>> >>> In FLUME_HOME/conf, open flume-env.sh and go to the FLUME_CLASSPATH
>> >>> line.
>> >>> Uncomment it if needed. Append the path to your cassandra sink and its
>> >>> associated jars, and see if that works.
>> >>>
>> >>> There doesn't appear to be a version incompatability; simply, the
>> >>> Cassandra jar is not found in the classpath. And the reason for that
>> >>> is it
>> >>> looks like you have a typo in your flume.conf.
>> >>>
>> >>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>> >>>
>> >>> should probably be
>> >>>
>> >>> com.btoddb.flume.sinks.cassandra.CassandraSink
>> >>>
>> >>> - Connor
>> >>>
>> >>>
>> >>> On Mon, Jan 14, 2013 at 5:57 PM, Jeff Bachtel <jeff.bachtel@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> I have a build of Flume 1.4.0 on FreeBSD 8, installed into
>> >>>> /usr/local/flume . I manually created
>> >>>> /usr/local/flume/plugins.d/flume-ng-cassandra-sink and untarred
>> >>>> flume-ng-cassandra-sink-1.0.0-SNAPSHOT-dist.tar.gz (from a local
>> >>>> pull/build
>> >>>> of btoddb's sources on github), so I have a lib/ directory that's
>> >>>> completely
>> >>>> populated with both the sink plugin and any possible jarfiles it
>> >>>> might rely
>> >>>> on.
>> >>>>
>> >>>> I've likewise tried just copying
>> >>>> flume-ng-cassandra-sink-1.0.0-SNAPSHOT.jar to flume/lib/ for it
to
>> >>>> get
>> >>>> picked up.
>> >>>>
>> >>>> When I have a CassandraSink define in flume.conf as such:
>> >>>> webserver.sinks.cassandraSink.type =
>> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>> >>>> webserver.sinks.cassandraSink.channel = memoryChannel
>> >>>> webserver.sinks.cassandraSink.hosts = localhost
>> >>>> webserver.sinks.cassandraSink.cluster-name = MyCluster
>> >>>> webserver.sinks.cassandraSink.keyspace-name = ApacheLogs
>> >>>> webserver.sinks.cassandraSink.records-coldfam = Requests
>> >>>>
>> >>>> and try to start Flume with:
>> >>>>
>> >>>> /usr/local/flume/bin/flume-ng agent -c /usr/local/flume/conf/ -f
>> >>>> /usr/local/flume/conf/flume.conf -n webserver
>> >>>> -Dflume.root.logger=DEBUG,console
>> >>>>
>> >>>>
>> >>>> I get the following output (and obviously, no success):
>> >>>> 2013-01-14 19:55:33,594 (conf-file-poller-0) [ERROR -
>> >>>>
>> >>>> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:142)]
>> >>>> Failed to load configuration data. Exception follows.
>> >>>> org.apache.flume.FlumeException: Unable to load sink type:
>> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink, class:
>> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>> >>>>         at
>> >>>>
>> >>>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:69)
>> >>>>         at
>> >>>>
>> >>>> org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
>> >>>>         at
>> >>>>
>> >>>> org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
>> >>>>         at
>> >>>>
>> >>>> org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
>> >>>>         at
>> >>>>
>> >>>> org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
>> >>>>         at
>> >>>>
>> >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> >>>>         at
>> >>>>
>> >>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>> >>>>         at
>> >>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>> >>>>         at
>> >>>>
>> >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>> >>>>         at
>> >>>>
>> >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> >>>>         at
>> >>>>
>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >>>>         at
>> >>>>
>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >>>>         at java.lang.Thread.run(Thread.java:722)
>> >>>> Caused by: java.lang.ClassNotFoundException:
>> >>>> com.tbtoddb.flume.sinks.cassandra.CassandraSink
>> >>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> >>>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> >>>>         at java.security.AccessController.doPrivileged(Native Method)
>> >>>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> >>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
>> >>>>         at
>> >>>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> >>>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>> >>>>         at java.lang.Class.forName0(Native Method)
>> >>>>         at java.lang.Class.forName(Class.java:186)
>> >>>>         at
>> >>>>
>> >>>> org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
>> >>>>         ... 12 more
>> >>>>
>> >>>>
>> >>>> does anyone have any idea why this plugin isn't being picked up
by
>> >>>> Flume? Is it an incompatibility with the plugin expecting to run
>> >>>> under Flume
>> >>>> 1.3.1, perhaps? I'm at a loss on debugging this.
>> >>>>
>> >>>> Thanks,
>> >>>> Jeff
>> >>>
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>
>



-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Mime
View raw message