flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dongliang Sun <davidsu...@gmail.com>
Subject Re: Could not instantiate class org.apache.flume.clients.log4jappender
Date Mon, 14 Jan 2013 06:24:15 GMT
It does not work, -Dpig.additional.jars={REALPATH}/lib/*.

Also tried Chhaya said, I put all the libraries from FLUME_HOME/lib/* to
PIG_HOME/lib/, got following error:

13/01/14 14:19:03 WARN api.NettyAvroRpcClient: Invalid value for batchSize:
0; Using default value.
Exception in thread "main" java.lang.NoSuchMethodError:
org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
    at
org.apache.commons.logging.impl.SLF4JLocationAwareLog.error(SLF4JLocationAwareLog.java:211)
    at org.apache.pig.impl.util.LogUtils.writeLog(LogUtils.java:125)
    at org.apache.pig.impl.util.LogUtils.writeLog(LogUtils.java:79)
    at org.apache.pig.Main.run(Main.java:589)
    at org.apache.pig.Main.main(Main.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)


2013/1/14 Connor Woodson <cwoodson.dev@gmail.com>

> Alright so it is just before you weren't adding them properly to the
> classpath. I am not quite sure how to do that; a look at Pig's run script
> (bin/pig) might yield insight into an option to add them. Putting the files
> in pig's lib's folder almost works; the issue is those libraries have more
> dependencies that aren't in the classpath yet. Looking at their
> (log4jappender and sdk) poms could tell you exactly which libs you need to
> include, but let's not do that.
>
> I mistyped the -D command from before. Try this:
>
> -Dpig.additional.jars=${FLUME_HOME}/lib/*
>
> FLUME_HOME obviously needs to be declared beforehand, or just replace that
> with the real path (and make sure the log4jappender and sdk jars, and all
> other jars, are in the libs folder)
>
>
> On Sun, Jan 13, 2013 at 10:05 PM, Dongliang Sun <davidsundl@gmail.com>wrote:
>
>> REGISTER and -Dpig.additional.libraries both do not work, same error,
>> could not find the class.
>>
>> If put the particular jars in pig's lib, the following error:
>> Error before Pig is launched
>> ----------------------------
>> ERROR 2998: Unhandled internal error.
>> org/jboss/netty/channel/ChannelFactory
>>
>> java.lang.NoClassDefFoundError: org/jboss/netty/channel/ChannelFactory
>>         at
>> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:168)
>>         at
>> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:128)
>>         at
>> org.apache.flume.clients.log4jappender.Log4jAppender.activateOptions(Log4jAppender.java:184)
>>         at
>> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:257)
>>         at
>> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:133)
>>         at
>> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:97)
>>         at
>> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:689)
>>
>>         at
>> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>>         at
>> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
>>         at
>> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
>>         at
>> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
>>         at org.apache.pig.Main.configureLog4J(Main.java:678)
>>         at org.apache.pig.Main.run(Main.java:337)
>>         at org.apache.pig.Main.main(Main.java:111)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> Caused by: java.lang.ClassNotFoundException:
>> org.jboss.netty.channel.ChannelFactory
>>
>>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>         at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>         ... 19 more
>>
>>
>> 2013/1/14 Connor Woodson <cwoodson.dev@gmail.com>
>>
>>> You could try calling 'REGISTER <log4jappender jar>' at the start of
>>> your script, or on the command line by adding
>>> -Dpig.additional.libraries=<log4jappender jar> to your Pig call. I don't
>>> know if that will work or not though.
>>>
>>> You could try placing it in pig's lib folder.
>>>
>>> You also need to include the flume-ng-sdk jar the same way as the
>>> log4jappender one, as the sdk is a dependency of the appender.
>>>
>>>
>>> On Sun, Jan 13, 2013 at 9:24 PM, Dongliang Sun <davidsundl@gmail.com>wrote:
>>>
>>>> Actually I have append the FLUME_HOME/lib/ to my CLASSPATH.
>>>> Whether there is some problem in my PIG log4j.properties?
>>>> Do you have any experience using log4jappender on PIG?
>>>>
>>>> Thanks,
>>>> Dongliang
>>>>
>>>>
>>>>
>>>> 2013/1/14 Chhaya Vishwakarma <Chhaya.Vishwakarma@lntinfotech.com>
>>>>
>>>>>  Put flume-log4j-appender<https://repository.cloudera.com/content/repositories/releases/com/cloudera/flume-log4j-appender/0.9.4-cdh3u1/flume-log4j-appender-0.9.4-cdh3u1.jar>.jar
>>>>> in CLASSPATH. Jar can be found in FLUME_HOME/lib/****
>>>>>
>>>>> ** **
>>>>>
>>>>> *From:* 孙东亮 [mailto:davidsundl@gmail.com]
>>>>> *Sent:* Monday, January 14, 2013 8:55 AM
>>>>> *To:* user@flume.apache.org
>>>>> *Subject:* Could not instantiate class
>>>>> org.apache.flume.clients.log4jappender****
>>>>>
>>>>> ** **
>>>>>
>>>>> I'm a newbie for the Flume, and I just set up flume for a test.
>>>>> I want to use the log4jappender to get log info from PIG scripts, the
>>>>> log4j.properties is:****
>>>>>
>>>>> log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
 ****
>>>>>
>>>>> log4j.appender.flume.Hostname = localhost  ****
>>>>>
>>>>> log4j.appender.flume.Port = 41414   ****
>>>>>
>>>>> log4j.logger.org.apache.pig=DEBUG,flume****
>>>>>
>>>>> And for the flume.config: ****
>>>>>
>>>>> a1.sources = r1   ****
>>>>>
>>>>> a1.sinks = k1   ****
>>>>>
>>>>> a1.channels = c1  ****
>>>>>
>>>>> ** **
>>>>>
>>>>> # Describe/configure the source****
>>>>>
>>>>> a1.sources.r1.type = avro   ****
>>>>>
>>>>> a1.sources.r1.bind = localhost   ****
>>>>>
>>>>> a1.sources.r1.port = 41414   ****
>>>>>
>>>>> ** **
>>>>>
>>>>> # Describe the sink   ****
>>>>>
>>>>> a1.sinks.k1.type = logger   ****
>>>>>
>>>>> # Use a channel which buffers events in memory   ****
>>>>>
>>>>> a1.channels.c1.type = memory   ****
>>>>>
>>>>> a1.channels.c1.capacity = 1000   ****
>>>>>
>>>>> a1.channels.c1.transactionCapacity = 100   ****
>>>>>
>>>>> ** **
>>>>>
>>>>> # Bind the source and sink to the channel   ****
>>>>>
>>>>> a1.sources.r1.channels = c1   ****
>>>>>
>>>>> a1.sinks.k1.channel = c1   ****
>>>>>
>>>>> But I got the following error when run a pig script:
>>>>> java.lang.ClassNotFoundException:
>>>>> org.apache.flume.clients.log4jappender.Log4jAppender
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>> at java.lang.Class.forName0(Native Method)
>>>>> at java.lang.Class.forName(Class.java:169)
>>>>> at org.apache.log4j.helpers.Loader.loadClass(Loader.java:179)
>>>>> at
>>>>> org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:320)
>>>>> at
>>>>> org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
>>>>> at
>>>>> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
>>>>> at
>>>>> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
>>>>> at
>>>>> org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:568)
>>>>> at
>>>>> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:442)
>>>>> at
>>>>> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:367)
>>>>> at org.apache.pig.Main.configureLog4J(Main.java:678)
>>>>> at org.apache.pig.Main.run(Main.java:337)
>>>>> at org.apache.pig.Main.main(Main.java:111)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>> log4j:ERROR Could not instantiate appender named "flume".****
>>>>>
>>>>> Could you please help me find what I miss or where is incorrect for
>>>>> the configuration.****
>>>>>
>>>>> Thanks a lot!
>>>>> Dongliang****
>>>>>
>>>>> ------------------------------
>>>>> The contents of this e-mail and any attachment(s) may contain
>>>>> confidential or privileged information for the intended recipient(s).
>>>>> Unintended recipients are prohibited from taking action on the basis
of
>>>>> information in this e-mail and using or disseminating the information,
and
>>>>> must notify the sender and delete it from their system. L&T Infotech
will
>>>>> not accept responsibility or liability for the accuracy or completeness
of,
>>>>> or the presence of any virus or disabling code in this e-mail"
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message