flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yogendra reddy <yogendra...@gmail.com>
Subject Re: Flume log4j Appender issue
Date Tue, 01 Dec 2015 10:48:55 GMT
I din't follow. I'm adding flume libraries to hadoop classpath i.e
hadoop-hdfs lib folder and this is causing the issue. I need these jars to
be in hdfs lib as I have added log4j appender to hdfs log4j properties.

On Tue, Dec 1, 2015 at 4:09 PM, Gonzalo Herreros <gherreros@gmail.com>
wrote:

> Adding a library to Flume shouldn't affect hive or any other tools.
> You can add the jar to the lib or plugin.d directories.
>
> Regards,
> Gonzalo
>
> On 1 December 2015 at 10:13, yogendra reddy <yogendra.60@gmail.com> wrote:
>
>> update
>>
>> I ran the flume agent first and then made changes to hadoop log4j
>> properties file and after the restart this started working fine. but now
>> Hive service is not coming up because of avro-ipc jar that I had to add to
>> hadoop-hdfs lib to get flume log4j appender working
>>
>> I would like to know if anybody here has used flume to copy Hadoop
>> daemon/service logs?
>>
>> Thanks,
>> Yogendra
>>
>> On Wed, Nov 25, 2015 at 2:03 PM, yogendra reddy <yogendra.60@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> I'm trying to configure flume to write hadoop service logs to a common
>>> sink.
>>>
>>> Here's what I have added to hdfs log4j.properties
>>>
>>> # Define the root logger to the system property "hadoop.root.logger".
>>> log4j.rootLogger=${hadoop.root.logger}, flume
>>>
>>> #Flume Appender
>>> log4j.appender.flume =
>>> org.apache.flume.clients.log4jappender.Log4jAppender
>>> log4j.appender.flume.Hostname = localhost
>>> log4j.appender.flume.Port = 41414
>>>
>>> and when I run sample pi job I get this error
>>>
>>> $ hadoop jar hadoop-mapreduce-examples.jar pi 10 10
>>> log4j:ERROR Could not find value for key log4j.appender.flume.layout
>>> 15/11/25 07:23:26 WARN api.NettyAvroRpcClient: Using default maxIOWorkers
>>> log4j:ERROR RPC client creation failed! NettyAvroRpcClient { host:
>>> localhost, port: 41414 }: RPC connection error
>>> Exception in thread "main" java.lang.ExceptionInInitializerError
>>>         at org.apache.hadoop.util.RunJar.run(RunJar.java:200)
>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>>> Caused by: org.apache.commons.logging.LogConfigurationException:
>>> User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
>>> cannot be found or is not useable.
>>>         at
>>> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:804)
>>>         at
>>> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541)
>>>         at
>>> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292)
>>>         at
>>> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269)
>>>         at
>>> org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
>>>         at
>>> org.apache.hadoop.util.ShutdownHookManager.<clinit>(ShutdownHookManager.java:44)
>>>         ... 2 more
>>>
>>> I have added these jars to hadoop-hdfs lib -
>>>
>>> avro-ipc-1.7.3.jar
>>> flume-avro-source-1.5.2.2.2.7.1-33.jar
>>> flume-ng-log4jappender-1.5.2.2.2.7.1-33.jar
>>> flume-hdfs-sink-1.5.2.2.2.7.1-33.jar
>>> flume-ng-sdk-1.5.2.2.2.7.1-33.jar
>>>
>>> and I do have the commons-logging( commons-logging-1.1.3.jar) and
>>> log4j(1.2.17) jars present in the hdfs lib. Any pointers to debug this
>>> issue?
>>>
>>> Thanks,
>>> Yogendra
>>>
>>>
>>>
>>
>

Mime
View raw message