flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yogendra reddy <yogendra...@gmail.com>
Subject Re: Flume log4j Appender issue
Date Tue, 01 Dec 2015 10:13:41 GMT
update

I ran the flume agent first and then made changes to hadoop log4j
properties file and after the restart this started working fine. but now
Hive service is not coming up because of avro-ipc jar that I had to add to
hadoop-hdfs lib to get flume log4j appender working

I would like to know if anybody here has used flume to copy Hadoop
daemon/service logs?

Thanks,
Yogendra

On Wed, Nov 25, 2015 at 2:03 PM, yogendra reddy <yogendra.60@gmail.com>
wrote:

> Hi All,
>
> I'm trying to configure flume to write hadoop service logs to a common
> sink.
>
> Here's what I have added to hdfs log4j.properties
>
> # Define the root logger to the system property "hadoop.root.logger".
> log4j.rootLogger=${hadoop.root.logger}, flume
>
> #Flume Appender
> log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppender
> log4j.appender.flume.Hostname = localhost
> log4j.appender.flume.Port = 41414
>
> and when I run sample pi job I get this error
>
> $ hadoop jar hadoop-mapreduce-examples.jar pi 10 10
> log4j:ERROR Could not find value for key log4j.appender.flume.layout
> 15/11/25 07:23:26 WARN api.NettyAvroRpcClient: Using default maxIOWorkers
> log4j:ERROR RPC client creation failed! NettyAvroRpcClient { host:
> localhost, port: 41414 }: RPC connection error
> Exception in thread "main" java.lang.ExceptionInInitializerError
>         at org.apache.hadoop.util.RunJar.run(RunJar.java:200)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: org.apache.commons.logging.LogConfigurationException:
> User-specified log class 'org.apache.commons.logging.impl.Log4JLogger'
> cannot be found or is not useable.
>         at
> org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:804)
>         at
> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541)
>         at
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292)
>         at
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269)
>         at
> org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
>         at
> org.apache.hadoop.util.ShutdownHookManager.<clinit>(ShutdownHookManager.java:44)
>         ... 2 more
>
> I have added these jars to hadoop-hdfs lib -
>
> avro-ipc-1.7.3.jar
> flume-avro-source-1.5.2.2.2.7.1-33.jar
> flume-ng-log4jappender-1.5.2.2.2.7.1-33.jar
> flume-hdfs-sink-1.5.2.2.2.7.1-33.jar
> flume-ng-sdk-1.5.2.2.2.7.1-33.jar
>
> and I do have the commons-logging( commons-logging-1.1.3.jar) and
> log4j(1.2.17) jars present in the hdfs lib. Any pointers to debug this
> issue?
>
> Thanks,
> Yogendra
>
>
>

Mime
View raw message