flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Jansen" <chris.jan...@cognitomobile.com>
Subject Log4j Appender Pattern Layout
Date Fri, 12 Aug 2011 14:26:37 GMT
Hello all,


I've spent a few hours today playing with the new Log4j avro appender,
thanks to Jonathan et al for getting that in the latest release. It's
seems to work very reliably, however it only seems to take the message
body of a log4j event and ignores the other elements. This is a bit of a
problem for me as our software tracing relies on displaying the class,
thread, MDC and stacktrace entries for debugging purposes. Essentially
flume is ignoring the log4j pattern. 


I had a look in the source and found that the Log4JEventAdaptor class
and found that only the message of the event was being used, and the
layout ignored. So I created a new constructor to allow me to pass in a
Log4j layout to the event adaptor, I also modified the getBody method to
use the layout if the layout object is not null and I added some code to
print the stacktrace if it is present. See my hack below:


    public Log4JEventAdaptor(LoggingEvent evt, Layout layout) {


        this.evt = evt;

        this.layout = layout;

        // This is needed to differentiate between events at the same

        this.nanos = System.nanoTime();



    public byte[] getBody() {

        if (layout == null) {

            return evt.getRenderedMessage().getBytes();

        } else {


            String body = layout.format(evt);


            String[] s = evt.getThrowableStrRep();

            if (s != null) {

                int len = s.length;

                for (int i = 0; i < len; i++) {

                    body = body + s[i];

                    body = body + Layout.LINE_SEP;




            return body.getBytes();




So now I get formatted log4j entries in flume! My question is; was there
a reason for ignoring the log4j layout in the event adaptor and might
what I have done be useful to anyone else?


Also stack traces are appearing all on one line with the line separator
escape character instead of a new line. From what I have read the tail
source can cope with stacktraces and make them into a single event, is
it possible to do this with the log4j/avro source?






NOTICE: Cognito Limited. Benham Valence, Newbury, Berkshire, RG20 8LU.  UK. Company number
02723032.  This e-mail message and any attachment is confidential. It may not be disclosed
to or used by anyone other than the intended recipient. If you have received this e-mail in
error please notify the sender immediately then delete it from your system. Whilst every effort
has been made to check this mail is virus free we accept no responsibility for software viruses
and you should check for viruses before opening any attachments. Opinions, conclusions and
other information in this email and any attachments which do not relate to the official business
of the company are neither given by the company nor endorsed by it.

This email message has been scanned for viruses by Mimecast
View raw message