From flume-user-return-124-apmail-incubator-flume-user-archive=incubator.apache.org@incubator.apache.org Fri Aug 12 14:27:18 2011 Return-Path: X-Original-To: apmail-incubator-flume-user-archive@minotaur.apache.org Delivered-To: apmail-incubator-flume-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 872BC8EF6 for ; Fri, 12 Aug 2011 14:27:18 +0000 (UTC) Received: (qmail 41247 invoked by uid 500); 12 Aug 2011 14:27:18 -0000 Delivered-To: apmail-incubator-flume-user-archive@incubator.apache.org Received: (qmail 41151 invoked by uid 500); 12 Aug 2011 14:27:18 -0000 Mailing-List: contact flume-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: flume-user@incubator.apache.org Delivered-To: mailing list flume-user@incubator.apache.org Received: (qmail 41143 invoked by uid 99); 12 Aug 2011 14:27:17 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Aug 2011 14:27:17 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [213.235.63.77] (HELO service23.mimecast.com) (213.235.63.77) by apache.org (qpsmtpd/0.29) with SMTP; Fri, 12 Aug 2011 14:27:09 +0000 Received: from emailproxy-nby.cognito.co.uk (213.122.177.133 [213.122.177.133]) by service23.mimecast.com; Fri, 12 Aug 2011 15:26:46 +0100 Received: from localhost (localhost.localdomain [127.0.0.1]) by emailproxy-nby.cognito.co.uk (Postfix) with ESMTP id E05622D18F3 for ; Fri, 12 Aug 2011 15:26:43 +0100 (BST) Received: from seamonkey.exchange.cognito.co.uk (unknown [192.9.220.3]) by emailproxy-nby.cognito.co.uk (Postfix) with ESMTP id C98892D1503 for ; Fri, 12 Aug 2011 15:26:43 +0100 (BST) X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Subject: Log4j Appender Pattern Layout Date: Fri, 12 Aug 2011 15:26:37 +0100 Message-ID: <634087A402B83643BAD920A30B0F1D550744F88C@seamonkey.exchange.cognito.co.uk> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Log4j Appender Pattern Layout Thread-Index: AcxY+9Jmf4dS5xsqR3WprLlxCdlbWA== From: "Chris Jansen" To: X-MC-Unique: 111081215264600301 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01CC58FB.DBEF7DF5" X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. ------_=_NextPart_001_01CC58FB.DBEF7DF5 Content-Type: text/plain; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable Hello all, =20 I've spent a few hours today playing with the new Log4j avro appender, thanks to Jonathan et al for getting that in the latest release. It's seems to work very reliably, however it only seems to take the message body of a log4j event and ignores the other elements. This is a bit of a problem for me as our software tracing relies on displaying the class, thread, MDC and stacktrace entries for debugging purposes. Essentially flume is ignoring the log4j pattern.=20 =20 I had a look in the source and found that the Log4JEventAdaptor class and found that only the message of the event was being used, and the layout ignored. So I created a new constructor to allow me to pass in a Log4j layout to the event adaptor, I also modified the getBody method to use the layout if the layout object is not null and I added some code to print the stacktrace if it is present. See my hack below: =20 public Log4JEventAdaptor(LoggingEvent evt, Layout layout) { super(); this.evt =3D evt; this.layout =3D layout; // This is needed to differentiate between events at the same millisecond. this.nanos =3D System.nanoTime(); } =20 public byte[] getBody() { if (layout =3D=3D null) { return evt.getRenderedMessage().getBytes(); } else { =20 String body =3D layout.format(evt); =20 String[] s =3D evt.getThrowableStrRep(); if (s !=3D null) { int len =3D s.length; for (int i =3D 0; i < len; i++) { body =3D body + s[i]; body =3D body + Layout.LINE_SEP; } } =20 return body.getBytes(); } } =20 So now I get formatted log4j entries in flume! My question is; was there a reason for ignoring the log4j layout in the event adaptor and might what I have done be useful to anyone else? =20 Also stack traces are appearing all on one line with the line separator escape character instead of a new line. From what I have read the tail source can cope with stacktraces and make them into a single event, is it possible to do this with the log4j/avro source? =20 Thanks =20 Chris =20 NOTICE: Cognito Limited. Benham Valence, Newbury, Berkshire, RG20 8LU. UK.= Company number 02723032. This e-mail message and any attachment is confid= ential. It may not be disclosed to or used by anyone other than the intende= d recipient. If you have received this e-mail in error please notify the se= nder immediately then delete it from your system. Whilst every effort has b= een made to check this mail is virus free we accept no responsibility for s= oftware viruses and you should check for viruses before opening any attachm= ents. Opinions, conclusions and other information in this email and any att= achments which do not relate to the official business of the company are ne= ither given by the company nor endorsed by it. This email message has been scanned for viruses by Mimecast ------_=_NextPart_001_01CC58FB.DBEF7DF5 Content-Type: text/html; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable =20

Hello all,

 

I’ve= spent a few hours today playing with the new Log4j avro appender, thanks t= o Jonathan et al for getting that in the latest release. It’s seems t= o work very reliably, however it only seems to take the message body of a l= og4j event and ignores the other elements. This is a bit of a problem for m= e as our software tracing relies on displaying the class, thread, MDC and s= tacktrace entries for debugging purposes. Essentially flume is ignoring the= log4j pattern.

 

I had a look in the source and found that the Log4JEvent= Adaptor class and found that only the message of the event was being used, = and the layout ignored. So I created a new constructor to allow me to pass = in a Log4j layout to the event adaptor, I also modified the getBody method = to use the layout if the layout object is not null and I added some code to= print the stacktrace if it is present. See my hack below:

 

  &n= bsp; public Log4JEventAdaptor(LoggingEvent evt, Layout layout) {=

        super()= ;

      &n= bsp; this.evt =3D evt;

   = ;     this.layout =3D layout;

        // This is needed to d= ifferentiate between events at the same millisecond.

        this.nanos =3D Syst= em.nanoTime();

    }=

 

&n= bsp;   public byte[] getBody() {

        if (layout =3D=3D null) {

       &= nbsp;    return evt.getRenderedMessage().getBytes();

        } el= se {

 

            S= tring body =3D layout.format(evt);

=  

      &n= bsp;     String[] s =3D evt.getThrowableStrRep();<= /o:p>

       &nb= sp;    if (s !=3D null) {

            &n= bsp;   int len =3D s.length;

&= nbsp;           &nbs= p;   for (int i =3D 0; i < len; i++) {

          &n= bsp;         body =3D body + s[i];<= o:p>

      &nbs= p;             = body =3D body + Layout.LINE_SEP;

 &= nbsp;           &nbs= p;  }

     = ;       }

=            

        = ;    return body.getBytes();

        }

    }

=  

So now I get formatted log4j entries i= n flume! My question is; was there a reason for ignoring the log4j layout i= n the event adaptor and might what I have done be useful to anyone else?

 

= Also stack traces are appearing all on one line with the line separator esc= ape character instead of a new line. From what I have read the tail source = can cope with stacktraces and make them into a single event, is it possible= to do this with the log4j/avro source?

=  

Thanks

 

Chris

 




=20 NOTICE: Cognito Limited. Benham Valence, Newbury, Berkshire, RG20 8LU. UK.= Company number 02723032. This e-mail message and any attachment is confide= ntial. It may not be disclosed to or used by anyone other than the intended= recipient. If you have received this e-mail in error please notify the sen= der immediately then delete it from your system. Whilst every effort has be= en made to check this mail is virus free we accept no responsibility for so= ftware viruses and you should check for viruses before opening any attachme= nts. Opinions, conclusions and other information in this email and any atta= chments which do not relate to the official business of the company are nei= ther given by the company nor endorsed by it.

=20 =20 This email message has been scanned for viruses by Mimecast

=20 ------_=_NextPart_001_01CC58FB.DBEF7DF5--