flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bao Thai Ngo <baothai...@gmail.com>
Subject Re: collectorSink is always using default from flume-conf.xml
Date Wed, 21 Sep 2011 08:41:42 GMT
Hello list,

After updating to Flume v0.9.4-cdh3u1, I do meet this error:
WARN com.cloudera.flume.conf.FlumeBuilder: Deprecated syntax: Expected a
format spec but instead had a (String) raw
As a result, collector does not save events on storage backend

Before the updating, my plug-ins worked completely well. Below is my typical
config:
exec config node1 test 'tail("/path/to/test.txt",startFromEnd="true")'
'{deco_modiy_event_body()=>autoDFOChain}'

Any thoughts?

Thanks,
~Thai

On Sat, Sep 10, 2011 at 4:14 AM, Jeff Hansen <dscheffy@gmail.com> wrote:

> Sorry, I should have also mentioned that the output itself is coming
> out in the raw format (consistent with the xml config rather than the
> runtime config)
>
> On Fri, Sep 9, 2011 at 4:10 PM, Jeff Hansen <dscheffy@gmail.com> wrote:
> > When I configure a collectorSink to use a format different from the
> > default specified in the flume-conf.xml file, it continues to use the
> > default.  I get the following error
> >
> > 2011-09-09 16:02:29,059 [Roll-TriggerThread-0] WARN conf.FlumeBuilder:
> > Deprecated syntax: Expected a format spec but instead had a (String)
> > raw
> >
> > even though I'm using the following combination of configurations.  Is
> > the xml file configuration supposed to supercede that of the runtime
> > configuration?
> >
> > I'm running version "Flume 0.9.4-cdh3u1"
> >
> > Here's the complete startup script I'm using:
> >
> >    #!/bin/sh
> >    gnome-terminal -e "flume master"
> >    sleep 10
> >    flume shell -c localhost -e "exec config agent
> > 'tail(\"/var/log/apache2/access.log\")' '[console,
> > collectorSink(\"hdfs://localhost/flume/avro/\",\"log\",60000,avrojson)]'"
> >    gnome-terminal -e "flume node -n agent"
> >
> > Here's the appropriate entry from my flume-conf.xml file:
> >  <property>
> >    <name>flume.collector.output.format</name>
> >    <value>raw</value>
> >    <description>The output format for the data written by a Flume
> >    collector node.  There are several formats available:
> >      syslog - outputs events in a syslog-like format
> >      log4j - outputs events in a pattern similar to Hadoop's log4j
> pattern
> >      raw - Event body only.  This is most similar to copying a file but
> >        does not preserve any uniqifying metadata like
> host/timestamp/nanos.
> >      avro - Avro Native file format.  Default currently is uncompressed.
> >      avrojson - this outputs data as json encoded by avro
> >      avrodata - this outputs data as a avro binary encoded data
> >      debug - used only for debugging
> >    </description>
> >  </property>
> >
>

Mime
View raw message