flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bao Thai Ngo <baothai...@gmail.com>
Subject Re: collectorSink is always using default from flume-conf.xml
Date Tue, 27 Sep 2011 10:59:28 GMT
any thought?

On Wed, Sep 21, 2011 at 4:19 PM, Bao Thai Ngo <baothaingo@gmail.com> wrote:

> I'd also like to keep you informed that when compiling FLume Helloworld
> (modified according to some changes in version 0.9.4):
> ----------------------------------
>  public static SinkDecoBuilder builder() {
>     return new SinkDecoBuilder() {
>       // construct a new parameterized decorator
>       @Override
>       public EventSinkDecorator<EventSink> create(Context context,Object...
> argv) {
>         Preconditions.checkArgument(argv.length == 0,"usage:
> helloWorldDecorator");
>
>         return new HelloWorldDecorator<EventSink>(null);
>       }
>
>     };
>   }
>
> -----------------------
>
> I get the following error:
>
> [javac] /usr/lib/flume/plugins/helloworld/src/HelloWorldDecorator.java:34:
> <anonymous helloworld.HelloWorldDecorator$1> is not abstract and does not
> override abstract method
> build(com.cloudera.flume.conf.Context,java.lang.String...) in
> com.cloudera.flume.conf.SinkFactory.SinkDecoBuilder
>     [javac]     return new SinkDecoBuilder() {
>
> Was I missing something?
>
> ~Thai
>
>
> On Wed, Sep 21, 2011 at 3:41 PM, Bao Thai Ngo <baothaingo@gmail.com>wrote:
>
>> Hello list,
>>
>> After updating to Flume v0.9.4-cdh3u1, I do meet this error:
>> WARN com.cloudera.flume.conf.FlumeBuilder: Deprecated syntax: Expected a
>> format spec but instead had a (String) raw
>> As a result, collector does not save events on storage backend
>>
>> Before the updating, my plug-ins worked completely well. Below is my
>> typical config:
>> exec config node1 test 'tail("/path/to/test.txt",startFromEnd="true")'
>> '{deco_modiy_event_body()=>autoDFOChain}'
>>
>> Any thoughts?
>>
>> Thanks,
>> ~Thai
>>
>> On Sat, Sep 10, 2011 at 4:14 AM, Jeff Hansen <dscheffy@gmail.com> wrote:
>>
>>> Sorry, I should have also mentioned that the output itself is coming
>>> out in the raw format (consistent with the xml config rather than the
>>> runtime config)
>>>
>>> On Fri, Sep 9, 2011 at 4:10 PM, Jeff Hansen <dscheffy@gmail.com> wrote:
>>> > When I configure a collectorSink to use a format different from the
>>> > default specified in the flume-conf.xml file, it continues to use the
>>> > default.  I get the following error
>>> >
>>> > 2011-09-09 16:02:29,059 [Roll-TriggerThread-0] WARN conf.FlumeBuilder:
>>> > Deprecated syntax: Expected a format spec but instead had a (String)
>>> > raw
>>> >
>>> > even though I'm using the following combination of configurations.  Is
>>> > the xml file configuration supposed to supercede that of the runtime
>>> > configuration?
>>> >
>>> > I'm running version "Flume 0.9.4-cdh3u1"
>>> >
>>> > Here's the complete startup script I'm using:
>>> >
>>> >    #!/bin/sh
>>> >    gnome-terminal -e "flume master"
>>> >    sleep 10
>>> >    flume shell -c localhost -e "exec config agent
>>> > 'tail(\"/var/log/apache2/access.log\")' '[console,
>>> >
>>> collectorSink(\"hdfs://localhost/flume/avro/\",\"log\",60000,avrojson)]'"
>>> >    gnome-terminal -e "flume node -n agent"
>>> >
>>> > Here's the appropriate entry from my flume-conf.xml file:
>>> >  <property>
>>> >    <name>flume.collector.output.format</name>
>>> >    <value>raw</value>
>>> >    <description>The output format for the data written by a Flume
>>> >    collector node.  There are several formats available:
>>> >      syslog - outputs events in a syslog-like format
>>> >      log4j - outputs events in a pattern similar to Hadoop's log4j
>>> pattern
>>> >      raw - Event body only.  This is most similar to copying a file but
>>> >        does not preserve any uniqifying metadata like
>>> host/timestamp/nanos.
>>> >      avro - Avro Native file format.  Default currently is
>>> uncompressed.
>>> >      avrojson - this outputs data as json encoded by avro
>>> >      avrodata - this outputs data as a avro binary encoded data
>>> >      debug - used only for debugging
>>> >    </description>
>>> >  </property>
>>> >
>>>
>>
>>
>

Mime
View raw message