flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bao Thai Ngo <baothai...@gmail.com>
Subject Re: collectorSink is always using default from flume-conf.xml
Date Wed, 28 Sep 2011 03:02:55 GMT
Mingjie,

Thanks for the reply.

I did override build() instead of create() but flume gave out the error as I
said in previous email (WARN com.cloudera.flume.conf.**FlumeBuilder:
Deprecated syntax: Expected a format spec but instead had a (String) raw)

I then overrided create() according to the information Jonathan Hsieh
indicated at
https://groups.google.com/a/cloudera.org/group/flume-user/browse_thread/thread/19b3c9ebb21e4b06

any suggestions?

~Thai

On Wed, Sep 28, 2011 at 2:00 AM, Mingjie Lai <mjlai09@gmail.com> wrote:

> Thai.
>
> I checked out the 0.9.4-branch from git://github.com/cloudera/**flume.git<http://github.com/cloudera/flume.git>,
> but didn't see the problem.
>
> In the source code, you should override build() instead of create() here.
>
> -mingjie
>
>
> On 09/27/2011 03:59 AM, Bao Thai Ngo wrote:
>
>> any thought?
>>
>> On Wed, Sep 21, 2011 at 4:19 PM, Bao Thai Ngo <baothaingo@gmail.com
>> <mailto:baothaingo@gmail.com>> wrote:
>>
>>    I'd also like to keep you informed that when compiling FLume
>>    Helloworld (modified according to some changes in version 0.9.4):
>>    ------------------------------**----
>>      public static SinkDecoBuilder builder() {
>>         return new SinkDecoBuilder() {
>>           // construct a new parameterized decorator
>>           @Override
>>           public EventSinkDecorator<EventSink> create(Context
>>    context,Object... argv) {
>>             Preconditions.checkArgument(**argv.length == 0,"usage:
>>    helloWorldDecorator");
>>
>>             return new HelloWorldDecorator<EventSink>**(null);
>>           }
>>
>>         };
>>       }
>>
>>    -----------------------
>>
>>    I get the following error:
>>
>>    [javac]
>>    /usr/lib/flume/plugins/**helloworld/src/**HelloWorldDecorator.java:34:
>>    <anonymous helloworld.**HelloWorldDecorator$1> is not abstract and
>>    does not override abstract method
>>    build(com.cloudera.flume.conf.**Context,java.lang.String...) in
>>    com.cloudera.flume.conf.**SinkFactory.SinkDecoBuilder
>>         [javac]     return new SinkDecoBuilder() {
>>
>>    Was I missing something?
>>
>>    ~Thai
>>
>>
>>    On Wed, Sep 21, 2011 at 3:41 PM, Bao Thai Ngo <baothaingo@gmail.com
>>    <mailto:baothaingo@gmail.com>> wrote:
>>
>>        Hello list,
>>
>>        After updating to Flume v0.9.4-cdh3u1, I do meet this error:
>>        WARN com.cloudera.flume.conf.**FlumeBuilder: Deprecated syntax:
>>        Expected a format spec but instead had a (String) raw
>>        As a result, collector does not save events on storage backend
>>
>>        Before the updating, my plug-ins worked completely well. Below
>>        is my typical config:
>>        exec config node1 test
>>        'tail("/path/to/test.txt",**startFromEnd="true")'
>>        '{deco_modiy_event_body()=>**autoDFOChain}'
>>
>>        Any thoughts?
>>
>>        Thanks,
>>        ~Thai
>>
>>        On Sat, Sep 10, 2011 at 4:14 AM, Jeff Hansen <dscheffy@gmail.com
>>        <mailto:dscheffy@gmail.com>> wrote:
>>
>>            Sorry, I should have also mentioned that the output itself
>>            is coming
>>            out in the raw format (consistent with the xml config rather
>>            than the
>>            runtime config)
>>
>>            On Fri, Sep 9, 2011 at 4:10 PM, Jeff Hansen
>>            <dscheffy@gmail.com <mailto:dscheffy@gmail.com>> wrote:
>>             > When I configure a collectorSink to use a format
>>            different from the
>>             > default specified in the flume-conf.xml file, it
>>            continues to use the
>>             > default.  I get the following error
>>             >
>>             > 2011-09-09 16:02:29,059 [Roll-TriggerThread-0] WARN
>>            conf.FlumeBuilder:
>>             > Deprecated syntax: Expected a format spec but instead had
>>            a (String)
>>             > raw
>>             >
>>             > even though I'm using the following combination of
>>            configurations.  Is
>>             > the xml file configuration supposed to supercede that of
>>            the runtime
>>             > configuration?
>>             >
>>             > I'm running version "Flume 0.9.4-cdh3u1"
>>             >
>>             > Here's the complete startup script I'm using:
>>             >
>>             >    #!/bin/sh
>>             >    gnome-terminal -e "flume master"
>>             >    sleep 10
>>             >    flume shell -c localhost -e "exec config agent
>>             > 'tail(\"/var/log/apache2/**access.log\")' '[console,
>>             >
>>            collectorSink(\"hdfs://**localhost/flume/avro/\",\"log\**
>> ",60000,avrojson)]'"
>>             >    gnome-terminal -e "flume node -n agent"
>>             >
>>             > Here's the appropriate entry from my flume-conf.xml file:
>>             > <property>
>>             > <name>flume.collector.output.**format</name>
>>             > <value>raw</value>
>>             > <description>The output format for the data written by a
>>            Flume
>>             >    collector node.  There are several formats available:
>>             >      syslog - outputs events in a syslog-like format
>>             >      log4j - outputs events in a pattern similar to
>>            Hadoop's log4j pattern
>>             >      raw - Event body only.  This is most similar to
>>            copying a file but
>>             >        does not preserve any uniqifying metadata like
>>            host/timestamp/nanos.
>>             >      avro - Avro Native file format.  Default currently
>>            is uncompressed.
>>             >      avrojson - this outputs data as json encoded by avro
>>             >      avrodata - this outputs data as a avro binary
>>            encoded data
>>             >      debug - used only for debugging
>>             > </description>
>>             > </property>
>>             >
>>
>>
>>
>>
>>

Mime
View raw message