flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bao Thai Ngo <baothai...@gmail.com>
Subject Some strange issues with v0.9.4 (was: collectorSink is always using default from flume-conf.xml)
Date Thu, 29 Sep 2011 11:09:22 GMT
Hello,

I did re-install version 0.94 (rpm package) and tested flume system with
same configuration of previous working system. Unfortunately, the Flume
system does not work any more. The test system consists of:

1 master
2 collectors
1 agent

Log at Master:

2011-09-29 16:59:57,774 INFO com.cloudera.flume.handlers.rolling.RollSink:
Created RollSink: trigger=[TimeTrigger: maxAge=30000
tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@687ec028]
checkPeriodMs = 250
spec='escapedCustomDfs("file:///mnt/flume/%Y-%m-%d/%H/","fl-%{rolltag}" )'
2011-09-29 16:59:57,776 INFO com.cloudera.flume.handlers.rolling.RollSink:
Created RollSink: trigger=[TimeTrigger: maxAge=30000
tagger=com.cloudera.flume.handlers.rolling.ProcessTagger@5979b7b0]
checkPeriodMs = 250
spec='escapedCustomDfs("file:///mnt/flume/%Y-%m-%d/%H/","fl-%{rolltag}" )'
2011-09-29 16:59:57,781 INFO com.cloudera.flume.master.ZooKeeperConfigStore:
Config was updated - reloading

Log at agent:

2011-09-29 17:49:03,330 INFO
com.cloudera.flume.agent.diskfailover.NaiveFileFailoverManager: opening new
file for 20110929-174853249+0700.4709680046277103.00000030
2011-09-29 17:49:03,330 INFO
com.cloudera.flume.handlers.hdfs.SeqfileEventSink: constructed new seqfile
event sink:
file=/tmp/flume-flume/agent/node2/dfo_writing/20110929-174903330+0700.4709690127666103.00000030
2011-09-29 17:49:03,331 INFO
com.cloudera.flume.agent.diskfailover.DiskFailoverSource: end of file
com.cloudera.flume.agent.diskfailover.NaiveFileFailoverManager$StateChangeDeco@2e19fc25
2011-09-29 17:49:13,410 INFO
com.cloudera.flume.handlers.hdfs.SeqfileEventSink: closed
/tmp/flume-flume/agent/node2/dfo_writing/20110929-174903330+0700.4709690127666103.00000030
2011-09-29 17:49:13,410 INFO
com.cloudera.flume.agent.diskfailover.NaiveFileFailoverManager: File lives
in
/tmp/flume-flume/agent/node2/dfo_writing/20110929-174903330+0700.4709690127666103.00000030
2011-09-29 17:49:13,410 INFO
com.cloudera.flume.agent.diskfailover.NaiveFileFailoverManager: opening new
file for 20110929-174903330+0700.4709690127666103.00000030
2011-09-29 17:49:13,411 INFO
com.cloudera.flume.handlers.hdfs.SeqfileEventSink: constructed new seqfile
event sink:
file=/tmp/flume-flume/agent/node2/dfo_writing/20110929-174913410+0700.4709700208035103.00000030
2011-09-29 17:49:13,412 INFO
com.cloudera.flume.agent.diskfailover.DiskFailoverSource: end of file
com.cloudera.flume.agent.diskfailover.NaiveFileFailoverManager$StateChangeDeco@2bf8f8c8
2011-09-29 17:49:23,494 INFO
com.cloudera.flume.handlers.hdfs.SeqfileEventSink: closed
/tmp/flume-flume/agent/node2/dfo_writing/20110929-174913410+0700.4709700208035103.00000030
2011-09-29 17:49:23,494 INFO
com.cloudera.flume.agent.diskfailover.NaiveFileFailoverManager: File lives
in
/tmp/flume-flume/agent/node2/dfo_writing/20110929-174913410+0700.4709700208035103.00000030


Log at collector:

2011-09-29 17:43:28,828 WARN com.cloudera.flume.conf.FlumeBuilder:
Deprecated syntax: Expected a format spec but instead had a (String)
avrojson
2011-09-29 17:43:59,068 WARN com.cloudera.flume.conf.FlumeBuilder:
Deprecated syntax: Expected a format spec but instead had a (String)
avrojson
2011-09-29 17:44:29,312 WARN com.cloudera.flume.conf.FlumeBuilder:
Deprecated syntax: Expected a format spec but instead had a (String)
avrojson
2011-09-29 17:44:59,556 WARN com.cloudera.flume.conf.FlumeBuilder:
Deprecated syntax: Expected a format spec but instead had a (String)
avrojson
2011-09-29 17:45:29,802 WARN com.cloudera.flume.conf.FlumeBuilder:
Deprecated syntax: Expected a format spec but instead had a (String)
avrojson
2011-09-29 17:46:00,045 WARN com.cloudera.flume.conf.FlumeBuilder:
Deprecated syntax: Expected a format spec but instead had a (String)
avrojson

As a result, no data is being sent to collector and saved onto back-end
distributed storage.

My configurations look like this:

exec config node2 test 'tail("/test/log.txt",startFromEnd="true")'
autoDFOChain
exec config collector4 test autoCollectorSource
'collectorSink("file:///mnt/flume/%Y-%m-%d/%H/","fl-")'
exec config collector5 test autoCollectorSource
'collectorSink("file:///mnt/flume/%Y-%m-%d/%H/","fl-")'
exec map lab4 collector4
exec map lab5 collector5
exec map lab2 node2

Please also keep informed that I have not implemented any decorators
developed by me.

I appreciate any help from you.

Thanks,
~Thai

On Thu, Sep 29, 2011 at 12:15 AM, Mingjie Lai <mjlai09@gmail.com> wrote:

> Thai.
>
> I got your point when looking at the source code.
>
> index 1843248..582ae82 100644
> --- a/flume-core/src/main/java/**com/cloudera/flume/conf/**
> SourceFactory.java
> +++ b/flume-core/src/main/java/**com/cloudera/flume/conf/**
> SourceFactory.java
> ...
>
> @@ -33,10 +35,20 @@ abstract public class SourceFactory {
>       return build(new Context(), argv);
>     }
>
> +    @Deprecated
>     public abstract EventSource build(Context ctx, String... argv);
> +
> +    public EventSource create(Context ctx, Object... argv) {
> +      return build(ctx, toStrings(argv));
> +    }
>   };
>
> It's about deprecating an abstract method. I'm not quite sure the best
> practice to deprecate an abstract method, maybe it should get concreted (by
> an empty body) in flume?
>
> Since the build() method is still abstract, right now (for 0.9,4) you still
> have to override it in your client code, and ignore the deprecate warning.
> Another reason is that I didn't see the create() method really gets called
> in the source code.
>
> -mingjie
>
>
> On 09/27/2011 08:02 PM, Bao Thai Ngo wrote:
>
>> Mingjie,
>>
>> Thanks for the reply.
>>
>> I did override build() instead of create() but flume gave out the error
>> as I said in previous email (WARN
>> com.cloudera.flume.conf.__**FlumeBuilder: Deprecated syntax: Expected a
>> format spec but instead had a (String) raw)
>>
>> I then overrided create() according to the information Jonathan Hsieh
>> indicated at
>> https://groups.google.com/a/**cloudera.org/group/flume-user/**
>> browse_thread/thread/**19b3c9ebb21e4b06<https://groups.google.com/a/cloudera.org/group/flume-user/browse_thread/thread/19b3c9ebb21e4b06>
>>
>> any suggestions?
>>
>> ~Thai
>>
>> On Wed, Sep 28, 2011 at 2:00 AM, Mingjie Lai <mjlai09@gmail.com
>> <mailto:mjlai09@gmail.com>> wrote:
>>
>>    Thai.
>>
>>    I checked out the 0.9.4-branch from
>>    git://github.com/cloudera/__**flume.git<http://github.com/cloudera/__flume.git>
>>    <http://github.com/cloudera/**flume.git<http://github.com/cloudera/flume.git>>,
>> but didn't see the problem.
>>
>>
>>    In the source code, you should override build() instead of create()
>>    here.
>>
>>    -mingjie
>>
>>
>>    On 09/27/2011 03:59 AM, Bao Thai Ngo wrote:
>>
>>        any thought?
>>
>>        On Wed, Sep 21, 2011 at 4:19 PM, Bao Thai Ngo
>>        <baothaingo@gmail.com <mailto:baothaingo@gmail.com>
>>        <mailto:baothaingo@gmail.com <mailto:baothaingo@gmail.com>>**>
>> wrote:
>>
>>            I'd also like to keep you informed that when compiling FLume
>>            Helloworld (modified according to some changes in version
>>        0.9.4):
>>            ------------------------------**__----
>>              public static SinkDecoBuilder builder() {
>>                 return new SinkDecoBuilder() {
>>                   // construct a new parameterized decorator
>>                   @Override
>>                   public EventSinkDecorator<EventSink> create(Context
>>            context,Object... argv) {
>>                     Preconditions.checkArgument(__**argv.length ==
>> 0,"usage:
>>            helloWorldDecorator");
>>
>>                     return new HelloWorldDecorator<EventSink>**__(null);
>>                   }
>>
>>                 };
>>               }
>>
>>            -----------------------
>>
>>            I get the following error:
>>
>>            [javac]
>>
>>          /usr/lib/flume/plugins/__**helloworld/src/__**
>> HelloWorldDecorator.java:34:
>>        <anonymous helloworld.__**HelloWorldDecorator$1> is not abstract
>> and
>>            does not override abstract method
>>            build(com.cloudera.flume.conf.**__Context,java.lang.String...)
>> in
>>            com.cloudera.flume.conf.__**SinkFactory.SinkDecoBuilder
>>                 [javac]     return new SinkDecoBuilder() {
>>
>>            Was I missing something?
>>
>>            ~Thai
>>
>>
>>            On Wed, Sep 21, 2011 at 3:41 PM, Bao Thai Ngo
>>        <baothaingo@gmail.com <mailto:baothaingo@gmail.com>
>>        <mailto:baothaingo@gmail.com <mailto:baothaingo@gmail.com>>**>
>> wrote:
>>
>>                Hello list,
>>
>>                After updating to Flume v0.9.4-cdh3u1, I do meet this
>> error:
>>                WARN com.cloudera.flume.conf.__**FlumeBuilder: Deprecated
>>        syntax:
>>                Expected a format spec but instead had a (String) raw
>>                As a result, collector does not save events on storage
>>        backend
>>
>>                Before the updating, my plug-ins worked completely well.
>>        Below
>>                is my typical config:
>>                exec config node1 test
>>        'tail("/path/to/test.txt",__**startFromEnd="true")'
>>        '{deco_modiy_event_body()=>__**autoDFOChain}'
>>
>>                Any thoughts?
>>
>>                Thanks,
>>                ~Thai
>>
>>                On Sat, Sep 10, 2011 at 4:14 AM, Jeff Hansen
>>        <dscheffy@gmail.com <mailto:dscheffy@gmail.com>
>>        <mailto:dscheffy@gmail.com <mailto:dscheffy@gmail.com>>> wrote:
>>
>>                    Sorry, I should have also mentioned that the output
>>        itself
>>                    is coming
>>                    out in the raw format (consistent with the xml
>>        config rather
>>                    than the
>>                    runtime config)
>>
>>                    On Fri, Sep 9, 2011 at 4:10 PM, Jeff Hansen
>>        <dscheffy@gmail.com <mailto:dscheffy@gmail.com>
>>        <mailto:dscheffy@gmail.com <mailto:dscheffy@gmail.com>>> wrote:
>>         > When I configure a collectorSink to use a format
>>                    different from the
>>         > default specified in the flume-conf.xml file, it
>>                    continues to use the
>>         > default.  I get the following error
>>         >
>>         > 2011-09-09 16:02:29,059 [Roll-TriggerThread-0] WARN
>>                    conf.FlumeBuilder:
>>         > Deprecated syntax: Expected a format spec but instead had
>>                    a (String)
>>         > raw
>>         >
>>         > even though I'm using the following combination of
>>                    configurations.  Is
>>         > the xml file configuration supposed to supercede that of
>>                    the runtime
>>         > configuration?
>>         >
>>         > I'm running version "Flume 0.9.4-cdh3u1"
>>         >
>>         > Here's the complete startup script I'm using:
>>         >
>>         >    #!/bin/sh
>>         >    gnome-terminal -e "flume master"
>>         >    sleep 10
>>         >    flume shell -c localhost -e "exec config agent
>>         > 'tail(\"/var/log/apache2/__**access.log\")' '[console,
>>         >
>>
>>          collectorSink(\"hdfs://__**localhost/flume/avro/\",\"log\**
>> __",60000,avrojson)]'"
>>         >    gnome-terminal -e "flume node -n agent"
>>         >
>>         > Here's the appropriate entry from my flume-conf.xml file:
>>         > <property>
>>         > <name>flume.collector.output._**_format</name>
>>         > <value>raw</value>
>>         > <description>The output format for the data written by a
>>                    Flume
>>         >    collector node.  There are several formats available:
>>         >      syslog - outputs events in a syslog-like format
>>         >      log4j - outputs events in a pattern similar to
>>                    Hadoop's log4j pattern
>>         >      raw - Event body only.  This is most similar to
>>                    copying a file but
>>         >        does not preserve any uniqifying metadata like
>>                    host/timestamp/nanos.
>>         >      avro - Avro Native file format.  Default currently
>>                    is uncompressed.
>>         >      avrojson - this outputs data as json encoded by avro
>>         >      avrodata - this outputs data as a avro binary
>>                    encoded data
>>         >      debug - used only for debugging
>>         > </description>
>>         > </property>
>>         >
>>
>>
>>
>>
>>
>>

Mime
View raw message