flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish <paliwalash...@gmail.com>
Subject Re: Problems with Flume on windows with HTTP source
Date Fri, 21 Jun 2013 08:25:48 GMT
I pulled your source configuration. Here is the full configuration
(single-node-custom-source.properties), removed my custom source from here

# Exampple for single node config

# Base Config
a1.sources=src1
a1.sinks=sink1
a1.channels=ch1

# Configure the source
a1.sources.src1.type = org.apache.flume.source.http.HTTPSource
a1.sources.src1.port = 9001
a1.sources.src1.bind = localhost
a1.sources.src1.handler = org.apache.flume.source.http.JSONHandler

# Sink Configuration
a1.sinks.sink1.type=logger

# Channel configuration
a1.channels.ch1.type=memory
a1.channels.ch1.capacity=1000
a1.channels.ch1.transactionCapacity=100

# Link stuff together
a1.sources.src1.channels=ch1
a1.sinks.sink1.channel=ch1

and this is how I start my agent

bin/flume-ng agent --conf-file
conf/single-node-custom-source.properties --name a1 --conf ./conf/
-Dflume.root.logger=INFO,console


On Fri, Jun 21, 2013 at 1:50 PM, Nickolay Kolev <lem0na2@gmail.com> wrote:

> Hi Asish,
>
> I think I found the problem - there must be something wrong with the
> configuration and the source is not created. Unfortunately I can not figure
> what.
> Did you test it with my configuration?
>
> Here is what I found in server log:
> 2013-06-21 11:01:24,433 (conf-file-poller-0) [DEBUG -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:295)]
> Starting validation of configuration for agent: agent1,
> initial-configuration: AgentConfiguration[agent1]
> SOURCES: {r1={ parameters:{port=9001, handler.nickname=json handler,
> channels=ch1, type=org.apache.flume.source.http.HTTPSource,
> handler=org.apache.flume.source.http.JSONHandler} }
> }
> CHANNELS: {ch1={ parameters:{transactionCapacity=1000, capacity=10000,
> type=memory} }}
> SINKS: {k1={ parameters:{type=file_roll, channel=ch1,
> sink.directory=C:\Projects\nickyk\apache-flume-1.3.1-bin\log} }}
>
> 2013-06-21 11:01:24,451 (conf-file-poller-0) [DEBUG -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)]
> Created channel ch1
>
> 2013-06-21 11:01:24,451 (conf-file-poller-0) [WARN -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSources(FlumeConfiguration.java:489)]
> Agent configuration for 'agent1' has no sources.
> 2013-06-21 11:01:24,455 (conf-file-poller-0) [DEBUG -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:655)]
> Creating sink: k1 using FILE_ROLL
> 2013-06-21 11:01:24,457 (conf-file-poller-0) [DEBUG -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:353)]
> Post validation configuration for agent1
> AgentConfiguration created without Configuration stubs for which only
> basic syntactical validation was performed[agent1]
> SOURCES: {r1={ parameters:{port=9001, handler.nickname=json handler,
> channels=ch1, type=org.apache.flume.source.http.HTTPSource,
> handler=org.apache.flume.source.http.JSONHandler} }
> }
> CHANNELS: {ch1={ parameters:{transactionCapacity=1000, capacity=10000,
> type=memory} }}
> SINKS: {k1={ parameters:{type=file_roll, channel=ch1,
> sink.directory=C:\Projects\nickyk\apache-flume-1.3.1-bin\log} }}
>
> 2013-06-21 11:01:24,470 (conf-file-poller-0) [DEBUG -
> org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:117)]
> Channels:ch1
>
> 2013-06-21 11:01:24,471 (conf-file-poller-0) [DEBUG -
> org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:118)]
> Sinks k1
>
> 2013-06-21 11:01:24,471 (conf-file-poller-0) [DEBUG -
> org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:119)]
> Sources null
>
> best regards,
> nickolay kolev
>
>
>
>
> On Fri, Jun 21, 2013 at 10:40 AM, Ashish <paliwalashish@gmail.com> wrote:
>
>> I am using Flume 1.3.1 and ran the example before sending.
>>
>> I don't have access to a windows system at the moment so can't test.
>>
>> This is output at my end on Flume Agent console
>>
>> 2013-06-21 13:03:54,913 (conf-file-poller-0) [INFO -
>> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:152)]
>> Starting Source src1
>> 2013-06-21 13:03:54,927 (lifecycleSupervisor-1-2) [INFO -
>> org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Logging to
>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>> 2013-06-21 13:03:54,958 (lifecycleSupervisor-1-2) [INFO -
>> org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] jetty-6.1.26
>> 2013-06-21 13:03:55,022 (lifecycleSupervisor-1-2) [INFO -
>> org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Started
>> SocketConnector@0.0.0.0:9001
>> 2013-06-21 13:09:36,992 (SinkRunner-PollingRunner-DefaultSinkProcessor)
>> [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)]
>> Event: { headers:{timestamp=434324343, host=random_host.example.co
>> m} body: 72 61 6E 64 6F 6D 5F 62 6F 64 79                random_body }
>> 2013-06-21 13:09:36,992 (SinkRunner-PollingRunner-DefaultSinkProcessor)
>> [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)]
>> Event: { headers:{namenode=namenode.example.com, datanode=
>> random_datanode.example.com} body: 72 65 61 6C 6C 79 5F 72 61 6E 64 6F
>> 6D 5F 62 6F really_random_bo }
>>
>> Flume dev's can comment further on this.
>>
>>
>> On Fri, Jun 21, 2013 at 12:33 PM, Nickolay Kolev <lem0na2@gmail.com>wrote:
>>
>>> Hi Ashish,
>>>
>>> Thanks for pointing me that error. I am trying to read the code and this
>>> is the correct full class name. (last time I wrote java code was in 1998
>>> and my knowledge are a lot out of date)
>>>
>>> Unfortunately the result is the same.
>>> >curl -X POST -H "Content-Type: application/json" -d "[{ \"headers\" : {
>>> \"timestamp\" : \"434324343\", \"host\" : \"random_host.example.com\"
>>> }, \"body\" : \"random_body\" }, { \"headers\" : { \"namenode\" : \"
>>> namenode.example.com\", \"datanode\" : \"random_datanode.example.com\"
>>> }, \"body\" : \"really_random_body\"}]" http://localhost:9001
>>>
>>> curl: (7) Failed connect to localhost:9001; No error
>>>
>>> Could be the reason if this errors the fact that I use Windows? I see in
>>> wiki that Windows is not officially supported.
>>>
>>> Yesterday my first test was with netcat source and it worked as expected.
>>>
>>>
>>> best regards,
>>> nickolay kolev
>>>
>>>
>>> On Fri, Jun 21, 2013 at 9:48 AM, Ashish <paliwalashish@gmail.com> wrote:
>>>
>>>> update the handler config as
>>>>
>>>> agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
>>>>
>>>> and try. I used your source config and it worked for me. Let us know if
>>>> it work
>>>>
>>>>
>>>> On Fri, Jun 21, 2013 at 12:04 PM, Nickolay Kolev <lem0na2@gmail.com>wrote:
>>>>
>>>>> Hi Hari and the all members of this list,
>>>>>
>>>>> Thanks for pointing me that error. This is actually my second attempt
>>>>> to use Apache Flume - with avro client. The symptoms are the same - the
is
>>>>> no listener on the specified port and I am not able to connect.
>>>>>
>>>>> I started to evaluate Flume just yesterday and many things are new to
>>>>> me and probably I do many mistakes. I want to point that I use Flume
1.3.1
>>>>> compiled by me if this is important to be mentioned. I have tested too
with
>>>>> the official binary from Flume web site but the results are the same.
>>>>>
>>>>> Here is what I do to test HTTP source:
>>>>>
>>>>> my config file (httppost.conf):
>>>>> agent1.sources = r1
>>>>> agent1.channels = ch1
>>>>> agent1.sinks = k1
>>>>>
>>>>> agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
>>>>> agent1.sources.r1.port = 9001
>>>>> agent1.sources.r1.channels = ch1
>>>>> #agent1.sources.r1.handler = org.example.rest.RestHandler
>>>>> agent1.sources.r1.handler = org.apache.flume.http.JSONHandler
>>>>> agent1.sources.r1.handler.nickname = json handler
>>>>>
>>>>> agent1.sinks.k1.type = file_roll
>>>>> agent1.sinks.k1.channel = ch1
>>>>>
>>>>> agent1.sinks.k1.sink.directory =
>>>>> C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log
>>>>>
>>>>> agent1.channels.ch1.type = memory
>>>>> agent1.channels.ch1.capacity = 10000
>>>>> agent1.channels.ch1.transactionCapacity = 1000
>>>>>
>>>>> my server bat file:
>>>>>
>>>>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\
>>>>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
>>>>> set JAVA_OPTS=-Xmx60m
>>>>> set CONF=%FLUME_HOME%\conf\httppost.conf
>>>>> set AGENT=agent1
>>>>>
>>>>> %JAVA% %JAVA_OPTS%
>>>>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp
>>>>> "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n %AGENT%
>>>>>
>>>>> How I test:
>>>>>
>>>>> server console:
>>>>> >"C:\Java\jdk1.6.0_45\bin\java.exe" -Xmx60m
>>>>> -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\log4j.properties
>>>>> -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\\lib\*"
>>>>> org.apache.flume.node.Application -f
>>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\httppost.conf -n agent1
>>>>>
>>>>> client console:
>>>>> >C:\Projects\nickyk\apache-flume-1.3.1-bin>curl -X POST -H
>>>>> "Content-Type: application/json" -d "[{ \"headers\" : { \"timestamp\"
:
>>>>> \"434324343\", \"host\" : \"random_host.example.co
>>>>> m\" }, \"body\" : \"random_body\" }, { \"headers\" : { \"namenode\" :
>>>>> \"namenode.example.com\", \"datanode\" : \"random_datanode.example.com\"
>>>>> }, \"body\" : \"really_random_body\"
>>>>> }]" http://localhost:9001
>>>>> curl: (7) Failed connect to localhost:9001; No error
>>>>>
>>>>> If I understand correct the documentation I should expect that on the
>>>>> specified port (9001) it must be listener that will accept POST requests?
>>>>> Is this correct?
>>>>> Do I miss some important aspects that are critical to run Flume?
>>>>>
>>>>> Thanks in advance for your help.
>>>>>
>>>>>
>>>>> best regards,
>>>>> nickolay kolev
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jun 20, 2013 at 9:07 PM, Hari Shreedharan <
>>>>> hshreedharan@cloudera.com> wrote:
>>>>>
>>>>>>  Does not look like you are using the http source at all. Your source
>>>>>> type needs to be HTTP
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>> Hari
>>>>>>
>>>>>> On Thursday, June 20, 2013 at 8:57 AM, Nickolay Kolev wrote:
>>>>>>
>>>>>> Hi all,
>>>>>> I am new to flume and all that logging stuff and probably many things
>>>>>> are unclear to me despite I read the docs.
>>>>>> I want to run Flume on windows server to create proof of concept
>>>>>> solution. I was able to compile it successfully. I was able to make
>>>>>> successful test with netcat.
>>>>>> Next test that i wanted to do is with http source but unfortunately
>>>>>> it did not pass.
>>>>>>  Here is what I do:
>>>>>>
>>>>>> my server config (httppost.conf):
>>>>>> agent1.sources = r1
>>>>>> agent1.channels = memoryChannel
>>>>>> agent1.sinks = k1
>>>>>>
>>>>>> agent1.sources.r1.type = avro
>>>>>> agent1.sources.r1.bind = localhost
>>>>>> agent1.sources.r1.port = 9003
>>>>>>
>>>>>> agent1.sinks.k1.type = file_roll
>>>>>> agent1.sinks.k1.sink.directory =
>>>>>> C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log
>>>>>>
>>>>>> agent1.channels.memoryChannel.type = memory
>>>>>> agent1.channels.memoryChannel.capacity = 10000
>>>>>> agent1.channels.memoryChannel.transactionCapacity = 1000
>>>>>>
>>>>>>
>>>>>> agent1.sources.r1.channels = memoryChannel
>>>>>> agent1.sinks.k1.channel = memoryChannel
>>>>>>
>>>>>> I run the server with this bat file:
>>>>>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\
>>>>>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
>>>>>> set JAVA_OPTS=-Xmx60m
>>>>>> set CONF=%FLUME_HOME%\conf\httppost.conf
>>>>>> set AGENT=agent1
>>>>>>
>>>>>> %JAVA% %JAVA_OPTS%
>>>>>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties
-cp
>>>>>> "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF%
-n %AGENT%
>>>>>>
>>>>>> netstat does not show any listener on port 9003? Is this normal or
>>>>>> there must be a listener?
>>>>>>
>>>>>> I run avro client with this bat file:
>>>>>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin
>>>>>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
>>>>>> set JAVA_OPTS=-Xmx60m
>>>>>> set AVRO_HOST=localhost
>>>>>> set AVRO_PORT=9003
>>>>>> set SENDFILE=%FLUME_HOME%\logs\data.log
>>>>>> set HEADFILE=%FLUME_HOME%\conf\header.txt
>>>>>>
>>>>>> %JAVA%  %JAVA_OPTS%
>>>>>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties
-cp
>>>>>> "%FLUME_HOME%\lib\*" org.apache.flume.client.avro.AvroCLIClient -H
>>>>>> %AVRO_HOST% -p %AVRO_PORT%  -F %SENDFILE%  --headerFile %HEADFILE%
>>>>>>
>>>>>>
>>>>>> Then I get this output:
>>>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin>"C:\Java\jdk1.6.0_45\bin\java.exe"
>>>>>> -Xmx60m
>>>>>> -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\log4j.properties
>>>>>> -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\lib\*"
>>>>>> org.apache.flume.client.avro.AvroCLIClient -H  localhost -p 9003
 -F
>>>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin\logs\data.log --headerFile
>>>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\header.txt
>>>>>> 2013-06-20 17:55:58,285 (main) [DEBUG -
>>>>>> org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)]
>>>>>> Inserting Header Key [filename] header value [datalog]
>>>>>> 2013-06-20 17:55:58,287 (main) [DEBUG -
>>>>>> org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)]
>>>>>> Inserting Header Key [host] header value [zhouhh]
>>>>>> 2013-06-20 17:55:58,295 (main) [DEBUG -
>>>>>> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)]
>>>>>> Batch size string = 5
>>>>>> 2013-06-20 17:55:59,358 (main) [ERROR -
>>>>>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:79)]
>>>>>> Unable to open connection to Flume. Exception follows.
>>>>>> org.apache.flume.FlumeException: NettyAvroRpcClient { host:
>>>>>> localhost, port: 9003 }: RPC connection error
>>>>>>         at
>>>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>>>>>>         at
>>>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>>>>>>         at
>>>>>> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>>>>>>         at
>>>>>> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>>>>>>         at
>>>>>> org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>>>>>>         at
>>>>>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>>>>>> Caused by: java.io.IOException: Error connecting to localhost/
>>>>>> 127.0.0.1:9003
>>>>>>         at
>>>>>> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>>>>>>         at
>>>>>> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>>>>>>         at
>>>>>> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>>>>>>         at
>>>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>>>>>>         ... 5 more
>>>>>> Caused by: java.net.ConnectException: Connection refused: no further
>>>>>> information
>>>>>>         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>>>         at
>>>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
>>>>>>         at
>>>>>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>>>>>>         at
>>>>>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>>>>>>         at
>>>>>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>>>>>>         at
>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>         at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>>>         at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>> 2013-06-20 17:55:59,369 (main) [DEBUG -
>>>>>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:84)]
>>>>>> Exiting
>>>>>>
>>>>>>
>>>>>> My question is what I am doing wrong and what I need to test in order
>>>>>> to fix this situatioin.
>>>>>>
>>>>>> Thanks in advance.
>>>>>>
>>>>>> best regards,
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> thanks
>>>> ashish
>>>>
>>>> Blog: http://www.ashishpaliwal.com/blog
>>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>>>
>>>
>>>
>>
>>
>> --
>> thanks
>> ashish
>>
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>
>
>


-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Mime
View raw message