flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish <paliwalash...@gmail.com>
Subject Re: Problems with Flume on windows with HTTP source
Date Fri, 21 Jun 2013 06:48:50 GMT
update the handler config as

agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler

and try. I used your source config and it worked for me. Let us know if it
work


On Fri, Jun 21, 2013 at 12:04 PM, Nickolay Kolev <lem0na2@gmail.com> wrote:

> Hi Hari and the all members of this list,
>
> Thanks for pointing me that error. This is actually my second attempt to
> use Apache Flume - with avro client. The symptoms are the same - the is no
> listener on the specified port and I am not able to connect.
>
> I started to evaluate Flume just yesterday and many things are new to me
> and probably I do many mistakes. I want to point that I use Flume 1.3.1
> compiled by me if this is important to be mentioned. I have tested too with
> the official binary from Flume web site but the results are the same.
>
> Here is what I do to test HTTP source:
>
> my config file (httppost.conf):
> agent1.sources = r1
> agent1.channels = ch1
> agent1.sinks = k1
>
> agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
> agent1.sources.r1.port = 9001
> agent1.sources.r1.channels = ch1
> #agent1.sources.r1.handler = org.example.rest.RestHandler
> agent1.sources.r1.handler = org.apache.flume.http.JSONHandler
> agent1.sources.r1.handler.nickname = json handler
>
> agent1.sinks.k1.type = file_roll
> agent1.sinks.k1.channel = ch1
>
> agent1.sinks.k1.sink.directory =
> C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log
>
> agent1.channels.ch1.type = memory
> agent1.channels.ch1.capacity = 10000
> agent1.channels.ch1.transactionCapacity = 1000
>
> my server bat file:
>
> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\
> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
> set JAVA_OPTS=-Xmx60m
> set CONF=%FLUME_HOME%\conf\httppost.conf
> set AGENT=agent1
>
> %JAVA% %JAVA_OPTS%
> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp
> "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n %AGENT%
>
> How I test:
>
> server console:
> >"C:\Java\jdk1.6.0_45\bin\java.exe" -Xmx60m
> -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\log4j.properties
> -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\\lib\*"
> org.apache.flume.node.Application -f
> C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\httppost.conf -n agent1
>
> client console:
> >C:\Projects\nickyk\apache-flume-1.3.1-bin>curl -X POST -H "Content-Type:
> application/json" -d "[{ \"headers\" : { \"timestamp\" : \"434324343\",
> \"host\" : \"random_host.example.co
> m\" }, \"body\" : \"random_body\" }, { \"headers\" : { \"namenode\" : \"
> namenode.example.com\", \"datanode\" : \"random_datanode.example.com\" },
> \"body\" : \"really_random_body\"
> }]" http://localhost:9001
> curl: (7) Failed connect to localhost:9001; No error
>
> If I understand correct the documentation I should expect that on the
> specified port (9001) it must be listener that will accept POST requests?
> Is this correct?
> Do I miss some important aspects that are critical to run Flume?
>
> Thanks in advance for your help.
>
>
> best regards,
> nickolay kolev
>
>
>
> On Thu, Jun 20, 2013 at 9:07 PM, Hari Shreedharan <
> hshreedharan@cloudera.com> wrote:
>
>>  Does not look like you are using the http source at all. Your source
>> type needs to be HTTP
>>
>>
>> Cheers,
>> Hari
>>
>> On Thursday, June 20, 2013 at 8:57 AM, Nickolay Kolev wrote:
>>
>> Hi all,
>> I am new to flume and all that logging stuff and probably many things are
>> unclear to me despite I read the docs.
>> I want to run Flume on windows server to create proof of concept
>> solution. I was able to compile it successfully. I was able to make
>> successful test with netcat.
>> Next test that i wanted to do is with http source but unfortunately it
>> did not pass.
>>  Here is what I do:
>>
>> my server config (httppost.conf):
>> agent1.sources = r1
>> agent1.channels = memoryChannel
>> agent1.sinks = k1
>>
>> agent1.sources.r1.type = avro
>> agent1.sources.r1.bind = localhost
>> agent1.sources.r1.port = 9003
>>
>> agent1.sinks.k1.type = file_roll
>> agent1.sinks.k1.sink.directory =
>> C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log
>>
>> agent1.channels.memoryChannel.type = memory
>> agent1.channels.memoryChannel.capacity = 10000
>> agent1.channels.memoryChannel.transactionCapacity = 1000
>>
>>
>> agent1.sources.r1.channels = memoryChannel
>> agent1.sinks.k1.channel = memoryChannel
>>
>> I run the server with this bat file:
>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\
>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
>> set JAVA_OPTS=-Xmx60m
>> set CONF=%FLUME_HOME%\conf\httppost.conf
>> set AGENT=agent1
>>
>> %JAVA% %JAVA_OPTS%
>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp
>> "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n %AGENT%
>>
>> netstat does not show any listener on port 9003? Is this normal or there
>> must be a listener?
>>
>> I run avro client with this bat file:
>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin
>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
>> set JAVA_OPTS=-Xmx60m
>> set AVRO_HOST=localhost
>> set AVRO_PORT=9003
>> set SENDFILE=%FLUME_HOME%\logs\data.log
>> set HEADFILE=%FLUME_HOME%\conf\header.txt
>>
>> %JAVA%  %JAVA_OPTS%
>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp
>> "%FLUME_HOME%\lib\*" org.apache.flume.client.avro.AvroCLIClient -H
>> %AVRO_HOST% -p %AVRO_PORT%  -F %SENDFILE%  --headerFile %HEADFILE%
>>
>>
>> Then I get this output:
>> C:\Projects\nickyk\apache-flume-1.3.1-bin>"C:\Java\jdk1.6.0_45\bin\java.exe"
>> -Xmx60m
>> -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\log4j.properties
>> -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\lib\*"
>> org.apache.flume.client.avro.AvroCLIClient -H  localhost -p 9003  -F
>> C:\Projects\nickyk\apache-flume-1.3.1-bin\logs\data.log --headerFile
>> C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\header.txt
>> 2013-06-20 17:55:58,285 (main) [DEBUG -
>> org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)]
>> Inserting Header Key [filename] header value [datalog]
>> 2013-06-20 17:55:58,287 (main) [DEBUG -
>> org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)]
>> Inserting Header Key [host] header value [zhouhh]
>> 2013-06-20 17:55:58,295 (main) [DEBUG -
>> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)]
>> Batch size string = 5
>> 2013-06-20 17:55:59,358 (main) [ERROR -
>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:79)]
>> Unable to open connection to Flume. Exception follows.
>> org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost,
>> port: 9003 }: RPC connection error
>>         at
>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>>         at
>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>>         at
>> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>>         at
>> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>>         at
>> org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>>         at
>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>> Caused by: java.io.IOException: Error connecting to localhost/
>> 127.0.0.1:9003
>>         at
>> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>>         at
>> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>>         at
>> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>>         at
>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>>         ... 5 more
>> Caused by: java.net.ConnectException: Connection refused: no further
>> information
>>         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>         at
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
>>         at
>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>>         at
>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>>         at
>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>>         at
>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>         at java.lang.Thread.run(Thread.java:662)
>> 2013-06-20 17:55:59,369 (main) [DEBUG -
>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:84)]
>> Exiting
>>
>>
>> My question is what I am doing wrong and what I need to test in order to
>> fix this situatioin.
>>
>> Thanks in advance.
>>
>> best regards,
>>
>>
>>
>


-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Mime
View raw message