Hi Hari and the all members of this list,

Thanks for pointing me that error. This is actually my second attempt to use Apache Flume - with avro client. The symptoms are the same - the is no listener on the specified port and I am not able to connect.

I started to evaluate Flume just yesterday and many things are new to me and probably I do many mistakes. I want to point that I use Flume 1.3.1 compiled by me if this is important to be mentioned. I have tested too with the official binary from Flume web site but the results are the same.

Here is what I do to test HTTP source:

my config file (httppost.conf):
agent1.sources = r1
agent1.channels = ch1
agent1.sinks = k1

agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
agent1.sources.r1.port = 9001
agent1.sources.r1.channels = ch1
#agent1.sources.r1.handler = org.example.rest.RestHandler
agent1.sources.r1.handler = org.apache.flume.http.JSONHandler
agent1.sources.r1.handler.nickname = json handler

agent1.sinks.k1.type = file_roll
agent1.sinks.k1.channel = ch1
agent1.sinks.k1.sink.directory = C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log

agent1.channels.ch1.type = memory
agent1.channels.ch1.capacity = 10000
agent1.channels.ch1.transactionCapacity = 1000

my server bat file:
set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\
set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
set JAVA_OPTS=-Xmx60m
set CONF=%FLUME_HOME%\conf\httppost.conf
set AGENT=agent1

%JAVA% %JAVA_OPTS%  -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n %AGENT%

How I test:

server console:
>"C:\Java\jdk1.6.0_45\bin\java.exe" -Xmx60m  -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\log4j.properties -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\\lib\*" org.apache.flume.node.Application -f C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\httppost.conf -n agent1

client console:
>C:\Projects\nickyk\apache-flume-1.3.1-bin>curl -X POST -H "Content-Type: application/json" -d "[{ \"headers\" : { \"timestamp\" : \"434324343\", \"host\" : \"random_host.example.co
m\" }, \"body\" : \"random_body\" }, { \"headers\" : { \"namenode\" : \"namenode.example.com\", \"datanode\" : \"random_datanode.example.com\" }, \"body\" : \"really_random_body\"
}]" http://localhost:9001
curl: (7) Failed connect to localhost:9001; No error

If I understand correct the documentation I should expect that on the specified port (9001) it must be listener that will accept POST requests? Is this correct?
Do I miss some important aspects that are critical to run Flume?

Thanks in advance for your help.


best regards,
nickolay kolev



On Thu, Jun 20, 2013 at 9:07 PM, Hari Shreedharan <hshreedharan@cloudera.com> wrote:
Does not look like you are using the http source at all. Your source type needs to be HTTP


Cheers,
Hari

On Thursday, June 20, 2013 at 8:57 AM, Nickolay Kolev wrote:

Hi all,
I am new to flume and all that logging stuff and probably many things are unclear to me despite I read the docs.
I want to run Flume on windows server to create proof of concept solution. I was able to compile it successfully. I was able to make successful test with netcat.
Next test that i wanted to do is with http source but unfortunately it did not pass.
 Here is what I do:

my server config (httppost.conf):
agent1.sources = r1
agent1.channels = memoryChannel
agent1.sinks = k1

agent1.sources.r1.type = avro
agent1.sources.r1.bind = localhost
agent1.sources.r1.port = 9003

agent1.sinks.k1.type = file_roll
agent1.sinks.k1.sink.directory = C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log

agent1.channels.memoryChannel.type = memory
agent1.channels.memoryChannel.capacity = 10000
agent1.channels.memoryChannel.transactionCapacity = 1000


agent1.sources.r1.channels = memoryChannel
agent1.sinks.k1.channel = memoryChannel
 
I run the server with this bat file:
set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\
set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
set JAVA_OPTS=-Xmx60m
set CONF=%FLUME_HOME%\conf\httppost.conf
set AGENT=agent1

%JAVA% %JAVA_OPTS%  -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n %AGENT%

netstat does not show any listener on port 9003? Is this normal or there must be a listener?

I run avro client with this bat file:
set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin
set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
set JAVA_OPTS=-Xmx60m
set AVRO_HOST=localhost
set AVRO_PORT=9003
set SENDFILE=%FLUME_HOME%\logs\data.log
set HEADFILE=%FLUME_HOME%\conf\header.txt

%JAVA%  %JAVA_OPTS%  -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp "%FLUME_HOME%\lib\*" org.apache.flume.client.avro.AvroCLIClient -H  %AVRO_HOST% -p %AVRO_PORT%  -F %SENDFILE%  --headerFile %HEADFILE%


Then I get this output:
C:\Projects\nickyk\apache-flume-1.3.1-bin>"C:\Java\jdk1.6.0_45\bin\java.exe"  -Xmx60m  -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\log4j.properties  -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\lib\*" org.apache.flume.client.avro.AvroCLIClient -H  localhost -p 9003  -F C:\Projects\nickyk\apache-flume-1.3.1-bin\logs\data.log --headerFile C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\header.txt
2013-06-20 17:55:58,285 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)] Inserting Header Key [filename] header value [datalog]
2013-06-20 17:55:58,287 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)] Inserting Header Key [host] header value [zhouhh]
2013-06-20 17:55:58,295 (main) [DEBUG - org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)] Batch size string = 5
2013-06-20 17:55:59,358 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:79)] Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 9003 }: RPC connection error
        at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
        at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
        at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
        at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
        at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
        at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
Caused by: java.io.IOException: Error connecting to localhost/127.0.0.1:9003
        at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
        at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
        at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
        at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
        ... 5 more
Caused by: java.net.ConnectException: Connection refused: no further information
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
        at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
        at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
        at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)
2013-06-20 17:55:59,369 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:84)] Exiting


My question is what I am doing wrong and what I need to test in order to fix this situatioin.

Thanks in advance.

best regards,