flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shengyi.pan" <shengyi....@gmail.com>
Subject Re: Re: The flume exec source can not get content while using python command
Date Fri, 10 Oct 2014 03:24:41 GMT
I had found the issue reason, it's caused by Python stdout buffer strategy.

the standard input and output streams are fully buffered if and only if the streams do not
to refer to an interactive device.
                                                       -- from libc stdio manual 

So if you want to get Python stdout content while using exec source,  should call sys.stdout.flush()
after print someting...

Other ways:

1.  python -u , Force stdin, stdout and stderr to be totally unbuffered.

2. declare sys.stdout as sys.stdout = os.fdopen(sys.stdout.fileno(), ‘w’, 0)




2014-10-10



shengyi.pan



发件人:Roshan Naik <roshan@hortonworks.com>
发送时间:2014-10-01 03:46
主题:Re: The flume exec source can not get content while using python command
收件人:"user@flume.apache.org"<user@flume.apache.org>
抄送:

i use python with exec source quite often.  see if putting a full path to python helps...
/usr/bin/python or whatever it is on ur system.


-roshan


On Thu, Sep 18, 2014 at 5:30 PM, shengyi.pan <shengyi.pan@gmail.com> wrote:

  
hi, all:

I want to use exec source under Flume 1.5.0.  Follow the flume user guide, tail command is
running successfully on my computer, the flume.conf is :

a1.sources.r1.type = exec
      a1.sources.r1.command = tail -F /var/log/secure

When I want to run Python script,  change the flume.conf as follows:

a1.sources.r1.type = exec
      a1.sources.r1.command = python /home/test.py

The  test.py is very simpe,  just print string line periodically

cat test_flume.py 

#!/usr/bin/python

import sys
import time

sys.stdout.write("gogogo\n")
while True:
           sys.stdout.write("gogogo\n")
           time.sleep(3)

Start the flume with: nohup bin/flume-ng agent --conf ./conf/ -f ./conf/flume.conf -n a1 &
The flume is running and the python subprocess is existed, but no data is catched from command
: python /home/test.py, looks like Flume cann't read anything from Python stdout.

Can someone help me ? T_T   Thank you very much!




2014-09-18



shengyi.pan

2014-09-19



shengyi.pan



CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed
and may contain information that is confidential, privileged and exempt from disclosure under
applicable law. If the reader of this message is not the intended recipient, you are hereby
notified that any printing, copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have received this communication in error,
please contact the sender immediately and delete it from your system. Thank You. 
Mime
View raw message