flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Alten-Lorenz <wget.n...@gmail.com>
Subject Re: flume to collect log files from various sources
Date Tue, 29 Jan 2013 07:37:41 GMT
He's using flume-1.3.1, not 0.x:
C:\apache\apache-flume-1.3.1-bin

Yes, flume runs on Windows, just compile it.

@ Yogi:
15.80.67.94:61452 => /16.90.218.66:44444] BOUND: /15.80.67.94:61452
log4j:ERROR Flume append() failed.

Can you connect per Telnet to the ports you have defined? What says netstat (or a similar
tool on Windows to show open ports). Firewall disabled?

- Alex

On Jan 29, 2013, at 4:40 AM, Sri Ramya <ramya.1255@gmail.com> wrote:

> In side flume there is a folder conf/. in that u will find the
> flume_site.xml
> and bin/ you find binaries. download flume from this link
> http://archive.cloudera.com/cdh/3/flume-0.9.4-cdh3u4.tar.gz
> 
> and follow the flume user guide. u will able to connect .
> 
> 
> On Mon, Jan 28, 2013 at 9:59 PM, yogi nerella <ynerella999@gmail.com> wrote:
> 
>> Hello Ramya,
>> 
>> The above link shows example commands to configure master, node etc.,
>> But I donot see any "flume" binary?  or "config" binary?
>> or flume_site.xml file in OOB flume-1.3.1-bin or flume-1.4.0-bin.
>> 
>> 
>> Am I missing something?
>> How do I set my agent's as nodes, and my collector's as masters etc?
>> 
>> 
>> Thanks,
>> Yogi
>> 
>> 
>> 
>> 
>> On Sun, Jan 27, 2013 at 10:35 PM, Sri Ramya <ramya.1255@gmail.com> wrote:
>> 
>>> sorry for late reply
>>> http://archive.cloudera.com/cdh/3/flume/UserGuide/
>>> check out this u will get all details about how to configure. And i never
>>> tried  log4jappender.
>>> 
>>> 
>>> On Thu, Jan 24, 2013 at 10:18 PM, yogi nerella <ynerella999@gmail.com>wrote:
>>> 
>>>> Hello Ramya,
>>>> 
>>>> Flume agent have to be installed on every host?
>>>>    I want to avoid this by integrating log4jappender into my app, and
>>>> to remove one additional overhead for administrators.
>>>>    Is this a problem?
>>>> 
>>>> 
>>>> Collecting data using "taildir" command?
>>>>   1.   If I have one flume agent sending log files from one host to
>>>> another agent who is listening.
>>>>    2.   How do I configure the receiving agent such that, it can
>>>> separate the log files again and write as it recieved.
>>>> 
>>>> 
>>>> Any sample configuration files for either 'taildir'  or 'multitail'
>>>> commands?
>>>> 
>>>> Issue with log4jappender:
>>>> 
>>>> I am evaluating log4jappender, and using this I am not able to send
>>>> events to the flume agent.
>>>> 
>>>> When I turn on debugging all the log messages I see are as follows.
>>>> 
>>>> Classpath:
>>>> C:\ServiceManager\workspace\mongodb\Mongodb\bin;C:\apache\apache-flume-1.3.1-bin\lib\avro-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\avro-ipc-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-core-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-log4jappender-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-sdk-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-core-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-mapper-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\netty-3.4.0.Final.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-api-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-log4j12-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\log4j-1.2.16.jar
>>>> 14:27:29,600 DEBUG NettyAvroRpcClient:420 - Batch size string = 0
>>>> 14:27:29,604  WARN NettyAvroRpcClient:426 - Invalid value for batchSize:
>>>> 0; Using default value.
>>>> 14:27:29,649 DEBUG NettyTransceiver:195 - Using Netty bootstrap options:
>>>> {tcpNoDelay=true, connectTimeoutMillis=20000}
>>>> 14:27:29,649 DEBUG NettyTransceiver:252 - Connecting to /
>>>> 16.90.218.66:44444
>>>> 14:27:29,675 DEBUG NettyTransceiver:491 - [id: 0x01480773] OPEN
>>>> 14:27:29,699 DEBUG NettyTransceiver:491 - [id: 0x01480773, /
>>>> 15.80.67.94:61452 => /16.90.218.66:44444] BOUND: /15.80.67.94:61452
>>>> log4j:ERROR Flume append() failed.
>>>> 14:27:29,940  INFO LogTest:13 - main started at Wed Jan 23 14:27:29 PST
>>>> 2013
>>>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>>>> 16.90.218.66/16.90.218.66:44444
>>>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>>>> 16.90.218.66/16.90.218.66:44444
>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>> 16.90.218.66/16.90.218.66:44444
>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>> 16.90.218.66/16.90.218.66:44444
>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>> 16.90.218.66/16.90.218.66:44444
>>>> 
>>>> 
>>>> Yogi
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Wed, Jan 23, 2013 at 8:57 PM, Sri Ramya <ramya.1255@gmail.com> wrote:
>>>> 
>>>>> there is an error displayed on console Flume append() failed. Try to
>>>>> resolve .
>>>>> explain me what is your exact problem.
>>>>> 
>>>>> Note:
>>>>> 1, Flume agent have to install in every host from where u want to
>>>>> collect data.
>>>>> 2, If u want to collect data from a directory then you have to to use
>>>>> 'taildir' command.
>>>>> 3, If u want to collect data from different files u have to use
>>>>> 'multitail' command.
>>>>> 
>>>>> I think this info will be use full for u.
>>>>> thank you
>>>>> 
>>>>> 
>>>>> 
>>>>> On Thu, Jan 24, 2013 at 4:06 AM, yogi nerella <ynerella999@gmail.com>wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I am a new bee for flume, would like to get some feedback on if what
I
>>>>>> am doing works correctly.
>>>>>> 
>>>>>> 
>>>>>> My application runs on multiple hosts, I want to collect all log
files
>>>>>> to a central location.
>>>>>> 
>>>>>> 1.  In My application I will ship all relevant log4jappender jar
>>>>>> files, and configure the agent's host, port information.
>>>>>> 
>>>>>> 2. I will run a simple agent, and configure a source (avro), channel
>>>>>> (memory), and sink (file_roll).
>>>>>> 
>>>>>> File_roll will write the events recieved via AVRO source and write
>>>>>> them to the corresponding file_roll sink.
>>>>>> 
>>>>>> 3. Do I need one agent for each host, or is there a way I can
>>>>>> configure to write to multiple files, dynamically based on the real
source
>>>>>> the message is getting recieved.
>>>>>> 
>>>>>> 4. If I modify the configuration file, will the agent re-read the
>>>>>> information.   (Ex:  I want to add a new host, and collect the log
files to
>>>>>> a new directory).
>>>>>> 
>>>>>> 5.  In my simple test, messages sent from my app are getting lost,
and
>>>>>> I see the following information in the console.
>>>>>> 
>>>>>> 
>>>>>> Classpath:
>>>>>> C:\ServiceManager\workspace\mongodb\Mongodb\bin;C:\apache\apache-flume-1.3.1-bin\lib\avro-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\avro-ipc-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-core-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-log4jappender-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-sdk-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-core-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-mapper-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\netty-3.4.0.Final.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-api-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-log4j12-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\log4j-1.2.16.jar
>>>>>> 14:27:29,600 DEBUG NettyAvroRpcClient:420 - Batch size string = 0
>>>>>> 14:27:29,604  WARN NettyAvroRpcClient:426 - Invalid value for
>>>>>> batchSize: 0; Using default value.
>>>>>> 14:27:29,649 DEBUG NettyTransceiver:195 - Using Netty bootstrap
>>>>>> options: {tcpNoDelay=true, connectTimeoutMillis=20000}
>>>>>> 14:27:29,649 DEBUG NettyTransceiver:252 - Connecting to /
>>>>>> 16.90.218.66:44444
>>>>>> 14:27:29,675 DEBUG NettyTransceiver:491 - [id: 0x01480773] OPEN
>>>>>> 14:27:29,699 DEBUG NettyTransceiver:491 - [id: 0x01480773, /
>>>>>> 15.80.67.94:61452 => /16.90.218.66:44444] BOUND: /15.80.67.94:61452
>>>>>> log4j:ERROR Flume append() failed.
>>>>>> 14:27:29,940  INFO LogTest:13 - main started at Wed Jan 23 14:27:29
>>>>>> PST 2013
>>>>>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>>>>>> 16.90.218.66/16.90.218.66:44444
>>>>>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>>>>>> 16.90.218.66/16.90.218.66:44444
>>>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>>>> 16.90.218.66/16.90.218.66:44444
>>>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>>>> 16.90.218.66/16.90.218.66:44444
>>>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>>>> 16.90.218.66/16.90.218.66:44444
>>>>>> 
>>>>>> 
>>>>>> Appreciate your help
>>>>>> Yogi
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF


Mime
View raw message