flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gonzalo Herreros <gherre...@gmail.com>
Subject Re: Flume-agent conf on ambari
Date Mon, 16 Nov 2015 09:57:54 GMT
That is not on hdfs, it the home folder of the hdfs user on linux; it's
just checking your Flume configuration for changes, don't worry about it.

Use the FileSink and see if it works so you can know if the issue is hdfs
related or not.
I think it's more likely your issue is syslog related.

On 16 November 2015 at 08:47, zaenal rifai <togatta.fudo@gmail.com> wrote:

> thanks gonzalo, it's running but still cant write on hdfs.
>
> the error is
> http://tempel.blankon.in/1527360
>
> when Starting connection handler
>
> it always Checking file:/home/hdfs/adaw.conf for changes
>
> and i check on hdfs, there is no file
>
> On 16 November 2015 at 15:34, Gonzalo Herreros <gherreros@gmail.com>
> wrote:
>
>> I see your problem now. You are using "memory-channel" and "memoryChannel"
>> to refer to the same thing.
>> Change the 3rd line to:
>>
>> hdfs-agent.channels= memoryChannel
>>
>> On 16 November 2015 at 08:31, zaenal rifai <togatta.fudo@gmail.com>
>> wrote:
>>
>>> 1.Yes i get error, the error is " Agent configuration for hdfs-agent
>>> does not contain any valid channels. Marking it as invalid "
>>>     here is for more about the error
>>>
>>>     http://tempel.blankon.in/1527288
>>>
>>> 2.No, i don't have enable the kerberos
>>>
>>> 3.i'm not sure about the problem, i tried another method sink and it
>>> still failed.
>>>
>>> can you give me some simple example for agent configuration to collect
>>> log and write on hdfs ?
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 16 November 2015 at 14:47, Gonzalo Herreros <gherreros@gmail.com>
>>> wrote:
>>>
>>>> I looks fine to me. Do you get any errors?
>>>> Do you have kerberos enabled? maybe is a security issue
>>>> Are you sure the problem is hdfs and not some netcat error? Try with
>>>> another sink to confirm that.
>>>>
>>>> Regards,
>>>> Gonzalo
>>>>
>>>> On 16 November 2015 at 07:35, zaenal rifai <togatta.fudo@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello guys, i'm newbie on flume, i want to ask something about
>>>>> configuration on flume-agent which can write event on HDFS.
>>>>>
>>>>> for example which i can find on internet is like this.
>>>>>
>>>>> hdfs-agent.sources= netcat-collect
>>>>> hdfs-agent.sinks = hdfs-write
>>>>> hdfs-agent.channels= memory-channel
>>>>> hdfs-agent.sources.netcat-collect.type = netcat
>>>>> hdfs-agent.sources.netcat-collect.bind = 127.0.0.1
>>>>> hdfs-agent.sources.netcat-collect.port = 11111
>>>>> hdfs-agent.sinks.hdfs-write.type = hdfs
>>>>> hdfs-agent.sinks.hdfs-write.hdfs.path =/user/user/togatta/
>>>>> hdfs-agent.sinks.hdfs-write.rollInterval = 10
>>>>> hdfs-agent.sinks.hdfs-write.hdfs.writeFormat=Text
>>>>> hdfs-agent.sinks.hdfs-write.hdfs.fileType=DataStream
>>>>> hdfs-agent.channels.memoryChannel.type = memory
>>>>> hdfs-agent.channels.memoryChannel.capacity=10000
>>>>> hdfs-agent.sources.netcat-collect.channels=memoryChannel
>>>>> hdfs-agent.sinks.hdfs-write.channel=memoryChannel
>>>>>
>>>>> this configuration will collect about log netcat event, but when i run
>>>>> it, it cant write on path hdfs which i had to config. So any suggest
from
>>>>> all of you guys?
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message