flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zaenal rifai <togatta.f...@gmail.com>
Subject Re: Flume-agent conf on ambari
Date Tue, 17 Nov 2015 06:36:05 GMT
thanks gonzalo

you are my man :D

On 16 November 2015 at 16:57, Gonzalo Herreros <gherreros@gmail.com> wrote:

> That is not on hdfs, it the home folder of the hdfs user on linux; it's
> just checking your Flume configuration for changes, don't worry about it.
>
> Use the FileSink and see if it works so you can know if the issue is hdfs
> related or not.
> I think it's more likely your issue is syslog related.
>
> On 16 November 2015 at 08:47, zaenal rifai <togatta.fudo@gmail.com> wrote:
>
>> thanks gonzalo, it's running but still cant write on hdfs.
>>
>> the error is
>> http://tempel.blankon.in/1527360
>>
>> when Starting connection handler
>>
>> it always Checking file:/home/hdfs/adaw.conf for changes
>>
>> and i check on hdfs, there is no file
>>
>> On 16 November 2015 at 15:34, Gonzalo Herreros <gherreros@gmail.com>
>> wrote:
>>
>>> I see your problem now. You are using "memory-channel" and "memoryChannel"
>>> to refer to the same thing.
>>> Change the 3rd line to:
>>>
>>> hdfs-agent.channels= memoryChannel
>>>
>>> On 16 November 2015 at 08:31, zaenal rifai <togatta.fudo@gmail.com>
>>> wrote:
>>>
>>>> 1.Yes i get error, the error is " Agent configuration for hdfs-agent
>>>> does not contain any valid channels. Marking it as invalid "
>>>>     here is for more about the error
>>>>
>>>>     http://tempel.blankon.in/1527288
>>>>
>>>> 2.No, i don't have enable the kerberos
>>>>
>>>> 3.i'm not sure about the problem, i tried another method sink and it
>>>> still failed.
>>>>
>>>> can you give me some simple example for agent configuration to collect
>>>> log and write on hdfs ?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 16 November 2015 at 14:47, Gonzalo Herreros <gherreros@gmail.com>
>>>> wrote:
>>>>
>>>>> I looks fine to me. Do you get any errors?
>>>>> Do you have kerberos enabled? maybe is a security issue
>>>>> Are you sure the problem is hdfs and not some netcat error? Try with
>>>>> another sink to confirm that.
>>>>>
>>>>> Regards,
>>>>> Gonzalo
>>>>>
>>>>> On 16 November 2015 at 07:35, zaenal rifai <togatta.fudo@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello guys, i'm newbie on flume, i want to ask something about
>>>>>> configuration on flume-agent which can write event on HDFS.
>>>>>>
>>>>>> for example which i can find on internet is like this.
>>>>>>
>>>>>> hdfs-agent.sources= netcat-collect
>>>>>> hdfs-agent.sinks = hdfs-write
>>>>>> hdfs-agent.channels= memory-channel
>>>>>> hdfs-agent.sources.netcat-collect.type = netcat
>>>>>> hdfs-agent.sources.netcat-collect.bind = 127.0.0.1
>>>>>> hdfs-agent.sources.netcat-collect.port = 11111
>>>>>> hdfs-agent.sinks.hdfs-write.type = hdfs
>>>>>> hdfs-agent.sinks.hdfs-write.hdfs.path =/user/user/togatta/
>>>>>> hdfs-agent.sinks.hdfs-write.rollInterval = 10
>>>>>> hdfs-agent.sinks.hdfs-write.hdfs.writeFormat=Text
>>>>>> hdfs-agent.sinks.hdfs-write.hdfs.fileType=DataStream
>>>>>> hdfs-agent.channels.memoryChannel.type = memory
>>>>>> hdfs-agent.channels.memoryChannel.capacity=10000
>>>>>> hdfs-agent.sources.netcat-collect.channels=memoryChannel
>>>>>> hdfs-agent.sinks.hdfs-write.channel=memoryChannel
>>>>>>
>>>>>> this configuration will collect about log netcat event, but when
i
>>>>>> run it, it cant write on path hdfs which i had to config. So any
suggest
>>>>>> from all of you guys?
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message