flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raymond Ng <raymond...@gmail.com>
Subject Re: issue with hdfs sink config
Date Fri, 14 Sep 2012 07:35:27 GMT
Hi Jeff

removing the port number didn't work

what I should have mentioned was that I'm using flume 1.2, the same config
I mentioned below works for flume 1.1, so I'm not sure what has changed in
1.2 for the configs and I need to use 1.2 because of the File channel
I did see HDFSSinkConfiguration mentioned in SinkConfiguration.java but I
can't find this class in the flume 1.2 jars, and ClassNotFoundException is
being wrapped up in ConfigurationException which doesn't get reported in
the log


Ray


On Thu, Sep 13, 2012 at 5:28 PM, Jeff Lord <jeffrey.lord@gmail.com> wrote:

> Raymond,
>
> I don't think you need the port 50070 on your hdfs path.
> Try setting this to the fqdn of your hdfs namenode, leave off the port
> 50070 (this is just for the webui)
>
> -Jeff
>
> On Thu, Sep 13, 2012 at 9:04 AM, Raymond Ng <raymondair@gmail.com> wrote:
>
>> Hi
>>
>> I'm baffled as to what the problem is with my hdfs config below
>>
>> agent.sinks.probeHdfsSink2.type = hdfs
>> agent.sinks.probeHdfsSink2.hdfs.path =
>> hdfs://master:50070/user/test/data/flume/%{client}
>> agent.sinks.probeHdfsSink2.hdfs.filePrefix = %{site}_sessionData_%Y%m%d
>> agent.sinks.probeHdfsSink2.channel = probeFileChannel2
>> agent.sinks.probeHdfsSink2.hdfs.rollInterval = 120
>> agent.sinks.probeHdfsSink2.hdfs.rollSize = 209715200
>> agent.sinks.probeHdfsSink2.hdfs.rollCount = 0
>> agent.sinks.probeHdfsSink2.hdfs.batchSize = 100000
>> agent.sinks.probeHdfsSink2.hdfs.writeFormat = Text
>> agent.sinks.probeHdfsSink2.hdfs.fileType = DataStream
>> the flume log below does not provide enough info to help identify the
>> problem, FlumeConfiguration.AgentConfiguration holds a list of
>> <FlumeConfigurationError> but none of them get displayed in the log.
>>
>> If I replace hdfs sink with a file sink then everything works, so the
>> problem is definitely in my hdfs configs but I can't see where?  any advice
>> is appreciated
>>
>> 2012-09-13 16:49:43,145  INFO [main] LifecycleSupervisor.java - Starting
>> lifecycle supervisor 1
>> 2012-09-13 16:49:43,151  INFO [main] FlumeNode.java - Flume node starting
>> - agent
>> 2012-09-13 16:49:43,156  INFO [lifecycleSupervisor-1-0]
>> DefaultLogicalNodeManager.java - Node manager starting
>> 2012-09-13 16:49:43,156  INFO [lifecycleSupervisor-1-2]
>> AbstractFileConfigurationProvider.java - Configuration provider starting
>> 2012-09-13 16:49:43,157  INFO [lifecycleSupervisor-1-0]
>> LifecycleSupervisor.java - Starting lifecycle supervisor 11
>> 2012-09-13 16:49:43,158 DEBUG [lifecycleSupervisor-1-0]
>> DefaultLogicalNodeManager.java - Node manager started
>> 2012-09-13 16:49:43,160 DEBUG [lifecycleSupervisor-1-2]
>> AbstractFileConfigurationProvider.java - Configuration provider started
>> 2012-09-13 16:49:43,160 DEBUG [conf-file-poller-0]
>> AbstractFileConfigurationProvider.java - Checking
>> file:/usr/local/lib/flume-ng/conf/acme-agent.conf for changes
>> 2012-09-13 16:49:43,160  INFO [conf-file-poller-0]
>> AbstractFileConfigurationProvider.java - Reloading configuration
>> file:/usr/local/lib/flume-ng/conf/acme-agent.conf
>> 2012-09-13 16:49:43,170  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Added sinks: probeHdfsSink2 Agent: agent
>> 2012-09-13 16:49:43,171  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Processing:probeHdfsSink2
>> 2012-09-13 16:49:43,171 DEBUG [conf-file-poller-0]
>> FlumeConfiguration.java - Created context for probeHdfsSink2: hdfs.rollSize
>> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Processing:probeHdfsSink2
>> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Processing:probeHdfsSink2
>> 2012-09-13 16:49:43,172  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Processing:probeFileSink
>> 2012-09-13 16:49:43,172 DEBUG [conf-file-poller-0]
>> FlumeConfiguration.java - Created context for probeFileSink:
>> sink.rollInterval
>> 2012-09-13 16:49:43,176  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Processing:probeHdfsSink2
>> 2012-09-13 16:49:43,176  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Processing:probeHdfsSink2
>> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Processing:probeHdfsSink2
>> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Processing:probeFileSink
>> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Processing:probeFileSink
>> 2012-09-13 16:49:43,177  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Processing:probeHdfsSink2
>> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Processing:probeFileSink
>> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Processing:probeHdfsSink2
>> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Processing:probeHdfsSink2
>> 2012-09-13 16:49:43,178  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Processing:probeHdfsSink2
>> 2012-09-13 16:49:43,179  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Processing:probeFileSink
>> 2012-09-13 16:49:43,179 DEBUG [conf-file-poller-0]
>> FlumeConfiguration.java - Starting validation of configuration for agent:
>> agent, initial-configuration: AgentConfiguration[agent]
>> SOURCES: {probeAvroSrc2={ parameters:{port=5556,
>> channels=probeFileChannel2, type=avro, bind=192.168.200.170} }}
>> CHANNELS: {probeFileChannel2={
>> parameters:{checkpointDir=/home/test/flume-ng/filechannel2/checkpoint,
>> dataDirs=/home/test/flume-ng/filechannel2/data, write-timeout=3,
>> transactionCapacity=200000, checkpointInterval=10, capacity=1000000,
>> keep-alive=3, maxFileSize=1073741824, type=FILE} }}
>> SINKS: {probeHdfsSink2={ parameters:{hdfs.fileType=DataStream,
>> hdfs.path=hdfs://master:50070/user/test/data/flume/%{client},
>> hdfs.batchSize=100000, hdfs.rollInterval=120, hdfs.rollSize=209715200,
>> hdfs.filePrefix=%{site}_sessionData_%Y%m%d, hdfs.writeFormat=Text,
>> type=hdfs, channel=probeFileChannel2, hdfs.rollCount=0} }, probeFileSink={
>> parameters:{sink.serializer=TEXT, sink.rollInterval=0, type=FILE_ROLL,
>> channel=probeFileChannel2, sink.directory=/home/test/flume/filesink} }}
>> 2012-09-13 16:49:43,188 DEBUG [conf-file-poller-0]
>> FlumeConfiguration.java - Created channel probeFileChannel2
>> 2012-09-13 16:49:43,214 DEBUG [conf-file-poller-0]
>> FlumeConfiguration.java - Creating sink: probeHdfsSink2 using HDFS
>> 2012-09-13 16:49:43,218 DEBUG [conf-file-poller-0]
>> FlumeConfiguration.java - Post validation configuration for agent
>> AgentConfiguration created without Configuration stubs for which only
>> basic syntactical validation was performed[agent]
>> SOURCES: {probeAvroSrc2={ parameters:{port=5556,
>> channels=probeFileChannel2, type=avro, bind=192.168.200.170} }}
>> CHANNELS: {probeFileChannel2={
>> parameters:{checkpointDir=/home/test/flume-ng/filechannel2/checkpoint,
>> dataDirs=/home/test/flume-ng/filechannel2/data, write-timeout=3,
>> transactionCapacity=200000, checkpointInterval=10, capacity=1000000,
>> keep-alive=3, maxFileSize=1073741824, type=FILE} }}
>> SINKS: {probeHdfsSink2={ parameters:{hdfs.fileType=DataStream,
>> hdfs.path=hdfs://master:50070/user/test/data/flume/%{client},
>> hdfs.batchSize=100000, hdfs.rollInterval=120, hdfs.rollSize=209715200,
>> hdfs.filePrefix=%{site}_sessionData_%Y%m%d, hdfs.writeFormat=Text,
>> type=hdfs, channel=probeFileChannel2, hdfs.rollCount=0} }}
>> 2012-09-13 16:49:43,218 DEBUG [conf-file-poller-0]
>> FlumeConfiguration.java - Channels:probeFileChannel2
>> 2012-09-13 16:49:43,219 DEBUG [conf-file-poller-0]
>> FlumeConfiguration.java - Sinks probeHdfsSink2
>> 2012-09-13 16:49:43,219 DEBUG [conf-file-poller-0]
>> FlumeConfiguration.java - Sources probeAvroSrc2
>> 2012-09-13 16:49:43,219  INFO [conf-file-poller-0]
>> FlumeConfiguration.java - Post-validation flume configuration contains
>> configuration  for agents: [agent]
>> 2012-09-13 16:49:43,219  INFO [conf-file-poller-0]
>> PropertiesFileConfigurationProvider.java - Creating channels
>> 2012-09-13 16:49:43,220 DEBUG [conf-file-poller-0]
>> DefaultChannelFactory.java - Creating instance of channel probeFileChannel2
>> type FILE
>> 2012-09-13 16:49:43,236  INFO [conf-file-poller-0]
>> MonitoredCounterGroup.java - Monitoried counter group for type: CHANNEL,
>> name: probeFileChannel2, registered successfully.
>> 2012-09-13 16:49:43,236  INFO [conf-file-poller-0]
>> PropertiesFileConfigurationProvider.java - created channel probeFileChannel2
>> 2012-09-13 16:49:43,236 DEBUG [conf-file-poller-0]
>> DefaultSourceFactory.java - Creating instance of source probeAvroSrc2, type
>> avro
>> 2012-09-13 16:49:43,256  INFO [conf-file-poller-0]
>> MonitoredCounterGroup.java - Monitoried counter group for type: SOURCE,
>> name: probeAvroSrc2, registered successfully.
>> 2012-09-13 16:49:43,277  INFO [conf-file-poller-0]
>> DefaultSinkFactory.java - Creating instance of sink: probeHdfsSink2, type:
>> hdfs
>>
>>
>> --
>> Rgds
>> Ray
>>
>
>


-- 
Rgds
Ray

Mime
View raw message