flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raymond Ng <raymond...@gmail.com>
Subject issue with hdfs sink config
Date Thu, 13 Sep 2012 16:04:45 GMT
Hi

I'm baffled as to what the problem is with my hdfs config below

agent.sinks.probeHdfsSink2.type = hdfs
agent.sinks.probeHdfsSink2.hdfs.path =
hdfs://master:50070/user/test/data/flume/%{client}
agent.sinks.probeHdfsSink2.hdfs.filePrefix = %{site}_sessionData_%Y%m%d
agent.sinks.probeHdfsSink2.channel = probeFileChannel2
agent.sinks.probeHdfsSink2.hdfs.rollInterval = 120
agent.sinks.probeHdfsSink2.hdfs.rollSize = 209715200
agent.sinks.probeHdfsSink2.hdfs.rollCount = 0
agent.sinks.probeHdfsSink2.hdfs.batchSize = 100000
agent.sinks.probeHdfsSink2.hdfs.writeFormat = Text
agent.sinks.probeHdfsSink2.hdfs.fileType = DataStream
the flume log below does not provide enough info to help identify the
problem, FlumeConfiguration.AgentConfiguration holds a list of
<FlumeConfigurationError> but none of them get displayed in the log.

If I replace hdfs sink with a file sink then everything works, so the
problem is definitely in my hdfs configs but I can't see where?  any advice
is appreciated

2012-09-13 16:49:43,145  INFO [main] LifecycleSupervisor.java - Starting
lifecycle supervisor 1
2012-09-13 16:49:43,151  INFO [main] FlumeNode.java - Flume node starting -
agent
2012-09-13 16:49:43,156  INFO [lifecycleSupervisor-1-0]
DefaultLogicalNodeManager.java - Node manager starting
2012-09-13 16:49:43,156  INFO [lifecycleSupervisor-1-2]
AbstractFileConfigurationProvider.java - Configuration provider starting
2012-09-13 16:49:43,157  INFO [lifecycleSupervisor-1-0]
LifecycleSupervisor.java - Starting lifecycle supervisor 11
2012-09-13 16:49:43,158 DEBUG [lifecycleSupervisor-1-0]
DefaultLogicalNodeManager.java - Node manager started
2012-09-13 16:49:43,160 DEBUG [lifecycleSupervisor-1-2]
AbstractFileConfigurationProvider.java - Configuration provider started
2012-09-13 16:49:43,160 DEBUG [conf-file-poller-0]
AbstractFileConfigurationProvider.java - Checking
file:/usr/local/lib/flume-ng/conf/acme-agent.conf for changes
2012-09-13 16:49:43,160  INFO [conf-file-poller-0]
AbstractFileConfigurationProvider.java - Reloading configuration
file:/usr/local/lib/flume-ng/conf/acme-agent.conf
2012-09-13 16:49:43,170  INFO [conf-file-poller-0] FlumeConfiguration.java
- Added sinks: probeHdfsSink2 Agent: agent
2012-09-13 16:49:43,171  INFO [conf-file-poller-0] FlumeConfiguration.java
- Processing:probeHdfsSink2
2012-09-13 16:49:43,171 DEBUG [conf-file-poller-0] FlumeConfiguration.java
- Created context for probeHdfsSink2: hdfs.rollSize
2012-09-13 16:49:43,172  INFO [conf-file-poller-0] FlumeConfiguration.java
- Processing:probeHdfsSink2
2012-09-13 16:49:43,172  INFO [conf-file-poller-0] FlumeConfiguration.java
- Processing:probeHdfsSink2
2012-09-13 16:49:43,172  INFO [conf-file-poller-0] FlumeConfiguration.java
- Processing:probeFileSink
2012-09-13 16:49:43,172 DEBUG [conf-file-poller-0] FlumeConfiguration.java
- Created context for probeFileSink: sink.rollInterval
2012-09-13 16:49:43,176  INFO [conf-file-poller-0] FlumeConfiguration.java
- Processing:probeHdfsSink2
2012-09-13 16:49:43,176  INFO [conf-file-poller-0] FlumeConfiguration.java
- Processing:probeHdfsSink2
2012-09-13 16:49:43,177  INFO [conf-file-poller-0] FlumeConfiguration.java
- Processing:probeHdfsSink2
2012-09-13 16:49:43,177  INFO [conf-file-poller-0] FlumeConfiguration.java
- Processing:probeFileSink
2012-09-13 16:49:43,177  INFO [conf-file-poller-0] FlumeConfiguration.java
- Processing:probeFileSink
2012-09-13 16:49:43,177  INFO [conf-file-poller-0] FlumeConfiguration.java
- Processing:probeHdfsSink2
2012-09-13 16:49:43,178  INFO [conf-file-poller-0] FlumeConfiguration.java
- Processing:probeFileSink
2012-09-13 16:49:43,178  INFO [conf-file-poller-0] FlumeConfiguration.java
- Processing:probeHdfsSink2
2012-09-13 16:49:43,178  INFO [conf-file-poller-0] FlumeConfiguration.java
- Processing:probeHdfsSink2
2012-09-13 16:49:43,178  INFO [conf-file-poller-0] FlumeConfiguration.java
- Processing:probeHdfsSink2
2012-09-13 16:49:43,179  INFO [conf-file-poller-0] FlumeConfiguration.java
- Processing:probeFileSink
2012-09-13 16:49:43,179 DEBUG [conf-file-poller-0] FlumeConfiguration.java
- Starting validation of configuration for agent: agent,
initial-configuration: AgentConfiguration[agent]
SOURCES: {probeAvroSrc2={ parameters:{port=5556,
channels=probeFileChannel2, type=avro, bind=192.168.200.170} }}
CHANNELS: {probeFileChannel2={
parameters:{checkpointDir=/home/test/flume-ng/filechannel2/checkpoint,
dataDirs=/home/test/flume-ng/filechannel2/data, write-timeout=3,
transactionCapacity=200000, checkpointInterval=10, capacity=1000000,
keep-alive=3, maxFileSize=1073741824, type=FILE} }}
SINKS: {probeHdfsSink2={ parameters:{hdfs.fileType=DataStream,
hdfs.path=hdfs://master:50070/user/test/data/flume/%{client},
hdfs.batchSize=100000, hdfs.rollInterval=120, hdfs.rollSize=209715200,
hdfs.filePrefix=%{site}_sessionData_%Y%m%d, hdfs.writeFormat=Text,
type=hdfs, channel=probeFileChannel2, hdfs.rollCount=0} }, probeFileSink={
parameters:{sink.serializer=TEXT, sink.rollInterval=0, type=FILE_ROLL,
channel=probeFileChannel2, sink.directory=/home/test/flume/filesink} }}
2012-09-13 16:49:43,188 DEBUG [conf-file-poller-0] FlumeConfiguration.java
- Created channel probeFileChannel2
2012-09-13 16:49:43,214 DEBUG [conf-file-poller-0] FlumeConfiguration.java
- Creating sink: probeHdfsSink2 using HDFS
2012-09-13 16:49:43,218 DEBUG [conf-file-poller-0] FlumeConfiguration.java
- Post validation configuration for agent
AgentConfiguration created without Configuration stubs for which only basic
syntactical validation was performed[agent]
SOURCES: {probeAvroSrc2={ parameters:{port=5556,
channels=probeFileChannel2, type=avro, bind=192.168.200.170} }}
CHANNELS: {probeFileChannel2={
parameters:{checkpointDir=/home/test/flume-ng/filechannel2/checkpoint,
dataDirs=/home/test/flume-ng/filechannel2/data, write-timeout=3,
transactionCapacity=200000, checkpointInterval=10, capacity=1000000,
keep-alive=3, maxFileSize=1073741824, type=FILE} }}
SINKS: {probeHdfsSink2={ parameters:{hdfs.fileType=DataStream,
hdfs.path=hdfs://master:50070/user/test/data/flume/%{client},
hdfs.batchSize=100000, hdfs.rollInterval=120, hdfs.rollSize=209715200,
hdfs.filePrefix=%{site}_sessionData_%Y%m%d, hdfs.writeFormat=Text,
type=hdfs, channel=probeFileChannel2, hdfs.rollCount=0} }}
2012-09-13 16:49:43,218 DEBUG [conf-file-poller-0] FlumeConfiguration.java
- Channels:probeFileChannel2
2012-09-13 16:49:43,219 DEBUG [conf-file-poller-0] FlumeConfiguration.java
- Sinks probeHdfsSink2
2012-09-13 16:49:43,219 DEBUG [conf-file-poller-0] FlumeConfiguration.java
- Sources probeAvroSrc2
2012-09-13 16:49:43,219  INFO [conf-file-poller-0] FlumeConfiguration.java
- Post-validation flume configuration contains configuration  for agents:
[agent]
2012-09-13 16:49:43,219  INFO [conf-file-poller-0]
PropertiesFileConfigurationProvider.java - Creating channels
2012-09-13 16:49:43,220 DEBUG [conf-file-poller-0]
DefaultChannelFactory.java - Creating instance of channel probeFileChannel2
type FILE
2012-09-13 16:49:43,236  INFO [conf-file-poller-0]
MonitoredCounterGroup.java - Monitoried counter group for type: CHANNEL,
name: probeFileChannel2, registered successfully.
2012-09-13 16:49:43,236  INFO [conf-file-poller-0]
PropertiesFileConfigurationProvider.java - created channel probeFileChannel2
2012-09-13 16:49:43,236 DEBUG [conf-file-poller-0]
DefaultSourceFactory.java - Creating instance of source probeAvroSrc2, type
avro
2012-09-13 16:49:43,256  INFO [conf-file-poller-0]
MonitoredCounterGroup.java - Monitoried counter group for type: SOURCE,
name: probeAvroSrc2, registered successfully.
2012-09-13 16:49:43,277  INFO [conf-file-poller-0] DefaultSinkFactory.java
- Creating instance of sink: probeHdfsSink2, type: hdfs


-- 
Rgds
Ray

Mime
View raw message