flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: Error in Upload the log file into hdfs
Date Thu, 29 Nov 2012 15:54:02 GMT
Alex is right and our error message there needs much improvement. I have
created a JIRA here https://issues.apache.org/jira/browse/FLUME-1744

On Thu, Nov 29, 2012 at 9:39 AM, Alexander Alten-Lorenz <wget.null@gmail.com
> wrote:

> Hi,
>
> agent.channels.memoryChannel.transactionCapacity=1000
>
> Is wrong. You cant have equal or more transaction capacity in a channel as
> configured capacity. Use the opposite, when you want to use it.
> from our Guide:
>
> capacity                NUM     The max number of events stored in the
> channel
> transactionCapacity     NUM     The max number of events stored in the
> channel per transaction
>
> Try this:
>
> agent.channels.memoryChannel.capacity = 1000
> agent.channels.memoryChannel.transactionCapacity=10
>
> cheers
> - Alex
>
>
> On Nov 29, 2012, at 1:03 PM, kashif khan <drkashif8310@gmail.com> wrote:
>
> > Hi,
> >
> > I am just struggling to learn the flume and doing some testing. I am
> > running two agents (agent, agent1). The agent used to upload the log data
> > into hdfs and agent1 used as logger. The configuration of two agents as:
> >
> > agent.sources = tail
> > agent.channels = memoryChannel
> > agent.sinks = hdfs-clusterSink
> >
> > agent.sources.tail.type = exec
> > agent.sources.tail.command = tail -f /var/log/flume-ng/flume.log
> > agent.sources.tail.channels = memoryChannel
> >
> > agent.sinks.hdfs-clusterSink.
> > channel = memoryChannel
> > agent.sinks.hdfs-clusterSink.type = hdfs
> > agent.sinks.hdfs-clusterSink.hdfs.path = hdfs://
> > hadoop1.example.com/user/root/Test/
> >
> >
> > agent.channels.memoryChannel.type = memory
> > agent.channels.memoryChannel.transactionCapacity=1000
> > agent.channels.memoryChannel.capacity = 100
> >
> >
> >
> >
> > agent1.sources = source1
> > agent1.sinks = sink1
> > agent1.channels = channel1
> >
> > # Describe/configure source1
> > agent1.sources.source1.type = netcat
> > agent1.sources.source1.bind = localhost
> > agent1.sources.source1.port = 44444
> >
> > # Describe sink1
> > agent1.sinks.sink1.type = logger
> >
> > # Use a channel which buffers events in memory
> > agent1.channels.channel1.type = memory
> > agent1.channels.channel1.capacity = 1000
> > agent1.channels.channel1.transactionCapactiy = 100
> >
> > # Bind the source and sink to the channel
> > agent1.sources.source1.channels = channel1
> > agent1.sinks.sink1.channel = channel1
> >
> >
> > I dont know why it does not upload the log file into hdfs. where I am
> doing
> > mistake . If anyone who have solution please let me know.
> >
> >
> > The log file as:
> >
> >
> > 29 Nov 2012 11:49:13,046 INFO  [main]
> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67)  - Starting
> > lifecycle supervisor 1
> > 29 Nov 2012 11:49:13,050 INFO  [main]
> > (org.apache.flume.node.FlumeNode.start:54)  - Flume node starting - agent
> > 29 Nov 2012 11:49:13,051 INFO  [lifecycleSupervisor-1-0]
> > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
>  -
> > Node manager starting
> > 29 Nov 2012 11:49:13,053 INFO  [lifecycleSupervisor-1-0]
> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67)  - Starting
> > lifecycle supervisor 10
> > 29 Nov 2012 11:49:13,052 INFO  [lifecycleSupervisor-1-2]
> > (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)
>  -
> > Configuration provider starting
> > 29 Nov 2012 11:49:13,054 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
> > - Reloading configuration file:/etc/flume-ng/conf/flume.conf
> > 29 Nov 2012 11:49:13,057 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> > - Added sinks: hdfs-clusterSink Agent: agent
> > 29 Nov 2012 11:49:13,057 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:13,057 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:sink1
> > 29 Nov 2012 11:49:13,057 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:13,057 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:sink1
> > 29 Nov 2012 11:49:13,058 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> > - Added sinks: sink1 Agent: agent1
> > 29 Nov 2012 11:49:13,058 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:13,087 INFO  [conf-file-poller-0]
> > (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122)  -
> > Post-validation flume configuration contains configuration  for agents:
> > [agent, agent1]
> > 29 Nov 2012 11:49:13,088 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
> > - Creating channels
> > 29 Nov 2012 11:49:13,090 ERROR [conf-file-poller-0]
> >
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:204)
> > - Failed to load configuration data. Exception follows.
> > java.lang.IllegalStateException
> >    at
> > com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> >    at
> > org.apache.flume.channel.MemoryChannel.configure(MemoryChannel.java:189)
> >    at
> org.apache.flume.conf.Configurables.configure(Configurables.java:41)
> >    at
> >
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:271)
> >    at
> >
> org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:221)
> >    at
> >
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
> >    at
> >
> org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
> >    at
> >
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
> >    at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> >    at
> >
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> >    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> >    at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> >    at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> >    at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> >    at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >    at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >    at java.lang.Thread.run(Thread.java:662)
> > 29 Nov 2012 11:49:22,157 INFO  [main]
> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67)  - Starting
> > lifecycle supervisor 1
> > 29 Nov 2012 11:49:22,161 INFO  [main]
> > (org.apache.flume.node.FlumeNode.start:54)  - Flume node starting -
> agent1
> > 29 Nov 2012 11:49:22,162 INFO  [lifecycleSupervisor-1-0]
> > (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start:203)
>  -
> > Node manager starting
> > 29 Nov 2012 11:49:22,164 INFO  [lifecycleSupervisor-1-0]
> > (org.apache.flume.lifecycle.LifecycleSupervisor.start:67)  - Starting
> > lifecycle supervisor 10
> > 29 Nov 2012 11:49:22,162 INFO  [lifecycleSupervisor-1-1]
> > (org.apache.flume.conf.file.AbstractFileConfigurationProvider.start:67)
>  -
> > Configuration provider starting
> > 29 Nov 2012 11:49:22,165 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run:195)
> > - Reloading configuration file:/etc/flume-ng/conf/flume.conf
> > 29 Nov 2012 11:49:22,168 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> > - Added sinks: hdfs-clusterSink Agent: agent
> > 29 Nov 2012 11:49:22,168 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:22,168 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:sink1
> > 29 Nov 2012 11:49:22,168 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:22,168 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:sink1
> > 29 Nov 2012 11:49:22,168 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:912)
> > - Added sinks: sink1 Agent: agent1
> > 29 Nov 2012 11:49:22,168 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:998)
> > - Processing:hdfs-clusterSink
> > 29 Nov 2012 11:49:22,195 INFO  [conf-file-poller-0]
> > (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:122)  -
> > Post-validation flume configuration contains configuration  for agents:
> > [agent, agent1]
> > 29 Nov 2012 11:49:22,195 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:249)
> > - Creating channels
> > 29 Nov 2012 11:49:22,221 INFO  [conf-file-poller-0]
> > (org.apache.flume.instrumentation.MonitoredCounterGroup.<init>:68)  -
> > Monitoried counter group for type: CHANNEL, name: channel1, registered
> > successfully.
> > 29 Nov 2012 11:49:22,222 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels:273)
> > - created channel channel1
> > 29 Nov 2012 11:49:22,233 INFO  [conf-file-poller-0]
> > (org.apache.flume.sink.DefaultSinkFactory.create:70)  - Creating instance
> > of sink: sink1, type: logger
> > 29 Nov 2012 11:49:22,237 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:106)
> > - Starting new configuration:{
> > sourceRunners:{source1=EventDrivenSourceRunner: {
> > source:org.apache.flume.source.NetcatSource{name:source1,state:IDLE} }}
> > sinkRunners:{sink1=SinkRunner: {
> > policy:org.apache.flume.sink.DefaultSinkProcessor@4430d82dcounterGroup:{
> > name:null counters:{} } }}
> > channels:{channel1=org.apache.flume.channel.MemoryChannel{name:
> channel1}} }
> > 29 Nov 2012 11:49:22,238 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:113)
> > - Starting Channel channel1
> > 29 Nov 2012 11:49:22,239 INFO  [lifecycleSupervisor-1-1]
> > (org.apache.flume.instrumentation.MonitoredCounterGroup.start:82)  -
> > Component type: CHANNEL, name: channel1 started
> > 29 Nov 2012 11:49:22,239 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:128)
> > - Waiting for channel: channel1 to start. Sleeping for 500 ms
> > 29 Nov 2012 11:49:22,739 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:141)
> > - Starting Sink sink1
> > 29 Nov 2012 11:49:22,740 INFO  [conf-file-poller-0]
> >
> (org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents:152)
> > - Starting Source source1
> > 29 Nov 2012 11:49:22,741 INFO  [lifecycleSupervisor-1-3]
> > (org.apache.flume.source.NetcatSource.start:147)  - Source starting
> > 29 Nov 2012 11:49:22,752 INFO  [lifecycleSupervisor-1-3]
> > (org.apache.flume.source.NetcatSource.start:161)  - Created
> > serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
> > 29 Nov 2012 11:49:33,327 INFO
> > [SinkRunner-PollingRunner-DefaultSinkProcessor]
> > (org.apache.flume.sink.LoggerSink.process:70)  - Event: { headers:{}
> body:
> > 68 65 6C 6C 6F 0D                               hello. }
> > 29 Nov 2012 11:49:42,328 INFO
> > [SinkRunner-PollingRunner-DefaultSinkProcessor]
> > (org.apache.flume.sink.LoggerSink.process:70)  - Event: { headers:{}
> body:
> > 68 6F 77 20 69 73 20 67 6F 69 6E 67 0D          how is going. }
> > 29 Nov 2012 11:49:46,359 INFO
> > [SinkRunner-PollingRunner-DefaultSinkProcessor]
> > (org.apache.flume.sink.LoggerSink.process:70)  - Event: { headers:{}
> body:
> > 65 76 65 72 79 20 74 68 69 6E 67 73 20 61 72 65 every things are }
> >
> >
> > Many thanks
> >
> > Best regards,
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>


-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Mime
View raw message