flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeong-shik Jang <jsj...@gruter.com>
Subject Re: flume 0.9.4 windows version out of memory
Date Fri, 25 Jan 2013 09:21:18 GMT
Hi Andy,

Seeing your log, out of memory error had occurred during opening stage 
of the logical node, and so was the last status of the logical node 
reported to master; no more status was reported to master due to the 
error since then, so it stays as OPENNING in master.
Your error seems to be related to ack rolling because agentSink is 
agentE2ESink; agentDFO is not related to ack but still WAL for disk 
failover, though.

I remember you can change java memory option in flume procrun tray 
(flumenodew.exe) icon menu; one looks just like Apache web server tray icon.
You need to restart(stop and start) service through the tray menu once 
you make the change.

-JS

On 1/25/13 5:34 PM, 周梦想 wrote:
> I found that if I change windows agent configure from
> tail("c:\\test.txt") 	agentSink("hadoop48",35853) ;
>
> to
> tail("c:\\test.txt") 	agentDFOSink("hadoop48",35853) ;
>
>
> then the agent changes status from OPENING to ACTIVE.
> my windows agent is :192.168.20.81, while linux collector is : 
> 192.168.10.48
> another linux agent is 192.168.10.46. 10.46 have no problem of agentSink.
>
> Thanks!
> Andy.
>
> 2013/1/25 周梦想 <ablozhou@gmail.com <mailto:ablozhou@gmail.com>>
>
>     when i start the flume node service,the agent is OPENING.
>     Hello,
>
>     I need to gather logs from windows 7, but the flume master and
>     collector is running on linux. I download 0.9.4 exe file from
>     flume official web.
>
>     When I start the node from windows, the status is OPENING.
>
>     zhouhh.TKOffice.local 	zhouhh.TKOffice.local
>     zhouhh.TKOffice.local 	OPENING 	Fri Jan 25 11:45:01 CST 2013 	4
>
>
>     And the log reports OutOfMemoryError.
>
>     ================
>     D:\Program Files\Cloudera\Flume
>     0.9.4\log\flumenode-stdout.2013-01-25.log
>
>     2013-01-25 15:31:03 Commons Daemon procrun stdout
>     initializedListening for transport dt_socket at address: 8888
>     java.lang.OutOfMemoryError: Java heap space
>     Dumping heap to java_pid8100.hprof ...
>     Heap dump file created [13270531 bytes in 0.115 secs]
>
>     ===============
>     D:\Program Files\Cloudera\Flume
>     0.9.4\log\flumenode-stderr.2013-01-25.log
>
>     2013-01-25 15:31:03 Commons Daemon procrun stderr initialized
>     13/01/25 15:31:03 INFO agent.FlumeNode: Flume 0.9.4
>     13/01/25 15:31:03 INFO agent.FlumeNode:  rev
>     8d0aa4be4ee50bb664cb0d1624e5634f46d7b62a
>     13/01/25 15:31:03 INFO agent.FlumeNode: Compiled  on  Mon Jun 20
>     14:46:53 PDT 2011
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.runtime.name <http://java.runtime.name>=Java(TM) SE Runtime
>     Environment
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     sun.boot.library.path=C:\PROGRA~1\Java\jre7\bin
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.vm.version=23.3-b01
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.vm.vendor=Oracle Corporation
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.vendor.url=http://java.oracle.com/
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     path.separator=;
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.vm.name <http://java.vm.name>=Java HotSpot(TM) Client VM
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     file.encoding.pkg=sun.io <http://sun.io>
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     user.country=CN
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property user.script=
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     sun.os.patch.level=Service Pack 1
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.vm.specification.name
>     <http://java.vm.specification.name>=Java Virtual Machine Specification
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     user.dir=C:\Windows\system32
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.runtime.version=1.7.0_07-b11
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property flume=
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.awt.graphicsenv=sun.awt.Win32GraphicsEnvironment
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.endorsed.dirs=C:\PROGRA~1\Java\jre7\lib\endorsed
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property os.arch=x86
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.io.tmpdir=C:\Windows\TEMP\
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     line.separator=
>
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.vm.specification.vendor=Oracle Corporation
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property user.variant=
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property os.name
>     <http://os.name>=Windows 7
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     flume.home=d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     sun.jnu.encoding=GBK
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.library.path=d:\PROGRA~1\Cloudera\FLUME0~1.4\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program
>     Files\Common
>     Files\NetSarang;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program
>     Files\Microsoft SQL Server\100\Tools\Binn\;C:\Program
>     Files\Microsoft SQL Server\100\DTS\Binn\;C:\Program Files\Common
>     Files\Thunder Network\KanKan\Codecs;d:\php;d:\php\ext;D:\Program
>     Files\MySQL\MySQL Server
>     5.5\bin;D:\mysql\bin;D:\wxWidgets-2.9.1\include;D:\wxWidgets-2.9.1\include\msvc;W:\Server\include;W:\Base\include;W:\Client\include;d:\Program
>     Files\Lua\5.1;d:\Program Files\Lua\5.1\clibs;D:\Program
>     Files\TortoiseSVN\bin;d:\Program Files\Git\cmd;d:\Program
>     Files\Git\bin;D:\Program Files\TortoiseGit\bin;D:\docbook\bin;;.
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.specification.name <http://java.specification.name>=Java
>     Platform API Specification
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.class.version=51.0
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     sun.management.compiler=HotSpot Client Compiler
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property os.version=6.1
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property user.home=C:\
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     user.timezone=Asia/Shanghai
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.awt.printerjob=sun.awt.windows.WPrinterJob
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     file.encoding=GBK
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.specification.version=1.7
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.class.path=d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\build\classes;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\activation-1.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\ahocorasick-2.x.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\ant-1.6.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\antlr-2.7.7.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\antlr-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\antlr-runtime-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\asm-3.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\automaton-1.0.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\avro-1.5.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\avro-compiler-1.5.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\avro-ipc-1.5.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-cli-1.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-codec-1.3.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-collections-3.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-el-1.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-httpclient-3.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-lang-2.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-logging-1.0.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-net-1.4.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\core-3.1.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\flume-core-0.9.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\guava-r07.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\hadoop-core-0.20.2-CDH3B4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\hsqldb-1.8.0.7.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\irclib-1.0.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson-core-asl-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson-jaxrs-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson-mapper-asl-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson-xc-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jasper-compiler-5.5.23.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jasper-runtime-5.5.23.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jaxb-api-2.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jaxb-impl-2.1.12.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jersey-core-1.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jersey-json-1.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jersey-server-1.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jets3t-0.6.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jettison-1.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jetty-6.1.26.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jetty-util-6.1.26.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jline-0.9.94.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jms-1.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jsp-api-2.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jsp-api-2.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\libthrift-0.6.0-cdh.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\log4j-1.2.15.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\mail-1.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\netty-3.2.4.Final.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\oro-2.0.8.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\paranamer-2.3.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\servlet-api-2.5-20081211.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\servlet-api-2.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\slf4j-api-1.5.11.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\slf4j-log4j12-1.5.11.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\snappy-java-1.0.1-rc3.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\stax-api-1.0-2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\stax-api-1.0.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\stringtemplate-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\velocity-1.6.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\xmlenc-0.52.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\zookeeper-3.3.1-10.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\conf
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property user.name
>     <http://user.name>=ZHOUHH$
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     flume.root.logger=INFO,console
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.vm.specification.version=1.7
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.home=C:\PROGRA~1\Java\jre7
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     sun.arch.data.model=32
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     user.language=zh
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.specification.vendor=Oracle Corporation
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     awt.toolkit=sun.awt.windows.WToolkit
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.vm.info <http://java.vm.info>=mixed mode
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.version=1.7.0_07
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.ext.dirs=C:\PROGRA~1\Java\jre7\lib\ext;C:\Windows\Sun\Java\lib\ext
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     sun.boot.class.path=C:\PROGRA~1\Java\jre7\lib\alt-rt.jar;C:\PROGRA~1\Java\jre7\lib\resources.jar;C:\PROGRA~1\Java\jre7\lib\rt.jar;C:\PROGRA~1\Java\jre7\lib\sunrsasign.jar;C:\PROGRA~1\Java\jre7\lib\jsse.jar;C:\PROGRA~1\Java\jre7\lib\jce.jar;C:\PROGRA~1\Java\jre7\lib\charsets.jar;C:\PROGRA~1\Java\jre7\lib\jfr.jar;C:\PROGRA~1\Java\jre7\classes
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.vendor=Oracle Corporation
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     file.separator=\
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     java.vendor.url.bug=http://bugreport.sun.com/bugreport/
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     sun.io.unicode.encoding=UnicodeLittle
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     sun.cpu.endian=little
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     sun.desktop=windows
>     13/01/25 15:31:03 INFO agent.FlumeNode: System property
>     sun.cpu.isalist=pentium_pro+mmx pentium_pro pentium+mmx pentium
>     i486 i386 i86
>     13/01/25 15:31:03 INFO agent.FlumeNode: Starting flume agent on:
>     zhouhh.TKOffice.local
>     13/01/25 15:31:03 INFO agent.FlumeNode:  Working directory is:
>     C:\Windows\system32\.
>     13/01/25 15:31:03 INFO conf.FlumeConfiguration: Loading
>     configurations from d:/PROGRA~1/Cloudera/FLUME0~1.4/conf
>     13/01/25 15:31:03 WARN text.FormatFactory: Unable to load output
>     format plugin class  - Class not found
>     13/01/25 15:31:03 INFO mortbay.log: Logging to
>     org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>     org.mortbay.log.Slf4jLog
>     13/01/25 15:31:03 INFO util.InternalHttpServer: Starting internal
>     HTTP server
>     13/01/25 15:31:03 INFO mortbay.log: jetty-6.1.26
>     13/01/25 15:31:03 INFO mortbay.log: Extract
>     d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\webapps\flumeagent.war to
>     C:\Windows\TEMP\Jetty_0_0_0_0_35862_flumeagent.war__flumeagent__4cgrz3\webapp
>     13/01/25 15:31:03 INFO mortbay.log: Started
>     SelectChannelConnector@0.0.0.0:35862
>     <http://SelectChannelConnector@0.0.0.0:35862>
>     13/01/25 15:31:03 INFO util.InternalHttpServer: Server started
>     13/01/25 15:31:03 INFO agent.MultiMasterRPC: No active master RPC
>     connection
>     13/01/25 15:31:04 INFO agent.ThriftMasterRPC: Connected to master
>     at hadoop48:35872
>     13/01/25 15:31:04 INFO agent.LogicalNodeManager: creating new
>     logical node zhouhh.TKOffice.local
>     13/01/25 15:31:04 INFO agent.FlumeNode: Hadoop Security enabled: false
>     13/01/25 15:31:04 INFO util.InternalHttpServer: Starting internal
>     HTTP server
>     13/01/25 15:31:04 INFO mortbay.log: jetty-6.1.26
>     13/01/25 15:31:04 INFO mortbay.log: Extract
>     d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\webapps\flumeagent.war to
>     C:\Windows\TEMP\Jetty_0_0_0_0_35862_flumeagent.war__flumeagent__4cgrz3\webapp
>     13/01/25 15:31:04 INFO mortbay.log: Started
>     SelectChannelConnector@0.0.0.0:35862
>     <http://SelectChannelConnector@0.0.0.0:35862>
>     13/01/25 15:31:04 INFO util.InternalHttpServer: Server started
>     13/01/25 15:31:09 INFO agent.LogicalNode: Node config successfully
>     set to com.cloudera.flume.conf.FlumeConfigData@231214
>     13/01/25 15:31:09 INFO durability.NaiveFileWALManager:
>     NaiveFileWALManager is now open
>     13/01/25 15:31:09 INFO durability.NaiveFileWALManager: Attempting
>     to recover
>     c:\tmp\flume-ZHOUHH$\agent\zhouhh.TKOffice.local\sending /
>     20130124-103034656+0800.2144885518902.00000028
>     13/01/25 15:31:09 INFO debug.MemorySinkSource: Opening mem source
>     sink that has with 0 elements
>     13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: Starting
>     checksum group called zhouhh.TKOffice.local2151099224995
>     13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: initial
>     checksum is 13c6a648fe8
>     13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: Finishing
>     checksum group called 'zhouhh.TKOffice.local2151099224995'
>     13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: Checksum
>     succeeded 13c6a648fe8
>     13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: moved from
>     partial to complete zhouhh.TKOffice.local2151099224995
>     Exception in thread "logicalNode zhouhh.TKOffice.local-25"
>     java.lang.OutOfMemoryError: Java heap space
>     at
>     org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:59)
>     at
>     org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
>     at
>     org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1945)
>     at
>     org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1845)
>     at
>     org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1891)
>     at
>     com.cloudera.flume.handlers.hdfs.SeqfileEventSource.next(SeqfileEventSource.java:62)
>     at com.cloudera.flume.core.EventUtil.dumpAll(EventUtil.java:47)
>     at
>     com.cloudera.flume.agent.durability.NaiveFileWALManager.checkAndStripAckFraming(NaiveFileWALManager.java:378)
>     at
>     com.cloudera.flume.agent.durability.NaiveFileWALManager.recoverLog(NaiveFileWALManager.java:288)
>     at
>     com.cloudera.flume.agent.durability.NaiveFileWALManager.recover(NaiveFileWALManager.java:411)
>     at
>     com.cloudera.flume.agent.durability.NaiveFileWALDeco.open(NaiveFileWALDeco.java:240)
>     at com.cloudera.flume.agent.AgentSink.open(AgentSink.java:150)
>     at
>     com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:88)
>
>
>     I have 4GB memory, and the source file to tail only 1KB, What can
>     I do to remove this error?
>
>     Thanks!
>     Andy
>
>


-- 
Jeong-shik Jang / jsjang@gruter.com
Gruter, Inc., R&D Team Leader
www.gruter.com
Enjoy Connecting


Mime
View raw message