flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Alten-Lorenz <wget.n...@gmail.com>
Subject Re: flume 0.9.4 windows version out of memory
Date Mon, 28 Jan 2013 07:07:07 GMT
+1

Thats the easiest option on Windows, unfortunately.

- Alex

On Jan 28, 2013, at 7:49 AM, Jeong-shik Jang <jsjang@gruter.com> wrote:

> How about editing registry using regedit?
> 
> 1. locate memory option parameters in registry:
> HKEY_LOCAL_MACHINE > SOFTWARE > Apache Software Foundation > Procrun 2.0 >
FlumeNode > Parameters > Java
> 
> 2. Increase values in JvmMs and JvmMx
> 
> 3. Stop/Start FlumeNode Service via Flume tray icon menu.
> 
> -JS
> 
> On 1/28/13 1:05 PM, 周梦想 wrote:
>> Thank you, Jeong-shik!
>> I also consider that the OPENNING status related to ack rolling. Because it happened
on Windows, not on Linux although I closed the firewall of windows. But I don't know why,
maybe there are some anti-virus software do some thing to stop ack.
>> 
>> The flume node tray icon can't modify the parameter of java VM. It's read only.
>> 
>> Best regards!
>> Andy
>> 
>> 2013/1/25 Jeong-shik Jang <jsjang@gruter.com <mailto:jsjang@gruter.com>>
>> 
>>    Hi Andy,
>> 
>>    Seeing your log, out of memory error had occurred during opening
>>    stage of the logical node, and so was the last status of the
>>    logical node reported to master; no more status was reported to
>>    master due to the error since then, so it stays as OPENNING in master.
>>    Your error seems to be related to ack rolling because agentSink is
>>    agentE2ESink; agentDFO is not related to ack but still WAL for
>>    disk failover, though.
>> 
>>    I remember you can change java memory option in flume procrun tray
>>    (flumenodew.exe) icon menu; one looks just like Apache web server
>>    tray icon.
>>    You need to restart(stop and start) service through the tray menu
>>    once you make the change.
>> 
>>    -JS
>> 
>> 
>>    On 1/25/13 5:34 PM, 周梦想 wrote:
>>>    I found that if I change windows agent configure from
>>>    tail("c:\\test.txt") 	agentSink("hadoop48",35853) ;
>>> 
>>>    to
>>>    tail("c:\\test.txt") 	agentDFOSink("hadoop48",35853) ;
>>> 
>>> 
>>>    then the agent changes status from OPENING to ACTIVE.
>>>    my windows agent is :192.168.20.81, while linux collector is :
>>>    192.168.10.48
>>>    another linux agent is 192.168.10.46. 10.46 have no problem of
>>>    agentSink.
>>> 
>>>    Thanks!
>>>    Andy.
>>> 
>>>    2013/1/25 周梦想 <ablozhou@gmail.com <mailto:ablozhou@gmail.com>>
>>> 
>>>        when i start the flume node service,the agent is OPENING.
>>>        Hello,
>>> 
>>>        I need to gather logs from windows 7, but the flume master
>>>        and collector is running on linux. I download 0.9.4 exe file
>>>        from flume official web.
>>> 
>>>        When I start the node from windows, the status is OPENING.
>>> 
>>>        zhouhh.TKOffice.local 	zhouhh.TKOffice.local
>>>        zhouhh.TKOffice.local 	OPENING 	Fri Jan 25 11:45:01 CST 2013 	4
>>> 
>>> 
>>>        And the log reports OutOfMemoryError.
>>> 
>>>        ================
>>>        D:\Program Files\Cloudera\Flume
>>>        0.9.4\log\flumenode-stdout.2013-01-25.log
>>> 
>>>        2013-01-25 15:31:03 Commons Daemon procrun stdout
>>>        initializedListening for transport dt_socket at address: 8888
>>>        java.lang.OutOfMemoryError: Java heap space
>>>        Dumping heap to java_pid8100.hprof ...
>>>        Heap dump file created [13270531 bytes in 0.115 secs]
>>> 
>>>        ===============
>>>        D:\Program Files\Cloudera\Flume
>>>        0.9.4\log\flumenode-stderr.2013-01-25.log
>>> 
>>>        2013-01-25 15:31:03 Commons Daemon procrun stderr initialized
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: Flume 0.9.4
>>>        13/01/25 15:31:03 INFO agent.FlumeNode:  rev
>>>        8d0aa4be4ee50bb664cb0d1624e5634f46d7b62a
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: Compiled  on  Mon Jun
>>>        20 14:46:53 PDT 2011
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.runtime.name <http://java.runtime.name>=Java(TM) SE
>>>        Runtime Environment
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        sun.boot.library.path=C:\PROGRA~1\Java\jre7\bin
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.vm.version=23.3-b01
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.vm.vendor=Oracle Corporation
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.vendor.url=http://java.oracle.com/
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        path.separator=;
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.vm.name <http://java.vm.name>=Java HotSpot(TM) Client VM
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        file.encoding.pkg=sun.io <http://sun.io>
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        user.country=CN
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        user.script=
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        sun.os.patch.level=Service Pack 1
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.vm.specification.name
>>>        <http://java.vm.specification.name>=Java Virtual Machine
>>>        Specification
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        user.dir=C:\Windows\system32
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.runtime.version=1.7.0_07-b11
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property flume=
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.awt.graphicsenv=sun.awt.Win32GraphicsEnvironment
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.endorsed.dirs=C:\PROGRA~1\Java\jre7\lib\endorsed
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        os.arch=x86
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.io.tmpdir=C:\Windows\TEMP\
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        line.separator=
>>> 
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.vm.specification.vendor=Oracle Corporation
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        user.variant=
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        os.name <http://os.name>=Windows 7
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        flume.home=d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        sun.jnu.encoding=GBK
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.library.path=d:\PROGRA~1\Cloudera\FLUME0~1.4\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program
>>>        Files\Common
>>>        Files\NetSarang;C:\Python27\;C:\Python27\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program
>>>        Files\Microsoft SQL Server\100\Tools\Binn\;C:\Program
>>>        Files\Microsoft SQL Server\100\DTS\Binn\;C:\Program
>>>        Files\Common Files\Thunder
>>>        Network\KanKan\Codecs;d:\php;d:\php\ext;D:\Program
>>>        Files\MySQL\MySQL Server
>>>        5.5\bin;D:\mysql\bin;D:\wxWidgets-2.9.1\include;D:\wxWidgets-2.9.1\include\msvc;W:\Server\include;W:\Base\include;W:\Client\include;d:\Program
>>>        Files\Lua\5.1;d:\Program Files\Lua\5.1\clibs;D:\Program
>>>        Files\TortoiseSVN\bin;d:\Program Files\Git\cmd;d:\Program
>>>        Files\Git\bin;D:\Program Files\TortoiseGit\bin;D:\docbook\bin;;.
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.specification.name <http://java.specification.name>=Java
>>>        Platform API Specification
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.class.version=51.0
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        sun.management.compiler=HotSpot Client Compiler
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        os.version=6.1
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        user.home=C:\
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        user.timezone=Asia/Shanghai
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.awt.printerjob=sun.awt.windows.WPrinterJob
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        file.encoding=GBK
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.specification.version=1.7
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.class.path=d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\build\classes;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\activation-1.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\ahocorasick-2.x.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\ant-1.6.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\antlr-2.7.7.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\antlr-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\antlr-runtime-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\asm-3.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\automaton-1.0.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\avro-1.5.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\avro-compiler-1.5.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\avro-ipc-1.5.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-cli-1.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-codec-1.3.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-collections-3.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-el-1.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-httpclient-3.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\common
>>>        s
>>>        -lang-2.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-logging-1.0.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\commons-net-1.4.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\core-3.1.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\flume-core-0.9.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\guava-r07.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\hadoop-core-0.20.2-CDH3B4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\hsqldb-1.8.0.7.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\irclib-1.0.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson-core-asl-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson-jaxrs-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson-mapper-asl-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jackson-xc-1.5.5.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jasper-compiler-5.5.23.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jasper-runtime-5.5.23.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jaxb-api-2.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jaxb-impl-2.1.12.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jersey-core-1.4.
>>>        j
>>>        ar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jersey-json-1.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jersey-server-1.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jets3t-0.6.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jettison-1.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jetty-6.1.26.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jetty-util-6.1.26.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jline-0.9.94.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jms-1.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jsp-api-2.0.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\jsp-api-2.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\libthrift-0.6.0-cdh.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\log4j-1.2.15.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\mail-1.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\netty-3.2.4.Final.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\oro-2.0.8.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\paranamer-2.3.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\servlet-api-2.5-20081211.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\servlet-api-2.5.jar;d:\PROGRA~1\Cl
>>>        o
>>>        udera\FLUME0~1.4\lib\slf4j-api-1.5.11.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\slf4j-log4j12-1.5.11.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\snappy-java-1.0.1-rc3.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\stax-api-1.0-2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\stax-api-1.0.1.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\stringtemplate-3.2.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\velocity-1.6.4.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\xmlenc-0.52.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\lib\zookeeper-3.3.1-10.jar;d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\conf
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        user.name <http://user.name>=ZHOUHH$
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        flume.root.logger=INFO,console
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.vm.specification.version=1.7
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.home=C:\PROGRA~1\Java\jre7
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        sun.arch.data.model=32
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        user.language=zh
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.specification.vendor=Oracle Corporation
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        awt.toolkit=sun.awt.windows.WToolkit
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.vm.info <http://java.vm.info>=mixed mode
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.version=1.7.0_07
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.ext.dirs=C:\PROGRA~1\Java\jre7\lib\ext;C:\Windows\Sun\Java\lib\ext
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        sun.boot.class.path=C:\PROGRA~1\Java\jre7\lib\alt-rt.jar;C:\PROGRA~1\Java\jre7\lib\resources.jar;C:\PROGRA~1\Java\jre7\lib\rt.jar;C:\PROGRA~1\Java\jre7\lib\sunrsasign.jar;C:\PROGRA~1\Java\jre7\lib\jsse.jar;C:\PROGRA~1\Java\jre7\lib\jce.jar;C:\PROGRA~1\Java\jre7\lib\charsets.jar;C:\PROGRA~1\Java\jre7\lib\jfr.jar;C:\PROGRA~1\Java\jre7\classes
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.vendor=Oracle Corporation
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        file.separator=\
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        java.vendor.url.bug=http://bugreport.sun.com/bugreport/
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        sun.io.unicode.encoding=UnicodeLittle
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        sun.cpu.endian=little
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        sun.desktop=windows
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: System property
>>>        sun.cpu.isalist=pentium_pro+mmx pentium_pro pentium+mmx
>>>        pentium i486 i386 i86
>>>        13/01/25 15:31:03 INFO agent.FlumeNode: Starting flume agent
>>>        on: zhouhh.TKOffice.local
>>>        13/01/25 15:31:03 INFO agent.FlumeNode:  Working directory
>>>        is: C:\Windows\system32\.
>>>        13/01/25 15:31:03 INFO conf.FlumeConfiguration: Loading
>>>        configurations from d:/PROGRA~1/Cloudera/FLUME0~1.4/conf
>>>        13/01/25 15:31:03 WARN text.FormatFactory: Unable to load
>>>        output format plugin class  - Class not found
>>>        13/01/25 15:31:03 INFO mortbay.log: Logging to
>>>        org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>>        org.mortbay.log.Slf4jLog
>>>        13/01/25 15:31:03 INFO util.InternalHttpServer: Starting
>>>        internal HTTP server
>>>        13/01/25 15:31:03 INFO mortbay.log: jetty-6.1.26
>>>        13/01/25 15:31:03 INFO mortbay.log: Extract
>>>        d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\webapps\flumeagent.war
>>>        to
>>>        C:\Windows\TEMP\Jetty_0_0_0_0_35862_flumeagent.war__flumeagent__4cgrz3\webapp
>>>        13/01/25 15:31:03 INFO mortbay.log: Started
>>>        SelectChannelConnector@0.0.0.0:35862
>>>        <http://SelectChannelConnector@0.0.0.0:35862>
>>>        13/01/25 15:31:03 INFO util.InternalHttpServer: Server started
>>>        13/01/25 15:31:03 INFO agent.MultiMasterRPC: No active master
>>>        RPC connection
>>>        13/01/25 15:31:04 INFO agent.ThriftMasterRPC: Connected to
>>>        master at hadoop48:35872
>>>        13/01/25 15:31:04 INFO agent.LogicalNodeManager: creating new
>>>        logical node zhouhh.TKOffice.local
>>>        13/01/25 15:31:04 INFO agent.FlumeNode: Hadoop Security
>>>        enabled: false
>>>        13/01/25 15:31:04 INFO util.InternalHttpServer: Starting
>>>        internal HTTP server
>>>        13/01/25 15:31:04 INFO mortbay.log: jetty-6.1.26
>>>        13/01/25 15:31:04 INFO mortbay.log: Extract
>>>        d:\PROGRA~1\Cloudera\FLUME0~1.4\bin\..\webapps\flumeagent.war
>>>        to
>>>        C:\Windows\TEMP\Jetty_0_0_0_0_35862_flumeagent.war__flumeagent__4cgrz3\webapp
>>>        13/01/25 15:31:04 INFO mortbay.log: Started
>>>        SelectChannelConnector@0.0.0.0:35862
>>>        <http://SelectChannelConnector@0.0.0.0:35862>
>>>        13/01/25 15:31:04 INFO util.InternalHttpServer: Server started
>>>        13/01/25 15:31:09 INFO agent.LogicalNode: Node config
>>>        successfully set to
>>>        com.cloudera.flume.conf.FlumeConfigData@231214
>>>        13/01/25 15:31:09 INFO durability.NaiveFileWALManager:
>>>        NaiveFileWALManager is now open
>>>        13/01/25 15:31:09 INFO durability.NaiveFileWALManager:
>>>        Attempting to recover
>>>        c:\tmp\flume-ZHOUHH$\agent\zhouhh.TKOffice.local\sending /
>>>        20130124-103034656+0800.2144885518902.00000028
>>>        13/01/25 15:31:09 INFO debug.MemorySinkSource: Opening mem
>>>        source sink that has with 0 elements
>>>        13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: Starting
>>>        checksum group called zhouhh.TKOffice.local2151099224995
>>>        13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: initial
>>>        checksum is 13c6a648fe8
>>>        13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: Finishing
>>>        checksum group called 'zhouhh.TKOffice.local2151099224995'
>>>        13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: Checksum
>>>        succeeded 13c6a648fe8
>>>        13/01/25 15:31:09 INFO endtoend.AckChecksumChecker: moved
>>>        from partial to complete zhouhh.TKOffice.local2151099224995
>>>        Exception in thread "logicalNode zhouhh.TKOffice.local-25"
>>>        java.lang.OutOfMemoryError: Java heap space
>>>        at
>>>        org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:59)
>>>        at
>>>        org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
>>>        at
>>>        org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1945)
>>>        at
>>>        org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1845)
>>>        at
>>>        org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1891)
>>>        at
>>>        com.cloudera.flume.handlers.hdfs.SeqfileEventSource.next(SeqfileEventSource.java:62)
>>>        at com.cloudera.flume.core.EventUtil.dumpAll(EventUtil.java:47)
>>>        at
>>>        com.cloudera.flume.agent.durability.NaiveFileWALManager.checkAndStripAckFraming(NaiveFileWALManager.java:378)
>>>        at
>>>        com.cloudera.flume.agent.durability.NaiveFileWALManager.recoverLog(NaiveFileWALManager.java:288)
>>>        at
>>>        com.cloudera.flume.agent.durability.NaiveFileWALManager.recover(NaiveFileWALManager.java:411)
>>>        at
>>>        com.cloudera.flume.agent.durability.NaiveFileWALDeco.open(NaiveFileWALDeco.java:240)
>>>        at com.cloudera.flume.agent.AgentSink.open(AgentSink.java:150)
>>>        at
>>>        com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:88)
>>> 
>>> 
>>>        I have 4GB memory, and the source file to tail only 1KB, What
>>>        can I do to remove this error?
>>> 
>>>        Thanks!
>>>        Andy
>>> 
>>> 
>> 
>> 
>>    --     Jeong-shik Jang /jsjang@gruter.com  <mailto:jsjang@gruter.com>
>>    Gruter, Inc., R&D Team Leader
>>    www.gruter.com  <http://www.gruter.com>
>>    Enjoy Connecting
>> 
>> 
> 
> 
> -- 
> Jeong-shik Jang / jsjang@gruter.com
> Gruter, Inc., R&D Team Leader
> www.gruter.com
> Enjoy Connecting
> 

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF


Mime
View raw message