flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Philippe Caruana ...@target2sell.com>
Subject Re: support for Google Storage ?
Date Thu, 04 Dec 2014 08:54:39 GMT
Yes, flume-env.sh starts the JVM with 20 Mo and GCS opens a 64 Mo buffer.

Any idea/reason why flume starts with such a low heap space ?

Le 04/12/2014 01:19, Hari Shreedharan a écrit :
> It looks like you are just running out of heap space. Try increasing
> the heap space by specifying a higher value in the flume-env.sh file.
>
> Thanks,
> Hari
>
>
> On Mon, Dec 1, 2014 at 9:02 AM, Jean-Philippe Caruana
> <jp@target2sell.com <mailto:jp@target2sell.com>> wrote:
>
>     I also asked the question on SO :
>     https://stackoverflow.com/questions/27232966/what-causes-flume-with-gcs-sink-to-throw-a-outofmemoryexepction
>
>
>     Le 01/12/2014 15:35, Jean-Philippe Caruana a écrit :
>>     Hi,
>>
>>     I managed to write to GS from flume [1], but this is not working
>>     100% yet:
>>     - files are created in the expected directories, but are empty
>>     - flume throws a java.lang.OutOfMemoryError: Java heap space:
>>
>>     java.lang.OutOfMemoryError: Java heap space
>>         at
>>     java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:76)
>>         at
>>     com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.<init>(GoogleHadoopOutputStream.java:79)
>>         at
>>     com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.create(GoogleHadoopFileSystemBase.java:820)
>>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
>>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
>>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
>>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:773)
>>         at
>>     org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:96)
>>
>>     (complete stack trace here: http://pastebin.com/i5iSgCM3)
>>
>>     Has anyone already experienced this ?
>>     Is it a bug from google's gcs-connector-latest-hadoop2.jar ?
>>     Where should I look to find out what's wrong ?
>>
>>     My configuration looks like this:
>>     a1.sinks.hdfs_sink.hdfs.path =
>>     gs://bucket_name/%{env}/%{tenant}/%{type}/%Y-%m-%d
>>
>>     I am running flume from Docker.
>>
>>     [1]
>>     http://stackoverflow.com/questions/27174033/what-is-the-minimal-setup-needed-to-write-to-hdfs-gs-on-google-cloud-storage-wit
>>
>>     Thanks.
>
>     -- 
>     Jean-Philippe Caruana 
>     http://www.barreverte.fr
>
>

-- 
Jean-Philippe Caruana - jp@target2sell.com
Target2sell, le turbo du e-commerce
43 rue de Turbigo - 75003 Paris
+33 (0) 9 51 92 63 20 | +33 (0) 1 44 54 94 55

http://www.target2sell.com
http://www.barreverte.fr


Mime
View raw message