flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish <paliwalash...@gmail.com>
Subject Re: support for Google Storage ?
Date Thu, 04 Dec 2014 10:03:16 GMT
Related JIRA that has already been fixed
https://issues.apache.org/jira/browse/FLUME-2337

On Thu, Dec 4, 2014 at 3:18 PM, Jean-Philippe Caruana
<jp@target2sell.com> wrote:
> https://issues.apache.org/jira/browse/FLUME-2569
>
> Le 04/12/2014 09:57, Hari Shreedharan a écrit :
>
> Reasons are mostly historical. Feel free to submit a patch to bump it up
>
> Thanks, Hari
>
>
> On Thu, Dec 4, 2014 at 12:55 AM, Jean-Philippe Caruana <jp@target2sell.com>
> wrote:
>>
>> Yes, flume-env.sh starts the JVM with 20 Mo and GCS opens a 64 Mo buffer.
>>
>> Any idea/reason why flume starts with such a low heap space ?
>>
>> Le 04/12/2014 01:19, Hari Shreedharan a écrit :
>>
>> It looks like you are just running out of heap space. Try increasing the
>> heap space by specifying a higher value in the flume-env.sh file.
>>
>> Thanks,
>> Hari
>>
>>
>> On Mon, Dec 1, 2014 at 9:02 AM, Jean-Philippe Caruana <jp@target2sell.com>
>> wrote:
>>>
>>> I also asked the question on SO :
>>> https://stackoverflow.com/questions/27232966/what-causes-flume-with-gcs-sink-to-throw-a-outofmemoryexepction
>>>
>>> Le 01/12/2014 15:35, Jean-Philippe Caruana a écrit :
>>>
>>> Hi,
>>>
>>> I managed to write to GS from flume [1], but this is not working 100%
>>> yet:
>>> - files are created in the expected directories, but are empty
>>> - flume throws a java.lang.OutOfMemoryError: Java heap space:
>>>
>>> java.lang.OutOfMemoryError: Java heap space
>>>     at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:76)
>>>     at
>>> com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.<init>(GoogleHadoopOutputStream.java:79)
>>>     at
>>> com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.create(GoogleHadoopFileSystemBase.java:820)
>>>     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
>>>     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
>>>     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
>>>     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:773)
>>>     at
>>> org.apache.flume.sink.hdfs.HDFSSequenceFile.open(HDFSSequenceFile.java:96)
>>>
>>> (complete stack trace here: http://pastebin.com/i5iSgCM3)
>>>
>>> Has anyone already experienced this ?
>>> Is it a bug from google's gcs-connector-latest-hadoop2.jar ?
>>> Where should I look to find out what's wrong ?
>>>
>>> My configuration looks like this:
>>> a1.sinks.hdfs_sink.hdfs.path =
>>> gs://bucket_name/%{env}/%{tenant}/%{type}/%Y-%m-%d
>>>
>>> I am running flume from Docker.
>>>
>>> [1]
>>> http://stackoverflow.com/questions/27174033/what-is-the-minimal-setup-needed-to-write-to-hdfs-gs-on-google-cloud-storage-wit
>>>
>>> Thanks.
>>>
>>> --
>>> Jean-Philippe Caruana
>>> http://www.barreverte.fr
>>
>>
>>
>> --
>> Jean-Philippe Caruana - jp@target2sell.com
>> Target2sell, le turbo du e-commerce
>> 43 rue de Turbigo - 75003 Paris
>> +33 (0) 9 51 92 63 20 | +33 (0) 1 44 54 94 55
>>
>> http://www.target2sell.com
>> http://www.barreverte.fr
>
>
>
> --
> Jean-Philippe Caruana - jp@target2sell.com
> Target2sell, le turbo du e-commerce
> 43 rue de Turbigo - 75003 Paris
> +33 (0) 9 51 92 63 20 | +33 (0) 1 44 54 94 55
>
> http://www.target2sell.com
> http://www.barreverte.fr



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Mime
View raw message