flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guyle M. Taber" <gu...@gmtech.net>
Subject Using flume and a Google Cloud Storage (GCS) Sink.
Date Fri, 12 Oct 2018 01:49:00 GMT
I’m having trouble getting a flume sink to GCS working. There’s not a lot of documentation
out there for GCS and Flume.

Does anyone have a GCS sink in Flume working?

I can successfully list the contents of the GCS bucket using “gustils” and the values
in core-site-xml seem valid, as I can list the contents of the bucket using "hdfs dfs -ls
gs://m <gs://m>y-bucket”.  This flume sink when configured for an avro sink to HDFS
works perfectly, but pointing to a GCS bucket, no data is sent and flume shows no errors.

Here’s my GCS sink in flume.conf

agent.sinks.HDFSSink.type = hdfs
agent.sinks.HDFSSink.hdfs.path = gs://my-bucket/
agent.sinks.HDFSSink.channel = memoryChannel
agent.sinks.HDFSSink.hdfs.useLocalTimeStamp = true
agent.sinks.HDFSSink.hdfs.rollCount = 0
agent.sinks.HDFSSink.hdfs.rollInterval = 0
agent.sinks.HDFSSink.hdfs.rollSize = 0
agent.sinks.HDFSSink.hdfs.fileType = DataStream
agent.sinks.HDFSSink.hdfs.writeFormat = Text
agent.sinks.HDFSSink.hdfs.filePrefix = %H-%M-%S
agent.sinks.HDFSSink.hdfs.filePrefix = .json

View raw message