flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sutanu Das <sd2...@att.com>
Subject Need Urgent Help (please) with HTTP Source/JSON Handling
Date Fri, 04 Sep 2015 17:38:38 GMT
Dear Community,

We are trying to send http/json messages and no errors in Flume get all files in HDFS is NULL
(no data seen), we are passing events as JSON Strings, yet, when we see files in HDFS, we
see not data

Is there a HDFS "sink" parameter to show JSON data in hdfs?

We are testing this a simple command as this -- curl -H "Accept: application/json" -H "Content-type:
application/json" -X POST -d ['{"id":100}'] http://localhost:8889

We are passing json as string yet hdfs data is created yet no data seen inside hdfs file.


Please HELP, please!




Here is our config:

ale.sources = source1
ale.channels = channel1
ale.sinks =  sink1

# Define the source
ale.sources.source1.type = http
#ale.sources.source1.handler = org.apache.flume.source.http.JSONHandler
ale.sources.source1.port = 8889
ale.sources.source1.bind = 0.0.0.0

# Define the channel 1
ale.channels.channel1.type = memory
ale.channels.channel1.capacity = 10000000
ale.channels.channel1.transactionCapacity = 10000000

# Define a logging sink
ale.sinks.sink1.type = hdfs
ale.sinks.sink1.channel = channel1
ale.sinks.sink1.hdfs.path = hdfs://ham-dal-d001.corp.wayport.net:8020/prod/hadoop/smallsite/flume_ingest_ale2_hak_dev/station/%Y/%m/%d/%H
#ale.sinks.sink1.hdfs.fileType = DataStream
#ale.sinks.sink1.hdfs.writeFormat = Text
ale.sinks.sink1.hdfs.filePrefix = Ale_2_topology_http_json_raw
ale.sinks.sink1.hdfs.useLocalTimeStamp = true
ale.sinks.sink1.hdfs.round = true
ale.sinks.sink1.hdfs.roundValue = 1
ale.sinks.sink1.hdfs.roundUnit = hour
ale.sinks.sink1.hdfs.callTimeout = 10000000000

Mime
View raw message