flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Goran Simic <gorsi...@gmail.com>
Subject How to transfer csv files, one-to-one
Date Tue, 28 Jul 2015 10:55:45 GMT
The closest I got is that every line of data from source (that are multiple
files) is written to just one file on the sink side. I use "spooldir" on
source side, and "file_roll" on sink side. How can I improve this to copy
files one-to-one, with the same file name?

This is my setup:

COMP1:
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = source
a1.sources.r1.fileHeader = true    #BTW this should add filename to
destination side, but not working

# Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = [ip_address]
a1.sinks.k1.port = 9000

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1


COMP2:
# Name the components on this agent
a2.sources = r2
a2.sinks = k2
a2.channels = c2

# Describe/configure the source
a2.sources.r2.type = avro
a2.sources.r2.bind = 0.0.0.0
a2.sources.r2.port = 9000

# Describe the sink
a2.sinks.k2.type = file_roll
a2.sinks.k2.sink.directory = sink

# Use a channel which buffers events in memory
a2.channels.c2.type = memory
a2.channels.c2.capacity = 1000
a2.channels.c2.transactionCapacity = 100

# Bind the source and sink to the channel
a2.sources.r2.channels = c2
a2.sinks.k2.channel = c2

If i understand correctly there is no out-of-the-box solution for my
example, so should this be done with interceptors, or should I implement
another sink? Or something else ...

Thanks for your help

Mime
View raw message