flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjay Ramanathan <sanjay.ramanat...@lucidworks.com>
Subject RE: "Hit max consecutive under-replication rotations" Error
Date Tue, 15 Jul 2014 22:48:57 GMT
?Hey Jonathan,


Thanks a lot for the advice. changing the minBlockReplicas did the trick, but I'll work on
the replication configurations with more care.

(Currently, I'm just testing it out on test/lab environment, with just one node).


Thanks,

Sanjay Ramanathan


________________________________
From: Jonathan Natkins <natty@streamsets.com>
Sent: Tuesday, July 15, 2014 3:35 PM
To: user@flume.apache.org
Subject: Re: "Hit max consecutive under-replication rotations" Error

Hi Sanjay,

Is this just a single-node test cluster? Playing with replication configs is probably a little
bit dangerous, since it means that your blocks will have no replicas, and if you lose a disk,
you're going to end up with no way to recover the blocks. If this is a cluster you actually
care about, I'd strongly recommend setting your dfs.replication to 3, and looking at HDFS
to determine why your blocks are not getting replicated in the first place.

My guess is that what is happening is that your Flume agent is a mismatch in configs. The
HDFS sink has a parameter called minBlockReplicas, which informs it as to how many block replicas
are necessary to have, and if not specified, it pulls that parameter from the default HDFS
configuration file. My guess is that, somehow, it's getting a different value for the dfs.replication
or for dfs.namenode.replication.min.

You can probably circumvent this error by modifying your Flume config with this:

a1.sinks.k1.hdfs.minBlockReplicas = 1

Thanks,
Natty


On Tue, Jul 15, 2014 at 3:17 PM, Sanjay Ramanathan <sanjay.ramanathan@lucidworks.com<mailto:sanjay.ramanathan@lucidworks.com>>
wrote:

Hi all,


While trying to input data from flume to HDFS sink, I'm getting this error


"

[ERROR - org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:566)] Hit max consecutive
under-replication rotations (30); will not continue rolling files under this path due to under-replication

"



I looked up the error online and it said to make the below modification(dfs.replication).
I did that and the problem still persists.

My hadoop configuration hdfs-site.xml has the property

"
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
"

I also get this message 30 times before the above error message:
"Block Under-replication detected. Rotating file."

My flume conf file has the configuration:
a1.sinks.k1.channel = c1
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://xx.xx.xx.xx:8020/input1/event/%y-%m-%d/%H%M
a1.sinks.k1.hdfs.useLocalTimeStamp = true
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute
a1.sinks.k1.hdfs.writeFormat = Text
a1.sinks.k1.hdfs.fileType = DataStream
#a1.sinks.k1.hdfs.filePrefix = events-
a1.sinks.k1.hdfs.rollCount = 1000
a1.sinks.k1.hdfs.batchSize = 10000
a1.sinks.k1.hdfs.rollSize = 0
a1.sinks.k1.hdfs.rollInterval = 30

?Kindly let me know what is it, that I'm doing wrong.


Sincerely,
Sanjay Ramanathan



Mime
View raw message