phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dominik Wagenknecht <domi...@wagenknecht.cc>
Subject Recommendations for continuous loading
Date Fri, 24 Oct 2014 07:55:05 GMT
Hello Phoenix-Users,

I was wondering what recommendations there are around continuously pushing
data into Phoenix from a Java-program. We have highly varying ingestion
rates, often < 10 / second but sometimes spiking up considerably to ~
10k-15k for short periods of time.

Hence our current approach in our loading-application is to use a buffer
and then by time and amount (so like max 1sec or 1000 UPSERT's, whichever
comes first) send a commit. So in essence we're doing PreparedStatements
and execute() all the time and manual commit control. This proved much
faster than auto-commit (which should in theory be the way to go given the
nature of the implementation) which only performed at a few hundred upserts
per second peak. The current approach is in the thousands but never 5-digit
against 3 datanodes backed by SSD's (!!).

To me it seems-- after digging in the Phoenix-Code -- it is stacking up the
modifications in-memory "again" (in contrast to just sending it off) and
then doing a batch-execution which makes sure the transaction level in [1]
is met.

[1] http://phoenix.apache.org/index.html#transactions

Any recommendations how to improve this? Would JDBC Transaction-Batch be
better?

Regards
Dominik

Mime
View raw message