phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Templeton <rtemple...@hortonworks.com>
Subject Phoenix upsert failing on stale connection
Date Mon, 28 Nov 2016 19:14:30 GMT
Hey Everyone!

I have a Storm demo topology I’ve been working on for a while. Recently I adapted the final
sink bolt to write data to my Phoenix cdr_fact table and initially everything works as expected
wrt being able to insert data into the table. The problem comes when the topology sits idle
for a period of time (I.e. No data arriving in the graph for a few minutes) records fail to
insert into the table but no exceptions are being thrown. I thought this was due to some kind
of timeout error and recently added code to check the validity of the connection but the validity
check passes yet records are just not being inserted. I looked in the Phoenix logs but don’t
find any entries for failed UPSERTS. I’m stuck on what to try next in order to track this
down.

The code for the bolt can be found here: https://github.com/rtempleton/StreamingCDR/blob/master/src/main/java/com/github/rtempleton/cdr_storm/bolts/PhoenixFactWriter.java

Here’s a snippet from the my app log file

2016-11-11 18:26:23.278 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.292 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.309 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.326 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.339 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.353 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.368 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.384 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.402 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.415 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.538 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.554 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.571 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.586 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.598 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.611 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.626 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.639 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.653 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.675 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.692 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:27:14.776 o.a.s.k.ZkCoordinator [INFO] Task [1/2] Refreshing partition manager
connections

2016-11-11 18:27:14.779 o.a.s.k.ZkCoordinator [INFO] Task [2/2] Refreshing partition manager
connections

2016-11-11 18:27:14.784 o.a.s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper:
GlobalPartitionInformation{topic=cdr-storm, partitionMap={0=rtempleton-3.field.hortonworks.com:6667,
1=rtempleton-1.field.hortonworks.com:6667}}


I wait a little while before pushing more data onto the Kafka queue – during this time when
no data was flowing through the graph, the state of the Phoenix writer somehow changes and
UPSERTS start failing…


2016-11-11 18:53:45.112 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 0

2016-11-11 18:53:45.120 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 0

2016-11-11 18:53:45.222 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 0

2016-11-11 18:53:45.403 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 0

2016-11-11 18:53:45.589 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 0

2016-11-11 18:53:45.689 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 0

2016-11-11 18:53:45.788 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 0


Thanks,
Ryan

Ryan Templeton | Technical Account Mgr | Mobile: 512-694-1965
[cid:497D6A42-9263-466E-BD5B-B0CAFE2C49BC]

Mime
View raw message