phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiaofeng Wang <cap...@gmail.com>
Subject Re: A strange issue about missing data
Date Thu, 29 Mar 2018 17:32:50 GMT
Thanks Sergey. I forgot to mention that the data is loaded via MapReduce. If I load data with
HFile, will Phoenix generate the stats automatically? This is what I didn’t understand.
I find this on official page "Statistics are also automatically collected during major compactions
and region splits so manually running this command may not be necessary."

> On 30 Mar 2018, at 12:44 AM, Sergey Soldatov <sergeysoldatov@gmail.com> wrote:
> 
> Usually such kind of problems may happen when something wrong with the statistic. You
may try to clean SYSTEM.STATS, restart the client and check whether it fixes the problem.
If not, you may try to turn on DEBUG log level on the client and check whether generated scans
are covering all regions (this task is easier to perform with clean stats).  
> 
> Thanks,
> Sergey
> 
> 
> On Thu, Mar 29, 2018 at 9:15 AM, Xiaofeng Wang <capt7w@gmail.com <mailto:capt7w@gmail.com>>
wrote:
> Hi, recently I encounter a strange issue about missing data. For instance, column A,
B and C is the primary keys in Phoenix and I can get all the relevant records via the condition
A=‘aaa'. But some time later(maybe one hour or more), I get the less records than before
with the same condition. I don't know why :(
> 
> BTW: The Phoenix table is salted with bucket 13
> 
> This is my environment:
> 
> Phoenix: 4.8.0
> HBase: 1.2.0
> 
> hbase-site.xml:
> 
> <configuration>
>  <property>
>    <name>hbase.regionserver.thrift.http</name>
>    <value>true</value>
>  </property>
>  <property>
>    <name>hbase.thrift.support.proxyuser</name>
>    <value>true</value>
>  </property>
>  <property>
>    <name>hbase.rootdir</name>
>    <value>hdfs://ns1/hbase</value> <>
>  </property>
>  <property>
>    <name>hbase.master.port</name>
>    <value>60000</value>
>  </property>
>  <property>
>    <name>hbase.master.info <http://hbase.master.info/>.port</name>
>    <value>60010</value>
>  </property>
>  <property>
>    <name>hbase.regionserver.port</name>
>    <value>60020</value>
>  </property>
>  <property>
>    <name>hbase.regionserver.info.port</name>
>    <value>60030</value>
>  </property>
>  <property>
>    <name>hbase.cluster.distributed</name>
>    <value>true</value>
>  </property>
>  <property>
>    <name>hbase.zookeeper.quorum</name>
>    <value>host1:2181,host2:2181,host3:2181</value>
>  </property>
>  <property>
>    <name>zookeeper.recovery.retry</name>
>    <value>3</value>
>  </property>
>  <property>
>    <name>zookeeper.session.timeout</name>
>    <value>60000</value>
>  </property>
>  <property>
>    <name>hbase.ipc.server.callqueue.handler.factor</name>
>    <value>0.2</value>
>  </property>
>  <property>
>    <name>hbase.ipc.server.callqueue.read.ratio</name>
>    <value>0.3</value>
>  </property>
>  <property>
>    <name>hbase.ipc.server.callqueue.scan.ratio</name>
>    <value>0.4</value>
>  </property>
>  <property>
>    <name>hbase.client.max.perserver.tasks</name>
>    <value>7</value>
>  </property>
>  <property>
>    <name>hbase.client.max.perregion.tasks</name>
>    <value>4</value>
>  </property>
>  <property>
>    <name>hbase.offpeak.start.hour</name>
>    <value>2</value>
>  </property>
>  <property>
>    <name>hbase.offpeak.end.hour</name>
>    <value>4</value>
>  </property>
>  <property>
>    <name>hbase.storescanner.parallel.seek.enable</name>
>    <value>true</value>
>  </property>
>  <property>
>    <name>hbase.storescanner.parallel.seek.threads</name>
>    <value>10</value>
>  </property>
>  <property>
>    <name>hbase.client.retries.number</name>
>    <value>5</value>
>  </property>
>  <property>
>    <name>hbase.client.scanner.caching</name>
>    <value>1000</value>
>  </property>
>  <property>
>    <name>hbase.regionserver.handler.count</name>
>    <value>100</value>
>  </property>
>  <property>
>    <name>hfile.block.cache.size</name>
>    <value>0.4</value>
>  </property>
>  <property>
>    <name>hbase.coprocessor.abortonerror</name>
>    <value>false</value>
>  </property>
>  <property>
>    <name>hbase.regionserver.thrift.framed</name>
>    <value>false</value>
>  </property>
>  <property>
>    <name>hbase.column.max.version</name>
>    <value>3</value>
>  </property>
>  <property>
>    <name>hbase.status.published</name>
>    <value>false</value>
>  </property>
>  <property>
>    <name>hbase.status.multicast.address.port</name>
>    <value>61000</value>
>  </property>
>  <property>
>    <name>hbase.lease.recovery.dfs.timeout</name>
>    <value>2300000</value>
>  </property>
>  <property>
>    <name>dfs.client.sockettimeout</name>
>    <value>1000000</value>
>  </property>
>  <property>
>    <name>hbase.master.distributed.log.replay</name>
>    <value>true</value>
>  </property>
>  <property>
>    <name>hbase.rpc.engine</name>
>    <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
>  </property>
>  <property>
>    <name>hbase.coprocessor.master.classes</name>
> <value>org.apache.hadoop.hbase.security.access.AccessController</value>
>  </property>
>  <property>
>    <name>hbase.coprocessor.region.classes</name>
> <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
>  </property>
>  <property>
>    <name>hbase.superuser</name>
>    <value>hbase</value>
>  </property>
>  <property>
>    <name>hbase.security.authorization</name>
>    <value>false</value>
>  </property>
>  <property>
>    <name>hbase.client.write.buffer</name>
>    <value>5242880</value>
>  </property>
>  <property>
>    <name>hbase.hregion.max.filesize</name>
>    <value>10737418240</value>
>  </property>
>  <property>
>    <name>hbase.rpc.timeout</name>
>    <value>1000000</value>
>  </property>
>  <property>
>    <name>hbase.regionserver.wal.codec</name>
> <value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value>
> </property>
>  <property>
>    <name>hbase.client.keyvalue.maxsize</name>
>    <value>20971520</value>
>  </property>
>  <property>
>    <name>hbase.hregion.memstore.flush.size</name>
>    <value>536870912</value>
>  </property>
>  <property>
>    <name>hbase.hregion.memstore.block.multiplier</name>
>    <value>8</value>
>  </property>
>  <property>
>    <name>hbase.hstore.blockingStoreFiles</name>
>    <value>300</value>
>  </property>
>  <property>
>    <name>hbase.hstore.compactionThreshold</name>
>    <value>10</value>
>  </property>
>  <property>
>    <name>hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily</name>
>    <value>320</value>
>  </property>
> </configuration>
> 
> Thank you all!
> 


Mime
View raw message