phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Taylor <jamestay...@apache.org>
Subject Re: Getting too many open files during table scan
Date Fri, 23 Jun 2017 01:26:22 GMT
My recommendation: don't use salt buckets unless you have a monatomically
increasing row key, for example one that leads with the current date/time.
Otherwise you'll be putting more load (# of salt buckets more load worst
case) for bread-and-butter small-range-scan Phoenix queries.

Thanks,
James

On Fri, Jun 23, 2017 at 10:06 AM Michael Young <yomaiquin@gmail.com> wrote:

> The ulimit open files was only 1024 for the user executing the query.
> After increasing, the queries behaves better.
>
> How can we tell if we need to reduce/increase the number of salt buckets?
>
> Our team set this based on read/write performance using data volume and
> expected queries to be run by users.
>
> However, now it seems the performance has degraded.  We can recreate the
> schemas using fewer/more buckets and reload the data, but I haven't seen a
> hard and fast rule for setting the number of buckets.
>
> We have 12 data nodes, 4 SSDs per node, 128 GB Ram per node, 24 core w/
> hyperthreading (HDP 2.5 running, hbase is primary service).
> and 800+ regions per RS (seems high)
>
> Any orientation on this would be greatly appreciated.
>
>
> On Tue, Jun 20, 2017 at 11:54 AM, Josh Elser <josh.elser@gmail.com> wrote:
>
>> I think this is more of an issue of your 78 salt buckets than the width
>> of your table. Each chunk, running in parallel, is spilling incremental
>> counts to disk.
>>
>> I'd check your ulimit settings on the node which you run this query from
>> and try to increase the number of open files allowed before going into this
>> one in more depth :)
>>
>>
>> On 6/16/17 2:31 PM, Michael Young wrote:
>>
>>>
>>> We are running a 13-node hbase cluster.  One table uses 78 SALT BUCKETS
>>> which seems to work reasonable well for both read and write.  This table
>>> has 130 columns with a PK having 30 columns (fairly wide table).
>>>
>>> However, after adding several new tables we are seeing errors about too
>>> many open files when running a full table scan.
>>>
>>>
>>> Caused by: org.apache.phoenix.exception.PhoenixIOException: Too many
>>> open files
>>>          at
>>> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)
>>>          at
>>> org.apache.phoenix.iterate.SpoolingResultIterator.<init>(SpoolingResultIterator.java:152)
>>>          at
>>> org.apache.phoenix.iterate.SpoolingResultIterator.<init>(SpoolingResultIterator.java:84)
>>>          at
>>> org.apache.phoenix.iterate.SpoolingResultIterator.<init>(SpoolingResultIterator.java:63)
>>>          at
>>> org.apache.phoenix.iterate.SpoolingResultIterator$SpoolingResultIteratorFactory.newIterator(SpoolingResultIterator.java:79)
>>>          at
>>> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:112)
>>>          at
>>> org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:103)
>>>          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>          at
>>> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
>>>          at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>          at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>          at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.io.IOException: Too many open files
>>>          at java.io.UnixFileSystem.createFileExclusively(Native Method)
>>>          at java.io.File.createTempFile(File.java:2024)
>>>          at org.apache.phoenix.shaded.org
>>> .apache.commons.io.output.DeferredFileOutputStream.thresholdReached(DeferredFileOutputStream.java:176)
>>>          at
>>> org.apache.phoenix.iterate.SpoolingResultIterator$1.thresholdReached(SpoolingResultIterator.java:116)
>>>          at org.apache.phoenix.shaded.org
>>> .apache.commons.io.output.ThresholdingOutputStream.checkThreshold(ThresholdingOutputStream.java:224)
>>>          at org.apache.phoenix.shaded.org
>>> .apache.commons.io.output.ThresholdingOutputStream.write(ThresholdingOutputStream.java:92)
>>>          at java.io.DataOutputStream.writeByte(DataOutputStream.java:153)
>>>          at org.apache.hadoop.io
>>> .WritableUtils.writeVLong(WritableUtils.java:273)
>>>          at org.apache.hadoop.io
>>> .WritableUtils.writeVInt(WritableUtils.java:253)
>>>          at org.apache.phoenix.util.TupleUtil.write(TupleUtil.java:149)
>>>          at
>>> org.apache.phoenix.iterate.SpoolingResultIterator.<init>(SpoolingResultIterator.java:127)
>>>          ... 10 more
>>>
>>>
>>> When running an explain plan:
>>> explain select count(1) from MYBIGTABLE
>>>
>>>
>>> +------------------------------------------------------------------------------------------------------------------+
>>> |                                                       PLAN
>>>                                            |
>>>
>>> +------------------------------------------------------------------------------------------------------------------+
>>> | CLIENT 8728-CHUNK 674830174 ROWS 2721056772632 BYTES PARALLEL 78-WAY
>>> FULL SCAN OVER ATT.PRE_ENG_CONVERSION_OLAP  |
>>> |     ROW TIMESTAMP FILTER [0, 9223372036854775807)
>>>                                           |
>>> |     SERVER FILTER BY FIRST KEY ONLY
>>>                                           |
>>> |     SERVER AGGREGATE INTO SINGLE ROW
>>>                                            |
>>>
>>> +------------------------------------------------------------------------------------------------------------------+
>>>
>>> I has a lot of chunks.  Normally this query would return at least some
>>> result after running for a few minutes.  With appropriate filters in the
>>> WHERE clause, the queries run fine.
>>>
>>> Any suggestions on how to avoid this error and get better performance
>>> from the table scans?  Realizing that we don't need to run full table scans
>>> regularly, just trying to understand better best practices for Phoenix
>>> Hbase.
>>>
>>> Thank you,
>>> Michael
>>>
>>
>

Mime
View raw message