phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rafa <raf...@gmail.com>
Subject Re: Retiring empty regions
Date Fri, 24 Mar 2017 09:19:47 GMT
Hi,

For everyone to know, Nick has published the script for retiring empty
regions in :

https://issues.apache.org/jira/browse/HBASE-15712

Nick, Thank you very much for your help and great work !!!

Best Regards,
Rafa.



On Mon, Mar 6, 2017 at 5:49 PM, rafa <rafa13@gmail.com> wrote:

> Hi Nick,
>
> We are facing the same issue. Increasingly number of empty regions derived
> from TTL and Timestamp in row key.
>
> Did you finally published that scripts? Are they available for public
> usage?
>
> Thank you very much in advance for your work and help,
> Best Regards,
> rafa
>
>
>
>
> On Thu, Apr 21, 2016 at 1:48 AM, Andrew Purtell <apurtell@apache.org>
> wrote:
>
>> >  the shell and find the empty ones, another to merge a given region
>> into a neighbor. We've run them without incident, looks like it all works
>> fine. One thing we did notice is that the AM leaves the old "retired"
>> regions around in its counts -- the master status page shows a large number
>> of "Other Regions". This was alarming at first,
>>
>> Good to know. I had seen this recently and had a mental note to circle
>> around and confirm it's just a temporary artifact.
>>
>> On Wed, Apr 20, 2016 at 3:16 PM, Nick Dimiduk <ndimiduk@gmail.com> wrote:
>>
>>> Circling back here and adding user@phoenix.
>>>
>>> I put together one script to dump region info from the shell and find
>>> the empty ones, another to merge a given region into a neighbor. We've run
>>> them without incident, looks like it all works fine. One thing we did
>>> notice is that the AM leaves the old "retired" regions around in its counts
>>> -- the master status page shows a large number of "Other Regions". This was
>>> alarming at first, but we verified it's just an artifact in the AM and in
>>> fact these regions are not on HDFS or in meta. Bouncing master resolved it.
>>>
>>> No one has volunteered any alternative schema designs, so as best we
>>> know, this will happen to anyone who has timestamp in their rowkey (ie,
>>> anyone using Phoenix's "Row timestamp" feature [0]) and is also using the
>>> TTL feature. Are folks interested in adding these scripts to our
>>> distribution and our book?
>>>
>>> -n
>>>
>>> [0]: https://phoenix.apache.org/rowtimestamp.html
>>>
>>> On Mon, Apr 4, 2016 at 8:34 AM, Nick Dimiduk <ndimiduk@gmail.com> wrote:
>>>
>>>> > Crazy idea, but you might be able to take stripped down version of
>>>> region
>>>> > normalizer code and make a Tool to run? Requesting split or merge is
>>>> done
>>>> > through the client API, and the only weighing information you need is
>>>> > whether region empty or not, that you could find out too?
>>>>
>>>> Yeah, that's the direction I'm headed.
>>>>
>>>> > A bit off topic, but I think unfortunately region normalizer now
>>>> ignores
>>>> > empty regions to avoid undoing pre-split on the table.
>>>>
>>>> Unfortunate indeed. Maybe we should be keeping around the initial
>>>> splits list as a metadata attribute on the table?
>>>>
>>>> > With a right row-key design you will never have empty regions due to
>>>> TTL.
>>>>
>>>> I'd love to hear your thoughts on this design, Vlad. Maybe you'd like
>>>> to write up a post for the blog? Meanwhile, I'm sure of a couple of us on
>>>> here on the list would appreciate your Cliff's Notes version. I can take
>>>> this into account for my v2 schema design.
>>>>
>>>> > So Nick, merge on 1.1 is not recommended??? Was working very well on
>>>> > previous versions. Is ProcV2 really impact it that bad??
>>>>
>>>> How to answer here carefully... I have no reason to believe merge is
>>>> not working on 1.1. I've been on the wrong end of enough "regions stuck in
>>>> transition" support tickets that I'm not keen to put undue stress on my
>>>> master. ProcV2 insures against many scenarios that cause master trauma,
>>>> hence my interest in the implementation details and my preference for
>>>> cluster administration tasks that use it as their source of authority.
>>>>
>>>> Thanks for the thoughts folks.
>>>> -n
>>>>
>>>> On Fri, Apr 1, 2016 at 10:52 AM, Jean-Marc Spaggiari <
>>>> jean-marc@spaggiari.org> wrote:
>>>>
>>>>> ;) That was not the question ;)
>>>>>
>>>>> So Nick, merge on 1.1 is not recommended??? Was working very well on
>>>>> previous versions. Is ProcV2 really impact it that bad??
>>>>>
>>>>> JMS
>>>>>
>>>>> 2016-04-01 13:49 GMT-04:00 Vladimir Rodionov <vladrodionov@gmail.com>:
>>>>>
>>>>> > >> This is something
>>>>> > >> which makes it far less useful for time-series databases
with
>>>>> short TTL
>>>>> > on
>>>>> > >> the tables.
>>>>> >
>>>>> > With a right row-key design you will never have empty regions due
to
>>>>> TTL.
>>>>> >
>>>>> > -Vlad
>>>>> >
>>>>> > On Thu, Mar 31, 2016 at 10:31 PM, Mikhail Antonov <
>>>>> olorinbant@gmail.com>
>>>>> > wrote:
>>>>> >
>>>>> > > Crazy idea, but you might be able to take stripped down version
of
>>>>> region
>>>>> > > normalizer code and make a Tool to run? Requesting split or
merge
>>>>> is done
>>>>> > > through the client API, and the only weighing information you
need
>>>>> is
>>>>> > > whether region empty or not, that you could find out too?
>>>>> > >
>>>>> > >
>>>>> > > "Short of upgrading to 1.2 for the region normalizer,"
>>>>> > >
>>>>> > > A bit off topic, but I think unfortunately region normalizer
now
>>>>> ignores
>>>>> > > empty regions to avoid undoing pre-split on the table. This
is
>>>>> something
>>>>> > > which makes it far less useful for time-series databases with
>>>>> short TTL
>>>>> > on
>>>>> > > the tables. We'll need to address that.
>>>>> > >
>>>>> > > -Mikhail
>>>>> > >
>>>>> > > On Thu, Mar 31, 2016 at 9:56 PM, Nick Dimiduk <ndimiduk@gmail.com>
>>>>> > wrote:
>>>>> > >
>>>>> > > > Hi folks,
>>>>> > > >
>>>>> > > > I have a table with TTL enabled. It's been receiving data
for a
>>>>> while
>>>>> > > > beyond the TTL and I now have a number of empty regions.
I'd
>>>>> like to
>>>>> > drop
>>>>> > > > those empty regions to free up heap space on the region
servers
>>>>> and
>>>>> > > reduce
>>>>> > > > master load. I'm running a 1.1 derivative.
>>>>> > > >
>>>>> > > > The only threads I found on this topic are from circa
0.92
>>>>> timeframe.
>>>>> > > >
>>>>> > > > Short of upgrading to 1.2 for the region normalizer, what's
the
>>>>> > > recommended
>>>>> > > > method of cleaning up this cruft? Should I be merging
empty
>>>>> regions
>>>>> > into
>>>>> > > > their neighbor's? Looks like region merge hasn't been
migrated to
>>>>> > ProcV2
>>>>> > > > yet so would be wise to reduce online table activity,
or at
>>>>> least aim
>>>>> > > for a
>>>>> > > > "quiet period"? Is there a documented process for off-lining
and
>>>>> > > deleting a
>>>>> > > > region by name? I don't see anything in the book about
it.
>>>>> > > >
>>>>> > > > I experimented with online merge on pseudodist, looks
like it's
>>>>> working
>>>>> > > > fine for the most basic case. I'll probably pursue this
unless
>>>>> someone
>>>>> > > has
>>>>> > > > some other ideas.
>>>>> > > >
>>>>> > > > Thanks,
>>>>> > > > Nick
>>>>> > > >
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > --
>>>>> > > Thanks,
>>>>> > > Michael Antonov
>>>>> > >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>>    - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
>
>

Mime
View raw message