phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Retiring empty regions
Date Wed, 20 Apr 2016 23:48:24 GMT
>  the shell and find the empty ones, another to merge a given region into
a neighbor. We've run them without incident, looks like it all works fine.
One thing we did notice is that the AM leaves the old "retired" regions
around in its counts -- the master status page shows a large number of
"Other Regions". This was alarming at first,

Good to know. I had seen this recently and had a mental note to circle
around and confirm it's just a temporary artifact.

On Wed, Apr 20, 2016 at 3:16 PM, Nick Dimiduk <ndimiduk@gmail.com> wrote:

> Circling back here and adding user@phoenix.
>
> I put together one script to dump region info from the shell and find the
> empty ones, another to merge a given region into a neighbor. We've run them
> without incident, looks like it all works fine. One thing we did notice is
> that the AM leaves the old "retired" regions around in its counts -- the
> master status page shows a large number of "Other Regions". This was
> alarming at first, but we verified it's just an artifact in the AM and in
> fact these regions are not on HDFS or in meta. Bouncing master resolved it.
>
> No one has volunteered any alternative schema designs, so as best we know,
> this will happen to anyone who has timestamp in their rowkey (ie, anyone
> using Phoenix's "Row timestamp" feature [0]) and is also using the TTL
> feature. Are folks interested in adding these scripts to our distribution
> and our book?
>
> -n
>
> [0]: https://phoenix.apache.org/rowtimestamp.html
>
> On Mon, Apr 4, 2016 at 8:34 AM, Nick Dimiduk <ndimiduk@gmail.com> wrote:
>
>> > Crazy idea, but you might be able to take stripped down version of
>> region
>> > normalizer code and make a Tool to run? Requesting split or merge is
>> done
>> > through the client API, and the only weighing information you need is
>> > whether region empty or not, that you could find out too?
>>
>> Yeah, that's the direction I'm headed.
>>
>> > A bit off topic, but I think unfortunately region normalizer now ignores
>> > empty regions to avoid undoing pre-split on the table.
>>
>> Unfortunate indeed. Maybe we should be keeping around the initial splits
>> list as a metadata attribute on the table?
>>
>> > With a right row-key design you will never have empty regions due to
>> TTL.
>>
>> I'd love to hear your thoughts on this design, Vlad. Maybe you'd like to
>> write up a post for the blog? Meanwhile, I'm sure of a couple of us on here
>> on the list would appreciate your Cliff's Notes version. I can take this
>> into account for my v2 schema design.
>>
>> > So Nick, merge on 1.1 is not recommended??? Was working very well on
>> > previous versions. Is ProcV2 really impact it that bad??
>>
>> How to answer here carefully... I have no reason to believe merge is not
>> working on 1.1. I've been on the wrong end of enough "regions stuck in
>> transition" support tickets that I'm not keen to put undue stress on my
>> master. ProcV2 insures against many scenarios that cause master trauma,
>> hence my interest in the implementation details and my preference for
>> cluster administration tasks that use it as their source of authority.
>>
>> Thanks for the thoughts folks.
>> -n
>>
>> On Fri, Apr 1, 2016 at 10:52 AM, Jean-Marc Spaggiari <
>> jean-marc@spaggiari.org> wrote:
>>
>>> ;) That was not the question ;)
>>>
>>> So Nick, merge on 1.1 is not recommended??? Was working very well on
>>> previous versions. Is ProcV2 really impact it that bad??
>>>
>>> JMS
>>>
>>> 2016-04-01 13:49 GMT-04:00 Vladimir Rodionov <vladrodionov@gmail.com>:
>>>
>>> > >> This is something
>>> > >> which makes it far less useful for time-series databases with short
>>> TTL
>>> > on
>>> > >> the tables.
>>> >
>>> > With a right row-key design you will never have empty regions due to
>>> TTL.
>>> >
>>> > -Vlad
>>> >
>>> > On Thu, Mar 31, 2016 at 10:31 PM, Mikhail Antonov <
>>> olorinbant@gmail.com>
>>> > wrote:
>>> >
>>> > > Crazy idea, but you might be able to take stripped down version of
>>> region
>>> > > normalizer code and make a Tool to run? Requesting split or merge is
>>> done
>>> > > through the client API, and the only weighing information you need
is
>>> > > whether region empty or not, that you could find out too?
>>> > >
>>> > >
>>> > > "Short of upgrading to 1.2 for the region normalizer,"
>>> > >
>>> > > A bit off topic, but I think unfortunately region normalizer now
>>> ignores
>>> > > empty regions to avoid undoing pre-split on the table. This is
>>> something
>>> > > which makes it far less useful for time-series databases with short
>>> TTL
>>> > on
>>> > > the tables. We'll need to address that.
>>> > >
>>> > > -Mikhail
>>> > >
>>> > > On Thu, Mar 31, 2016 at 9:56 PM, Nick Dimiduk <ndimiduk@gmail.com>
>>> > wrote:
>>> > >
>>> > > > Hi folks,
>>> > > >
>>> > > > I have a table with TTL enabled. It's been receiving data for
a
>>> while
>>> > > > beyond the TTL and I now have a number of empty regions. I'd like
>>> to
>>> > drop
>>> > > > those empty regions to free up heap space on the region servers
and
>>> > > reduce
>>> > > > master load. I'm running a 1.1 derivative.
>>> > > >
>>> > > > The only threads I found on this topic are from circa 0.92
>>> timeframe.
>>> > > >
>>> > > > Short of upgrading to 1.2 for the region normalizer, what's the
>>> > > recommended
>>> > > > method of cleaning up this cruft? Should I be merging empty regions
>>> > into
>>> > > > their neighbor's? Looks like region merge hasn't been migrated
to
>>> > ProcV2
>>> > > > yet so would be wise to reduce online table activity, or at least
>>> aim
>>> > > for a
>>> > > > "quiet period"? Is there a documented process for off-lining and
>>> > > deleting a
>>> > > > region by name? I don't see anything in the book about it.
>>> > > >
>>> > > > I experimented with online merge on pseudodist, looks like it's
>>> working
>>> > > > fine for the most basic case. I'll probably pursue this unless
>>> someone
>>> > > has
>>> > > > some other ideas.
>>> > > >
>>> > > > Thanks,
>>> > > > Nick
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Thanks,
>>> > > Michael Antonov
>>> > >
>>> >
>>>
>>
>>
>


-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Mime
View raw message