pulsar-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe F <j...@apache.org>
Subject Re: [GitHub] merlimat commented on issue #742: Avoid huge backlog on topic reloading: due to large gap between markDelete-offset and read-position of cursor.
Date Thu, 07 Sep 2017 15:07:45 GMT
I don't think we should attempt to store "regular" ack holes.  What I mean
there is that all shared(queue) consumers will have ack holes in the
regular order of business.  So if we store this unconditionally, we will
end up doing this for all queues.

Ideally we want the mark delete to keep pace with the dispatch rate (more
or less). When that does not happen, we end up with the a large set of
duplicate deliveries on topic reload.

So one way to specify the problem would be to set a size N (configurable,
with a sensible default) that would be the operating gap between mark
delete and the read cursor. Anything that falls beyond N is an ack hole.

The other comment I have is that the  "ack hole" is  a concept that is not
easy for end users to understand. Holes could vary in size.  There could be
10K messages in a single ack hole, or 1000 messages in 1000 ack holes (one
in each.  So in one case you could have 10,000 unacked messages and have no
issues, but in the other case you could have 1,001 messages and have
issues.   This would be difficult to explain, without users grasping what
happens under the covers.  I would rather we use the concept and
terminology  of the  "number of unacked messages".

This is an area where we have and continue to face significant user
education and support cases, hence my emphasis on user friendliness.

Joe

On Wed, Sep 6, 2017 at 5:02 PM, <git@git.apache.org> wrote:

> merlimat commented on issue #742: Avoid huge backlog on topic reloading:
> due to large gap between markDelete-offset and read-position of cursor.
> URL: https://github.com/apache/incubator-pulsar/issues/742#
> issuecomment-327644681
>
>
>    >  where large ack-holes can create backlog
>
>    large *number* of ack-holes
>
>    > So, I think having option to restrict distance between markDelete and
> readPosition is something useful for broker.
>
>    It's not related at all with the distance. I think the better metric is
> indeed the number of "holes"
>
> ----------------------------------------------------------------
> This is an automated message from the Apache Git Service.
> To respond to the message, please log on GitHub and use the
> URL above to go to the specific comment.
>
> For queries about this service, please contact Infrastructure at:
> users@infra.apache.org
>
>
> With regards,
> Apache Git Services
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message