incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <>
Subject Re: Slider Proposal
Date Mon, 14 Apr 2014 09:42:35 GMT
On 12 April 2014 23:38, Roman Shaposhnik <> wrote:

> On Sat, Apr 12, 2014 at 11:58 AM, Andrew Purtell <>
> wrote:
> >> > The reason I ask is I'm wondering how Slider differentiates from
> projects
> >> > like Apache Twill or Apache Bigtop that are already existing vehicles
> for
> >> > achieving the aims discussed in the Slider proposal.
> >>
> >>
> >> Twill: handles all the AM logic for running new code packaged as a JAR
> with
> >> an executor method
> >> Bigtop: stack testing
> >
> >
> > As a Bigtop committer, I disagree with this narrow interpretation of the
> > scope of the project, but this is my personal opinion and I am not PMC...
> A strong +1 here! Bigtop attacks a problem of packaging and deploying
> Hadoop stacks from a classical UNIX packaging background.

I think I'm a big top committer too --but sorry if I stepped on your toes.
I view it as testing primarily because the main bits of code I've been near
were stuff to do FS testing, and the script testing code ( we use as the basis for our remote functional
tests). I'd also like to get

> We are
> also slowly moving into container/VM/OSv packaging territory which
> could be an extremely exciting way of side-stepping the general installer
> issues (something that Ambari struggles mightily with).

Topic for another day. I believe I have some past experience in that area,
hence some opinions.

> Something like Slider tries to leapfrog and side-step UNIX packaging and
> deployment altogether. This is an interesting take on the problem, but
> ultimately the jury is still out on what the level of adoption for
> "everything
> is now YARN" will be.

What it's trying to do is say "end users are allowed to install and run
things too". It's like in linux, you can do rpm/deb installations -and
that's the best way to manage a set of machines, but there's nothing to
stop me, a non-root user, from running anything I want even if the admins
haven't installed it. We have formats for that .tar, .gz, arguably .war and

most of the standard java server apps come with .tar.gz/.zip distributions
(hadoop included), so we all know user-side code is a use case -what we're
thinking of here are : what are the problems we need to address to get this
to work in a YARN cluster.

-getting the configuration customised to the destination. This is the
perennial problem of Configuration Management -and I am not going to try
and solve that problem -again. There is 20+ years of history there, the
scripted vs desired state, centralised vs decentralised and static/dynamic
debates being the perennial ones, ones into which you can device them all
up. Oh, and there's policy driven, such as CompSolv [Hewson12], which is
not yet something that's trickled into the OSS tools yet.

What we're thinking of is something far less ambitious than "lets do a new
CM system" , and in doing so have something more practical "how to make it
easy to bundle non-YARN code so that YARN can deploy it, with enough
metadata and scripts to bring it up. At the same, I do want it to be
possible for CM systems, PaaS front ends and other management tools to be
able to not just spawn a slider app, but integrate better with it. Part of
our thinking here is the idea of "bonded" AM, which, rather than an
in-process  "unmanaged" AM, is a remote YARN-deployed AM but hooked up so
that only the owner process can work with it -events being published for
the owner to track and control it, and having both slider and the owner
being able to tell any agent what to do. This is work in progress -I'd like
a prototype web front end though.

We are going to have to do some of the dynamic config generation, because
we can't dictate up front ports and hosts, so will need to get that
information into both the configs used when starting the apps, and into
information published up for clients to pull down -the latter meaning we
are going to have to evolve a cluster registry, based on Curator or
something else -adding artifact sharing/generation too.

A registry is likely to go beyond slder, as it shouldn't be restricted to
just one YARN app, and indeed, there's no reason why, say, a bigtop
deployed HBase cluster can't publish its binding information.

YARN-913 <> has raised some
of this; I'd like to use slider to evolve some of the topics with a faster
cycle than Hadoop releases -but I'd also love to have any other YARN apps
involved (given I'm trying to re-use any existing code that works: Apache
Curator, Apache Directory, hopefully bits of Helix, I'd welcome other
contribs too).

> At the end of the day, we will need both for a really long time.

+1. YARN itself depends on a well-setup cluster, network and all the bits
where users take an ops team for granted

> > For example, we package Hadoop core and ecosystem services both for
> > deployment, have Puppet based deployment automation (which can be used
> more
> > generally than merely for setting up test clusters), and I have been
> > considering filing JIRAs to tie in cgroups at the whole stack level here.

> > What is missing of course is a hierarchical model for resource
> management,
> > and tools within the components for differentiated service levels, but
> that
> > is another discussion.
> On that note, I find YARN's attitude towards cgroups be, how shall I put
> it,
> optimistic. If you look carefully you can see that the Linux community
> has completely given up an pretending that one can use naked cgroup
> trees for reliable resource partitioning:
> It is now clear to me that the path Linux distros are endorsing is via
> the brokers such as systemd:
> As such I'd see Bigtop providing quite a bit of value out of the box
> via tight integration with systemd. YARN, at the moment, is in a trickier
> situation.

I'd take that up in the YARN list in general -its only just moving from
memory limiting to CPU, then we can worry about what is possible with IO
and network. Which may mean it is still time to impose a cgroup
architecture that all are happy with.

Of course, in a VM world, resource throttling can take place outside the
OS, and on a cluster-wide level. Things get -possibly- simpler. You can
stop worrying about cleanup too.


NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message