incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kent Yao <...@apache.org>
Subject Re: [DISCUSS] Incubating Proposal of Kyuubi
Date Sun, 06 Jun 2021 14:44:17 GMT
Hi Ming,

Thanks for your question.

As mentioned in the proposal, Kyuubi was first developed and applied in
NetEase in 2018. The early scenario we wanted to address was
straightforward - to provide data analysts with a Spark-based analytics
platform. This platform needed to support multi-tenancy [1] and
authentication and authorization [2]. From the project's perspective, it
was just an alternative implementation of Apache Spark Thrift Server. And
for the workloads and users that it serves are not very big in NetEase. So
at that time, I was the only one who was maintaining the project.

Another reason for this problem should be the limited usage scenario caused
by its architecture. It was implemented based on
*spark.driver.allowMultipleContexts*, which was not good judging from the
dismal feedback after we open-sourced it. Many people have found Kyuubi or
me through GitHub, hoping that Kyuubi could solve a problem they were
having in their scenario. From all of these things, I remember most and
feel sorry for is when they said they were using Kyuubi to help medical
staff fight COVID-19 [9]. Limited by design, there was very little that
could be done at the time.

Starting with Spark 3.0.0, the Spark community also removed this feature
[3] for Kyuubi's old architecture. This allowed us to rethink and redesign
Kyuubi architecture. A prototype was quickly designed, but we did not start
it in a hurry. As Kyuubi relies heavily on Spark, we decided to actively
and deeply participate in the Spark community beginning around September
2019 [4][5][6], both to help the Spark community and to verify that the
design of the new Kyuubi architecture [7] was future-proofing enough. In
the process of implementing the new architecture, Zen Jin became a
contributor to the project and contributed several great ideas, which made
Kyuubi's architecture and the scenarios it can cover more general and
comprehensive. This has made Kyuubi's architecture and the scenarios it can
cover more general. This advancement has made Kyuubi more attractive than
older architecture projects.

As a result, many like-minded people [6] started to gather around the
project and the new architecture, and some of them have already become our
contributors. There are also many people who have encouraged us to
contribute to this project to Apache, which will be very beneficial for
them to be more directly involved in the project.

In addition, we've learned a lot about community operations while
contributing to other Apache projects like Spark, Submarine, and Hudi. Some
of these communities are new to Apache, and some are ten years out of
school. We also actively participated in some online and offline
conferences or meetups, published technical articles in different media to
promote Kyuubi.

>From our practical experience, we have learned that promoting a project is
not an easy task. We want to donate Kyuubi to the community and bring in
potentially willing people to help the Kyuubi community grow.

Bests
Kent Yao

[1] https://issues.apache.org/jira/browse/SPARK-2243
[2] https://issues.apache.org/jira/browse/SPARK-5159
[3] https://issues.apache.org/jira/browse/SPARK-26362
[4] https://github.com/apache/spark/commits?author=yaooqinn
[5] https://github.com/apache/spark/commits?author=turboFei
[6] https://github.com/apache/spark/commits?author=ulysses-you
[7] https://github.com/NetEase/kyuubi/releases/tag/v1.0.0
[8]
https://github.com/NetEase/kyuubi/graphs/contributors?from=2021-01-01&to=2021-06-06&type=c
[9] https://github.com/NetEase/kyuubi/issues/228

Ming Wen <wenming@apache.org> 于2021年6月6日周日 上午7:37写道:

> You can see the trend chart from
>
> https://www.apiseven.com/zh/contributor-graph?chart=contributorMonthlyActivity&repo=NetEase/kyuubi
>
>
> Ming Wen <wenming@apache.org>于2021年6月6日 周日上午7:28写道:
>
> > hi, Kent,
> >
> > From 2018 to 2020, kuyybi is not an active project, most of the time
> there
> > is only one contributor per month.  But in 2021, there will be 8 active
> > contributors every month.  What is the reason for this?
> >
> > Willem Jiang <willem.jiang@gmail.com>于2021年6月3日 周四下午12:53写道:
> >
> >> I think we can revisit the PPMC part and have more discussion with the
> >> initial committers.
> >> It's a good chance to let them know better about  how to run an open
> >> source project in Apache Way.
> >>
> >>
> >> Willem Jiang
> >>
> >> Twitter: willemjiang
> >> Weibo: 姜宁willem
> >>
> >> On Thu, Jun 3, 2021 at 11:52 AM Atri Sharma <atri@apache.org> wrote:
> >> >
> >> > I would like to hear from Kyuubi's mentors on the topic before
> reaching
> >> a
> >> > conclusion.
> >> >
> >> > On Thu, 3 Jun 2021, 05:40 Sheng Wu, <wu.sheng.841108@gmail.com>
> wrote:
> >> >
> >> > > Kent
> >> > >
> >> > > Thanks.
> >> > > The proposal is generally better this time.
> >> > >
> >> > > If your initial committers and your mentors agree, you are ready to
> >> embrace
> >> > > the ASF culture, I am good with that.
> >> > > Just from my experience, your community looks a little too young,
> like
> >> > > Atri's concern, which is reasonable.
> >> > > But I would like to trust you and your mentors' judgment and
> >> determination.
> >> > >
> >> > > Sheng Wu 吴晟
> >> > > Twitter, wusheng1108
> >> > >
> >> > >
> >> > > Kent Yao <yao@apache.org> 于2021年6月3日周四 上午12:54写道:
> >> > >
> >> > > > Hi Sheng,
> >> > > >
> >> > > > Truly sorry for the format issue.
> >> > > >
> >> > > > Please trust me I did not mean to hurt anybody and push any
> >> contributor
> >> > > > away. We drafted this proposal together based on the New Podling
> >> > > Proposal.
> >> > > > We never tried to use PPMC and committer to measure whose
> >> contribution is
> >> > > > higher. This mistake has been fixed in the proposal.
> >> > > >
> >> > > > About releases, we will follow
> >> > > > https://www.apache.org/legal/release-policy.html and the guide
of
> >> our
> >> > > > mentors. We will also take the initiative to learn other
> documents.
> >> > > >
> >> > > > > This is not a block, the point I mentioned this is, you
will
> need
> >> more
> >> > > > people.
> >> > > >
> >> > > > We will keep working on this.
> >> > > >
> >> > > > Bests,
> >> > > >
> >> > > > Kent Yao
> >> > > >
> >> > > >
> >> > > > Sheng Wu <wu.sheng.841108@gmail.com> 于2021年6月2日周三
下午10:53写道:
> >> > > >
> >> > > > > Hi Kent
> >> > > > >
> >> > > > > Thanks for replying, but to be honest, your mail format
is
> chaos.
> >> > > > > Take a look at bellowing, which is what I saw in the gmail.
> >> > > > >
> >> > > > > This may be an issue of your mail client or some else.
> >> > > > >
> >> > > > > > The current #.2 - 5 top
> >> > > > > > contributors will stay as committers, and they will
join the
> >> PPMC
> >> > > > during
> >> > > > > > incubation.
> >> > > > >
> >> > > > > PPMC usually is the default for all initial committers,
> >> considering
> >> > > your
> >> > > > > community is still small and not very active.
> >> > > > > You should invite all people you trusted to join, rather
than
> >> pushing
> >> > > > them
> >> > > > > away.
> >> > > > > This is not just a too-high bar issue. Excluding 4 out of
5 is
> >> strange.
> >> > > > > I can't guess what is the root cause, why the other 4 PPMC
> members
> >> > > don't
> >> > > > > touch codes but are trusted completely. Meanwhile, people
with
> >> easily
> >> > > > > measurable code contributions are just as committers?
> >> > > > > We(ASF) always remind the community to recognize no-code
> >> contribution,
> >> > > > but
> >> > > > > in this case, it seems the opposite? Don't recognize code
> >> contributors
> >> > > > > enough?
> >> > > > > Please reach your mentors, this seems not a typical ASF
style.
> If
> >> you
> >> > > > have
> >> > > > > other people already involved in the ASF, asking them too.
> >> > > > >
> >> > > > > And from your description of the release process, I think
you
> >> should
> >> > > talk
> >> > > > > more with your mentors.
> >> > > > > Such as what is a qualified ASF release, what should be
checked.
> >> > > > > This is not a block, the point I mentioned this is, you
will
> need
> >> more
> >> > > > > people.
> >> > > > >
> >> > > > >
> >> > > > > Sheng Wu 吴晟
> >> > > > > Twitter, wusheng1108
> >> > > > >
> >> > > > >
> >> > > > > Kent Yao <yao@apache.org> 于2021年6月2日周三
下午10:37写道:
> >> > > > >
> >> > > > > > Hi Sheng,
> >> > > > > >
> >> > > > > > Thanks for your detailed check and valuable comments.
They are
> >> > > > > > accommodating. I have addressed some of them directly
in the
> >> > > proposal.
> >> > > > > > Please also check the table below for some Q&As.
> >> > > > > > QA
> >> > > > > > You listed only 5 core maintainers, but clearly only
one has
> >> > > dominated
> >> > > > > > contributions. The distortion and skewness of the
> contributions
> >> are
> >> > > > > mainly
> >> > > > > > caused by the way how we merge PRs changed(mentioned
in
> another
> >> email
> >> > > > on
> >> > > > > > the list).
> >> > > > > > I agree with Atri, orphaned is measurable. It is better
to
> >> adjust
> >> > > this
> >> > > > > > statement a little. Address the comment to notify the
risk in
> >> the
> >> > > > Kyuubi
> >> > > > > > proposal.
> >> > > > > > Also, with this fact, the core maintainers should notice
> >> whether this
> >> > > > > small
> >> > > > > > group of people to make sure your features iteration,
bug fix,
> >> and
> >> > > > > release
> >> > > > > > process working by following ASF's requirements.This
would not
> >> be an
> >> > > > easy
> >> > > > > > job. Currently, we learn the release process from some
Apache
> >> > > projects,
> >> > > > > > using major.minor.patch for versioning. As we can see,
since
> >> Kyuubi
> >> > > > 1.0,
> >> > > > > it
> >> > > > > > takes us 2 ~ 3 weeks for a patch release that contains
5-10
> bug
> >> fixes
> >> > > > > until
> >> > > > > > EOL. And it takes us about two months to finish a minor
> release
> >> with
> >> > > 3
> >> > > > ~
> >> > > > > 5
> >> > > > > > big improvements and various small features/bug fixes.
We also
> >> will
> >> > > > start
> >> > > > > > with some release candidates for preview and test before
the
> >> final.
> >> > > > This
> >> > > > > is
> >> > > > > > how we develop and release Kyuubi for now, but we do
have a
> >> long way
> >> > > to
> >> > > > > go
> >> > > > > > and learn to match the ASF requirement.
> >> > > > > > And your committer list is very strange... 1. Why some
of them
> >> are
> >> > > PPMC
> >> > > > > and
> >> > > > > > others are committers? Where this hierarchy comes from?
The
> >> current
> >> > > > PPMC
> >> > > > > > candidates help us a lot build the Kyuubi project and
> >> community, not
> >> > > > > > directly to the codebase but as mentors(before Apache).
Most
> of
> >> them
> >> > > > are
> >> > > > > > from Apache mature TLPs, which have close relationships
with
> >> Kyuubi.
> >> > > > They
> >> > > > > > have sufficient Apache Way experience to guide us to
become a
> >> more
> >> > > > > > Apache-like project before submitting this proposal
in many
> >> ways. We
> >> > > > will
> >> > > > > > continue to promote more contributors to the PPMC/committer
> >> based on
> >> > > > > their
> >> > > > > > code or non-code contributions during incubation.
> >> > > > > > 2. Your core maintainers have 5, which only 4 PPMC?
The
> current
> >> #.2
> >> > > - 5
> >> > > > > top
> >> > > > > > contributors will stay as committers, and they will
join the
> >> PPMC
> >> > > > during
> >> > > > > > incubation.
> >> > > > > > 3. I think only the top 4 have over 400 LOC, why you
declared
> >> you
> >> > > have
> >> > > > 5
> >> > > > > > core developers? I didn't notice that 400 LOC is a
bar for
> core
> >> > > > > developer,
> >> > > > > > will change 5 to 4 core developers accordingly
> >> > > > > > I think you don't have enough no-salaried developers,
am I
> >> missing
> >> > > > > > anything? Yes.
> >> > > > > > Changed in the proposal, most of the contributors are
> salaried,
> >> and
> >> > > > some
> >> > > > > of
> >> > > > > > them are willing to help in the spare time. Thanks
> >> > > > > > This project belongs to NetEase Inc. clearly, why NO
SGA?
> Added
> >> SGA
> >> > > in
> >> > > > > > Proposal. All work here is approved by NetEase and
can be
> >> forward.
> >> > > > > > At last, you are asking about
> >> > > > github.com/apache/incubator-kyuubi-shaded
> >> > > > > > repo,
> >> > > > > > what does this represent? I can only see you have only
one
> >> source
> >> > > repo
> >> > > > > and
> >> > > > > > a doc repo. Removed this requirement in the proposal,
and we
> >> will
> >> > > apply
> >> > > > > it
> >> > > > > > from IPMC if we do need this in the future.
> >> > > > > > ------------------------------
> >> > > > > >
> >> > > > > > Bests
> >> > > > > >
> >> > > > > > Kent Yao
> >> > > > > >
> >> > > > > > Sheng Wu <wu.sheng.841108@gmail.com> 于2021年6月2日周三
下午5:45写道:
> >> > > > > >
> >> > > > > > > Hi
> >> > > > > > >
> >> > > > > > > I read the contributor list and the proposal
> >> > > > > > > > The risk of the Kyuubi project being abandoned
is minimal.
> >> Many
> >> > > > > > > organizations are using Kyuubi to build critical
big data
> >> pipelines
> >> > > > and
> >> > > > > > > willing to help develop Kyuubi's community if
it becomes an
> >> ASF
> >> > > > > project.
> >> > > > > > >
> >> > > > > > > I think the team should evaluate the status again.
You
> listed
> >> only
> >> > > 5
> >> > > > > core
> >> > > > > > > maintainers, but clearly only one has dominated
> >> contributions. I
> >> > > > agree
> >> > > > > > with
> >> > > > > > > Atri, orphaned is measurable.
> >> > > > > > > It is better to adjust this statement a little.
Also, with
> >> this
> >> > > fact,
> >> > > > > the
> >> > > > > > > core maintainers should notice whether this small
group of
> >> people
> >> > > to
> >> > > > > make
> >> > > > > > > sure your features iteration, bug fix, and release
process
> >> working
> >> > > by
> >> > > > > > > following ASF's requirements.
> >> > > > > > > This would not be an easy job.
> >> > > > > > >
> >> > > > > > > And your committer list is very strange
> >> > > > > > > ```
> >> > > > > > > Kent Yao (yao@apache.org) (PPMC)
> >> > > > > > > Wenchen Fan (wenchen@apache.org) (PPMC)
> >> > > > > > > Wanqiang Ji (jiwq@apache.org) (PPMC)
> >> > > > > > > Xun Liu (liuxun@apache.org) (PPMC)
> >> > > > > > > Fei Wang (cn.feiwang@gmail.com)
> >> > > > > > > Ulysses You (ulyssesyou18@gmail.com)
> >> > > > > > > Cheng Pan (pan3793@gmail.com)
> >> > > > > > > Hongxiang Jiang (jhx1008@gmail.com)
> >> > > > > > > Zen Jin(xinjingziranchan@gmail.com)
> >> > > > > > > ```
> >> > > > > > > 1. Why some of them are PPMC and others are committers?
> Where
> >> this
> >> > > > > > > hierarchy comes from?
> >> > > > > > > 2. Your core maintainers have 5, which only 4
PPMC? (Even
> >> hierarchy
> >> > > > > makes
> >> > > > > > > sense, most likely not)
> >> > > > > > > 3. I think only the top 4 have over 400 LOC, why
you
> declared
> >> you
> >> > > > have
> >> > > > > 5
> >> > > > > > > core developers?
> >> > > > > > >
> >> > > > > > > Another re-evaluation request is about `Reliance
on Salaried
> >> > > > > Developers:`
> >> > > > > > > > They are all passionate about the project,
and we are
> >> confident
> >> > > > that
> >> > > > > > the
> >> > > > > > > project will continue even if no salaried developers
> >> contribute to
> >> > > > the
> >> > > > > > > project.
> >> > > > > > >
> >> > > > > > > I think you don't have enough no-salaried developers,
am I
> >> missing
> >> > > > > > > anything?
> >> > > > > > >
> >> > > > > > > About IP clearance, `Initial Source and Intellectual
> Property
> >> > > > > Submission
> >> > > > > > > Plan`
> >> > > > > > > > As soon as Kyuubi is approved to join Apache
Incubator,
> our
> >> > > initial
> >> > > > > > > committers will submit iCLA(s) and CCLA(s). The
codebase is
> >> already
> >> > > > > > > licensed under the Apache License 2.0.
> >> > > > > > >
> >> > > > > > > This project belongs to NetEase Inc. clearly,
why NO SGA?
> >> > > > > > >
> >> > > > > > > At last, you are asking about `
> >> > > > > > github.com/apache/incubator-kyuubi-shaded`
> <http://github.com/apache/incubator-kyuubi-shaded>
> >> <http://github.com/apache/incubator-kyuubi-shaded>
> >> > > <http://github.com/apache/incubator-kyuubi-shaded>
> >> > > > <http://github.com/apache/incubator-kyuubi-shaded>
> >> > > > > <http://github.com/apache/incubator-kyuubi-shaded>
> >> > > > > > <http://github.com/apache/incubator-kyuubi-shaded>
> >> > > > > > > <http://github.com/apache/incubator-kyuubi-shaded>
> >> > > > > > > repo, what does this represent? I can only see
you have only
> >> one
> >> > > > source
> >> > > > > > > repo and a doc repo.
> >> > > > > > >
> >> > > > > > > Sheng Wu 吴晟
> >> > > > > > > Twitter, wusheng1108
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Atri Sharma <atri@apache.org> 于2021年6月2日周三
下午4:49写道:
> >> > > > > > >
> >> > > > > > > > Thanks.
> >> > > > > > > >
> >> > > > > > > > I looked at the link you advised on and if
I am reading it
> >> > > > correctly,
> >> > > > > > top
> >> > > > > > > > committer has 630ish commits and the next
committer has 73
> >> > > commits?
> >> > > > > > > >
> >> > > > > > > > IMO this is an indication of the community
not being
> active
> >> > > enough
> >> > > > > and
> >> > > > > > > > hence the risk of project getting orphaned
is measurable.
> >> > > > > > > >
> >> > > > > > > > Atri
> >> > > > > > > >
> >> > > > > > > > On Wed, 2 Jun 2021, 06:57 Willem Jiang, <
> >> willem.jiang@gmail.com>
> >> > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hi Atri Sharma,
> >> > > > > > > > >
> >> > > > > > > > > It's easier to check out the recent
contribution of
> >> kyuubi from
> >> > > > > this
> >> > > > > > > > > link[1].
> >> > > > > > > > > We already identified the one person
contribution issue
> of
> >> > > kyuubi
> >> > > > > > when
> >> > > > > > > > > Kent came to us and we kept reminding
him to put more
> >> effort
> >> > > into
> >> > > > > > > > > building the community.
> >> > > > > > > > > Last week I had a long F2F conversation
with Kent, I
> >> think he
> >> > > > > already
> >> > > > > > > > > knows about it and there have been some
improvements in
> >> recent
> >> > > > > months
> >> > > > > > > > > but there is still a long way to go
if Kyuubi is
> accepted
> >> as a
> >> > > > > > podling
> >> > > > > > > > > in the Apache Incubator.
> >> > > > > > > > >
> >> > > > > > > > > [1]
> https://github.com/NetEase/kyuubi/graphs/contributors
> >> > > > > > > > >
> >> > > > > > > > > Willem Jiang
> >> > > > > > > > >
> >> > > > > > > > > Twitter: willemjiang
> >> > > > > > > > > Weibo: 姜宁willem
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Jun 1, 2021 at 10:52 PM Atri
Sharma <
> >> atri@apache.org>
> >> > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > Hi,
> >> > > > > > > > > >
> >> > > > > > > > > > I took a look at commit log and
seems like all the
> >> commits
> >> > > for
> >> > > > > > last 3
> >> > > > > > > > > > months have been done by a single
person.
> >> > > > > > > > > >
> >> > > > > > > > > > How active are the remaining committers?
> >> > > > > > > > > >
> >> > > > > > > > > > On Mon, 31 May 2021, 12:59 Kent
Yao, <yao@apache.org>
> >> wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Hi Alexander Alten,
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks for your interest and
+1 for Kyuubi.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Not being very familiar with
Dremel, I just took a
> >> quick
> >> > > look
> >> > > > > at
> >> > > > > > > the
> >> > > > > > > > > > > profile of a very similar
and exciting project,
> Apache
> >> > > Drill.
> >> > > > > If
> >> > > > > > I
> >> > > > > > > > > > > understand correctly, Kyuubi
is built on top of the
> >> compute
> >> > > > > layer
> >> > > > > > > > > > > (Apache Spark), while Apache
Drill is built on top
> of
> >> the
> >> > > > > storage
> >> > > > > > > > > > > layer.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Apache Spark has a large user
base and usage
> >> scenarios all
> >> > > > over
> >> > > > > > the
> >> > > > > > > > > > > world. When users have some
basic Spark programming
> >> and
> >> > > > tuning
> >> > > > > > > > skills,
> >> > > > > > > > > > > things are easy for them.
But many times, most users
> >> do not
> >> > > > > have
> >> > > > > > > all
> >> > > > > > > > > > > of these abilities when they
use Spark directly or
> >> > > > indirectly.
> >> > > > > We
> >> > > > > > > > want
> >> > > > > > > > > > > to build a more easy-to-use
platform on top of it to
> >> help
> >> > > > these
> >> > > > > > > > users.
> >> > > > > > > > > > > On the one hand, we will work
on the interface layer
> >> to
> >> > > make
> >> > > > it
> >> > > > > > > easy
> >> > > > > > > > > > > to access (of course, this
effort will make our
> >> project
> >> > > look
> >> > > > > very
> >> > > > > > > > > > > similar to other projects,
such as HiveServer2). On
> >> the
> >> > > other
> >> > > > > > hand,
> >> > > > > > > > we
> >> > > > > > > > > > > will work under Kyuubi's multi-tenant
architecture
> to
> >> > > extend
> >> > > > > our
> >> > > > > > > > > > > on-prem Spark SQL engine(a.k.a
Kyuubi engine) to
> make
> >> it
> >> > > more
> >> > > > > > > stable
> >> > > > > > > > > > > and efficient in our scenario.
> >> > > > > > > > > > >
> >> > > > > > > > > > > What's more, we are going
to add more on-prem Kyuubi
> >> engine
> >> > > > > > types,
> >> > > > > > > > e.g.
> >> > > > > > > > > > > Flink.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Alexander Alten <alo@scalytics.io>
于2021年5月31日周一
> >> 下午2:35写道:
> >> > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Hey,
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Looks like an interesting
project, competition to
> >> > > > Starburst.
> >> > > > > > How
> >> > > > > > > > > does it
> >> > > > > > > > > > > > compare to Apache Dremel?
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >  +1 (non-binding) for
incubation.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Thanks and stay safe,
> >> > > > > > > > > > > > --alex
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Mon, May 31, 2021,
08:21 Kent Yao <
> >> yao@apache.org>
> >> > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > Dear all,
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > We want to propose
Kyuubi as a new Apache
> >> Incubator
> >> > > > > project.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Kyuubi is a distributed
multi-tenant Thrift
> >> JDBC/ODBC
> >> > > > > server
> >> > > > > > > for
> >> > > > > > > > > > > > > large-scale data
management, processing, and
> >> analytics,
> >> > > > > built
> >> > > > > > > on
> >> > > > > > > > > top
> >> > > > > > > > > > > > > of Apache Spark
and designed to support more
> >> engines
> >> > > > (i.e.,
> >> > > > > > > > Apache
> >> > > > > > > > > > > > > Flink). We are aiming
to make Kyuubi an
> >> > > "out-of-the-box"
> >> > > > > tool
> >> > > > > > > for
> >> > > > > > > > > data
> >> > > > > > > > > > > > > warehouses and data
lakes. Kyuubi concentrates
> on
> >> > > > improving
> >> > > > > > the
> >> > > > > > > > > > > > > experience for non-Spark
users, making it easy
> >> for them
> >> > > > to
> >> > > > > > use
> >> > > > > > > > > Spark
> >> > > > > > > > > > > > > as a service and
focus on business data of their
> >> own,
> >> > > not
> >> > > > > > where
> >> > > > > > > > it
> >> > > > > > > > > > > > > stores, how it computes.
NetEase has
> open-sourced
> >> > > Kyuubi
> >> > > > > > since
> >> > > > > > > > > 2018,
> >> > > > > > > > > > > > > https://github.com/NetEase/kyuubi.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Many thanks to Willem
Jiang (Champion), Jeff
> >> Zhang, Duo
> >> > > > > > Zhang,
> >> > > > > > > > > Akira
> >> > > > > > > > > > > > > Ajisaka for being
the mentors of the Kyuubi
> >> project and
> >> > > > > > helping
> >> > > > > > > > us
> >> > > > > > > > > set
> >> > > > > > > > > > > > > up this proposal.
The proposal can be found at
> >> > > > > > > > > > > > >
> >> > > > > > > > >
> >> > > > >
> >> https://cwiki.apache.org/confluence/display/INCUBATOR/KyuubiProposal
> >> > > > > > .
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Looking forward
to your feedback and thank you
> >> very
> >> > > much.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Best regards,
> >> > > > > > > > > > > > > Kent Yao
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > >
> >> > > > >
> >> ---------------------------------------------------------------------
> >> > > > > > > > > > > > > To unsubscribe,
e-mail:
> >> > > > > > > general-unsubscribe@incubator.apache.org
> >> > > > > > > > > > > > > For additional commands,
e-mail:
> >> > > > > > > > general-help@incubator.apache.org
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > --
> >> > > > > > > > > > >
> >> > > > > > > > > > >  Kent Yao
> >> > > > > > > > > > > @ Data Science Center, Hangzhou
Research Institute,
> >> NetEase
> >> > > > > Corp.
> >> > > > > > > > > > > a spark enthusiast
> >> > > > > > > > > > > kyuubiis a unified multi-tenant
JDBC interface for
> >> > > > large-scale
> >> > > > > > data
> >> > > > > > > > > > > processing and analytics,
built on top of Apache
> >> Spark.
> >> > > > > > > > > > >
> >> > > > > > > > > > > spark-authorizerA Spark SQL
extension which provides
> >> SQL
> >> > > > > Standard
> >> > > > > > > > > > > Authorization for Apache Spark.
> >> > > > > > > > > > > spark-postgres A library for
reading data from and
> >> > > > transferring
> >> > > > > > > data
> >> > > > > > > > > > > to Postgres / Greenplum with
Spark SQL and
> DataFrames,
> >> > > > 10~100x
> >> > > > > > > > faster.
> >> > > > > > > > > > > itatchiA library that brings
useful functions from
> >> various
> >> > > > > modern
> >> > > > > > > > > > > database management systems
to Apache Spark.
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > >
> >> > >
> ---------------------------------------------------------------------
> >> > > > > > > > > > > To unsubscribe, e-mail:
> >> > > > > general-unsubscribe@incubator.apache.org
> >> > > > > > > > > > > For additional commands, e-mail:
> >> > > > > > general-help@incubator.apache.org
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > >
> >> ---------------------------------------------------------------------
> >> > > > > > > > > To unsubscribe, e-mail:
> >> > > general-unsubscribe@incubator.apache.org
> >> > > > > > > > > For additional commands, e-mail:
> >> > > > general-help@incubator.apache.org
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > >
> >> > > > > >  *Kent Yao *
> >> > > > > > @ Data Science Center, Hangzhou Research Institute,
NetEase
> >> Corp.
> >> > > > > > a spark enthusiast
> >> > > > > > kyuubi <https://github.com/yaooqinn/kyuubi>is
a unified
> >> multi-tenant
> >> > > > > JDBC
> >> > > > > > interface for large-scale data processing and analytics,
built
> >> on top
> >> > > > of
> >> > > > > > Apache
> >> > > > > > Spark <http://spark.apache.org/>.
> >> > > > > > spark-authorizer <
> https://github.com/yaooqinn/spark-authorizer
> >> >A
> >> > > Spark
> >> > > > > SQL
> >> > > > > > extension which provides SQL Standard Authorization
for Apache
> >> Spark
> >> > > > > > <http://spark.apache.org/>.
> >> > > > > > spark-postgres <https://github.com/yaooqinn/spark-postgres>
A
> >> > > library
> >> > > > > for
> >> > > > > > reading data from and transferring data to Postgres
/
> Greenplum
> >> with
> >> > > > > Spark
> >> > > > > > SQL and DataFrames, 10~100x faster.
> >> > > > > > itatchi <https://github.com/yaooqinn/spark-func-extras>A
> >> library
> >> > > that
> >> > > > > > brings useful functions from various modern database
> management
> >> > > > > > systems to Apache
> >> > > > > > Spark <http://spark.apache.org/>.
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > >
> >> > > >  *Kent Yao *
> >> > > > @ Data Science Center, Hangzhou Research Institute, NetEase Corp.
> >> > > > a spark enthusiast
> >> > > > kyuubi <https://github.com/yaooqinn/kyuubi>is a unified
> >> multi-tenant
> >> > > JDBC
> >> > > > interface for large-scale data processing and analytics, built
on
> >> top of
> >> > > > Apache
> >> > > > Spark <http://spark.apache.org/>.
> >> > > > spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A
> >> Spark
> >> > > SQL
> >> > > > extension which provides SQL Standard Authorization for Apache
> Spark
> >> > > > <http://spark.apache.org/>.
> >> > > > spark-postgres <https://github.com/yaooqinn/spark-postgres>
A
> >> library
> >> > > for
> >> > > > reading data from and transferring data to Postgres / Greenplum
> with
> >> > > Spark
> >> > > > SQL and DataFrames, 10~100x faster.
> >> > > > itatchi <https://github.com/yaooqinn/spark-func-extras>A
library
> >> that
> >> > > > brings useful functions from various modern database management
> >> > > > systems to Apache
> >> > > > Spark <http://spark.apache.org/>.
> >> > > >
> >> > >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: general-help@incubator.apache.org
> >>
> >> --
> > Thanks,
> > Ming Wen, Apache APISIX PMC Chair
> > Twitter: _WenMing
> >
> --
> Thanks,
> Ming Wen, Apache APISIX PMC Chair
> Twitter: _WenMing
>


-- 

 *Kent Yao *
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
a spark enthusiast
kyuubi <https://github.com/yaooqinn/kyuubi>is a unified multi-tenant JDBC
interface for large-scale data processing and analytics, built on top of Apache
Spark <http://spark.apache.org/>.
spark-authorizer <https://github.com/yaooqinn/spark-authorizer>A Spark SQL
extension which provides SQL Standard Authorization for Apache Spark
<http://spark.apache.org/>.
spark-postgres <https://github.com/yaooqinn/spark-postgres> A library for
reading data from and transferring data to Postgres / Greenplum with Spark
SQL and DataFrames, 10~100x faster.
itatchi <https://github.com/yaooqinn/spark-func-extras>A library that
brings useful functions from various modern database management
systems to​ Apache
Spark <http://spark.apache.org/>.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message