incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henry Saputra <henry.sapu...@gmail.com>
Subject Re: [DISCUSS] Mnemonic incubator proposal
Date Mon, 29 Feb 2016 16:20:28 GMT
Looks like the discussions had calmed down with no objection so I think we
could proceed with the VOTE thread.

- Henry

On Wed, Feb 24, 2016 at 12:15 PM, Wang, Yanping <yanping.wang@intel.com>
wrote:

> In general, Mnemonic can be integrated into many projects. First, projects
> can use Mnemonic to take data off Java heap so GC can be much reduced, and
> use GET/SET to access data fields so serdes can be eliminated.
> Later we can expand Mnemonic to excise persistent/non-volatile programming
> on large scaled distributed systems with TB sized fast persistent memory
> devices.
>
> Regarding solving Hadoop Namenode pressure of large scale of cluster
> scenarios. This issue is due to HDFS.
> Last year we found the use of FileInputStream in HDFS causes unpredicted
> long Garbage Collection pauses due to the overhead of finalizers and
> significantly impacted HDFS performance and its scalability. We recorded
> the issue in https://issues.apache.org/jira/browse/HDFS-8562
>
> Uma explained what we can do for using Mnemonic to improve HDFS
> performance and scalability. One big advantage is Mnemonic does not need to
> hold File System cache for random access, which will benefit large scale of
> clusters.
>
> Thanks
> yanping
>
>
>
> -----Original Message-----
> From: Gangumalla, Uma [mailto:uma.gangumalla@intel.com]
> Sent: Tuesday, February 23, 2016 8:06 PM
> To: general@incubator.apache.org
> Subject: Re: [DISCUSS] Mnemonic incubator proposal
>
> Hi Liang,
>
> Thank you for your interest. Sure we would consider you adding in
> interested contributors list.
>
> >Mnemonic is trying to solve performance issues associated
> with serialization/deserialization of java object when dealing with JVM &
> disk directly as well as GC pressure caused by caching ?
>
> Yes.
>
> >whether Mnemonic could solve Hadoop Namenode pressure of large
> scale of cluster scenaros, or not?
> Yeah, we are thinking on some aspects considering memory and GC overheads
> in Namenode too.
> Example couple of JIRAs already there in HDFS to move some of data
> structure to off heap. So, we had plans to get the standard data structures
> from this library and can make use of them push.
> Also we could make advantage if persistence here.
>
>
> @Yanping/Gary, may be you could add more points if you have?
>
> [Gary] Thanks Uma, in addition, you can plug-in your special allocators
> that could be optimized for namenode usage patterns, by this way, the
> performance could be better and more predictable. Thanks.
>
> Regards,
> Uma
>
> On 2/23/16, 6:48 PM, "Liang Chen" <chenliang613@huawei.com> wrote:
>
> >Interesting, would love to become the contributor
> >
> >My understanding: Mnemonic is trying to solve performance issues
> >associated with serialization/deserialization of java object when
> >dealing with JVM & disk directly as well as GC pressure caused by
> >caching ?
> >
> >one question: whether Mnemonic could solve Hadoop Namenode pressure of
> >large scale of cluster scenaros, or not?
> >
> >
> >
> >--
> >View this message in context:
> >http://apache-incubator-general.996316.n3.nabble.com/DISCUSS-Mnemonic-i
> >ncu
> >bator-proposal-tp48502p48533.html
> >Sent from the Apache Incubator - General mailing list archive at
> >Nabble.com.
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >For additional commands, e-mail: general-help@incubator.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message