lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shad Storhaug (Jira)" <j...@apache.org>
Subject [jira] [Updated] (LUCENENET-616) Make Collections from Lucene.Net.Support into a 1st Class Feature
Date Fri, 27 Dec 2019 21:05:00 GMT

     [ https://issues.apache.org/jira/browse/LUCENENET-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Shad Storhaug updated LUCENENET-616:
------------------------------------
            Attachment: NullableKey.cs
                        NullableKeyDictionary.cs
         Fix Version/s: Lucene.Net 4.8.0
                 Flags: Important
           Description: 
The collection types in {{Lucene.Net.Support}} were originally sourced to support Lucene.Net
itself. While they were made public, they were not considered to be features that would be
used by anyone except for advanced users.

However, it has become clear by user reports that some parts of Lucene's design require specialized
collections that don't exist in the .NET Framework in order to properly function (see [LUCENENET-612|https://issues.apache.org/jira/projects/LUCENENET/issues/LUCENENET-612]
and [LUCENENET-615|https://issues.apache.org/jira/projects/LUCENENET/issues/LUCENENET-615].
.NET users are generally not familiar with these specialized collection types and assume that
when {{IDictionary<TKey, TValue>}} is required by an API that using {{Dictionary<TKey,
TValue>}} is their best choice. We need to improve documentation and increase visibility
of the specialized collection types in order to help them along.

Some of the existing collections are composed of other nested collection objects and should
be replaced with a lower-level implementation, if possible. Since these are breaking API changes,
they should be done before the official release of Lucene.Net 4.8.0.

Additionally, many parts of Lucene.NET expect the collections to:
 * Be structurally equal
 * Format their contents in the ToString() method

The safest and most thorough way to achieve this is to replace the usage of all built-in .NET
collections with collections that we own that implement this functionality. .NET provides
interfaces to help achieve this:
 * {{IStructuralEquatable}}
 * {{IFormattable}}

But neither interface is implemented in any of the built-in collections in .NET.
h2. *Requirements for Lucene.NET Collections*

We must have the following collections for internal use within Lucene.NET, and each should
also be made available to end users.
 * {{J2N.Collections.Generic.List<T>}}
 ** Subclass of {{System.Collections.Generic.List<T>}}
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.HashSet<T>}}
 ** Subclass of {{System.Collections.Generic.List<T>}}
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.SortedSet<T>}}
 ** Subclass of {{System.Collections.Generic.SortedSet<T>}}
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.Dictionary<TKey, TValue>}}
 ** Same interface as {{System.Collections.Generic.Dictionary<TKey, TValue>}}
 ** Supports null keys
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.SortedDictionary<TKey, TValue>}}
 ** Same interface as {{System.Collections.Generic.SortedDictionary<TKey, TValue>}}
 ** Supports null keys
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.LinkedDictionary<TKey, TValue>}}
 ** Same interface as {{System.Collections.Generic.Dictionary<TKey, TValue>}}
 ** Preserves insertion order across adds/deletes
 ** Supports null keys
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.PriorityQueue<T>}}
 ** Migrate {{Lucene.Net.Support.PriorityQueue<T>}} implementation, but clean up the
API for .NET (AddAll() > AddRange(), etc.)

 * {{J2N.Collections.Generic.IdentityDictionary<TKey, TValue>}} (See {{Lucene.Net.Support.IdentityHashMap<TKey,
TValue>}})
 ** Same interface as {{System.Collections.Generic.Dictionary<TKey, TValue>}}
 ** Supports null keys
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.IdentityHashSet<T>}} (See {{Lucene.Net.Support.IdentityHashSet<T>}})
 ** Subclass of {{System.Collections.Generic.HashSet<T>}}
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{ConcurrentHashSet<T>}} - (See LUCENENET-634)

We should have the following collections for internal use within Lucene.NET.
 * {{J2N.Collections.Generic.LinkedHashSet<T>}}
 ** Same interface as {{System.Collections.Generic.List<T>}}
 ** Preserves insertion order across adds/deletes
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Runtime.CompilerServices.ConditionalWeakTable<TKey, TValue>}} (See LUCENENET-636)

h3. *Special Cases*

There are a few uses where the above collections won't suffice because they use low-level
methods that don't exist in .NET collections. However, these are closed systems and we don't
have to worry about supporting end users in these cases.
 * {{Lucene.Net.TestFramework.Analysis.MockCharFilter}} > Use C5's {{TreeDictionary<TKey,
TValue}}
 * {{Lucene.Net.Grouping.AbstractGroupFacetCollector}} > Use C5's {{TreeSet<T>}}
 * {{Lucene.Net.Highlighter.PostingsHighlight.PostingsHighlighter}} > Use C5's {{TreeSet<T>}}

Note that C5's {{TreeSet<T>}} does not currently support {{ISet<T>}}, and we will
need to push our implementation of it back to them to support this change.
h3. *Structural Equality*

J2N has {{J2N.Collections.Generic.ListEqualityComparer<T>}}, {{J2N.Collections.Generic.SetEqualityComparer<T>}},
{{J2N.Collections.Generic.DictionaryEqualityComparer<TKey, TValue>}}, and {{J2N.Collections.StructuralEqualityComparer<T>}}
that implement the structural equality behavior for each type of collection. These types should
be passed in through each collection's constructor, as each has a {{Default}} and an {{Aggressive}}
implementation.

{{Default}} is more efficient and should be used in all closed scenarios within Lucene.NET
(where the collection is not passed in from outside of the class). {{Aggressive}} is designed
to interoperate with built-in .NET collections.

See [these minimal examples of implementations|https://github.com/NightOwl888/J2N/tree/a50d41668cf459f3fe827dff5f1746422155c4f6/src/J2N.TestFramework/Util].
h3. *Collection Formatting*

J2N has a {{StringFormatter}} class that can be used to format collections, as follows.
{code:c#}
// IFormattable
public string ToString(IFormatProvider provider)
{
    return string.Format(provider, "{0}", this);
}

public override string ToString()
{
    return ToString(StringFormatter.CurrentCulture);
}
{code}
h3. *Nullable Keys*

Use the attached {{NullableKey<T>}} and {{NullableKeyDictionary<TKey, TValue>}}
implementations. {{NullableKeyDictionary<TKey, TValue>}} is intended to be a base class
for all nullable key dictionary implementations. The backing dictionary (the one we are trying
to mimic) is then passed in through its constructor.

h3. *Integration*

After the collections have been implemented, we can eliminate the usage of {{Lucene.Net.Support.Collections.Equals()}},
{{Lucene.Net.Support.Collections.GetHashCode(), and {{Lucene.Net.Support.Collections.ToString()}}.


Usage in of {{Lucene.Net.Support.Collections.Equals()}} and {{Lucene.Net.Support.Collections.GetHashCode()
in Lucene.Net can be changed to the equality comparer matching the collection type.

For string formatting, the usage can simply be changed to {{collection.ToString()}} in production
code, and we can use in {{collection.ToString(J2N.Text.StringFormatter.InvariantCulture)}}
to patch tests.

For types where collections are passed from the outside that may contain nested collections,
we should either use {{Aggressive}} structural comparison, or document that if nested collections
are needed, the end user should use collections that implement {{IStructuralEquatable}}, such
as those from J2N.

  was:
The collection types in {{Lucene.Net.Support}} were originally sourced to support Lucene.Net
itself. While they were made public, they were not considered to be features that would be
used by anyone except for advanced users.

However, it has become clear by user reports that some parts of Lucene's design require specialized
collections that don't exist in the .NET Framework in order to properly function (see [LUCENENET-612|https://issues.apache.org/jira/projects/LUCENENET/issues/LUCENENET-612]
and [LUCENENET-615|https://issues.apache.org/jira/projects/LUCENENET/issues/LUCENENET-615].
.NET users are generally not familiar with these specialized collection types and assume that
when {{IDictionary<TKey, TValue>}} is required by an API that using {{Dictionary<TKey,
TValue>}} is their best choice. We need to improve documentation and increase visibility
of the specialized collection types in order to help them along.

Some of the existing collections are composed of other nested collection objects and should
be replaced with a lower-level implementation, if possible. Since these are breaking API changes,
they should be done before the official release of Lucene.Net 4.8.0.

Additionally, many parts of Lucene.NET expect the collections to:
 * Be structurally equal
 * Format their contents in the ToString() method

The safest and most thorough way to achieve this is to replace the usage of all built-in .NET
collections with collections that we own that implement this functionality. .NET provides
interfaces to help achieve this:
 * {{IStructuralEquatable}}
 * {{IFormattable}}

But neither interface is implemented in any of the built-in collections in .NET.
h2. *Requirements for Lucene.NET Collections*

We must have the following collections for internal use within Lucene.NET, and each should
also be made available to end users.
 * {{J2N.Collections.Generic.List<T>}}
 ** Subclass of {{System.Collections.Generic.List<T>}}
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.HashSet<T>}}
 ** Subclass of {{System.Collections.Generic.List<T>}}
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.SortedSet<T>}}
 ** Subclass of {{System.Collections.Generic.SortedSet<T>}}
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.Dictionary<TKey, TValue>}}
 ** Same interface as {{System.Collections.Generic.Dictionary<TKey, TValue>}}
 ** Supports null keys
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.SortedDictionary<TKey, TValue>}}
 ** Same interface as {{System.Collections.Generic.SortedDictionary<TKey, TValue>}}
 ** Supports null keys
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.LinkedDictionary<TKey, TValue>}}
 ** Same interface as {{System.Collections.Generic.Dictionary<TKey, TValue>}}
 ** Preserves insertion order across adds/deletes
 ** Supports null keys
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.PriorityQueue<T>}}
 ** Migrate {{Lucene.Net.Support.PriorityQueue<T>}} implementation, but clean up the
API for .NET (AddAll() > AddRange(), etc.)

 * {{J2N.Collections.Generic.IdentityDictionary<TKey, TValue>}} (See {{Lucene.Net.Support.IdentityHashMap<TKey,
TValue>}})
 ** Same interface as {{System.Collections.Generic.Dictionary<TKey, TValue>}}
 ** Supports null keys
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Collections.Generic.IdentityHashSet<T>}} (See {{Lucene.Net.Support.IdentityHashSet<T>}})
 ** Subclass of {{System.Collections.Generic.HashSet<T>}}
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{ConcurrentHashSet<T>}} - (See LUCENENET-634)

We should have the following collections for internal use within Lucene.NET.
 * {{J2N.Collections.Generic.LinkedHashSet<T>}}
 ** Same interface as {{System.Collections.Generic.List<T>}}
 ** Preserves insertion order across adds/deletes
 ** Implements {{IStructuralEquatable}}
 ** Implements {{IFormattable}}

 * {{J2N.Runtime.CompilerServices.ConditionalWeakTable<TKey, TValue>}} (See LUCENENET-636)

h3. Special Cases

There are a few uses where the above collections won't suffice because they use low-level
methods that don't exist in .NET collections. However, these are closed systems and we don't
have to worry about supporting end users in these cases.
 * {{Lucene.Net.TestFramework.Analysis.MockCharFilter}} > Use C5's {{TreeDictionary<TKey,
TValue}}
 * {{Lucene.Net.Grouping.AbstractGroupFacetCollector}} > Use C5's {{TreeSet<T>}}
 * {{Lucene.Net.Highlighter.PostingsHighlight.PostingsHighlighter}} > Use C5's {{TreeSet<T>}}

Note that C5's {{TreeSet<T>}} does not currently support {{ISet<T>}}, and we will
need to push our implementation of it back to them to support this change.

Structural Equality

J2N has {{J2N.Collections.Generic.ListEqualityComparer<T>}}, {{J2N.Collections.Generic.SetEqualityComparer<T>}},
{{J2N.Collections.Generic.DictionaryEqualityComparer<TKey, TValue>}}, and {{J2N.Collections.StructuralEqualityComparer<T>}}
that implement the structural equality behavior for each type of collection. These types should
be passed in through each collection's constructor, as each has a {{Default}} and an {{Aggressive}}
implementation.

{{Default}} is more efficient and should be used in all closed scenarios within Lucene.NET
(where the collection is not passed in from outside of the class). {{Aggressive}} is designed
to interoperate with built-in .NET collections.

See [these minimal examples of implementations|https://github.com/NightOwl888/J2N/tree/a50d41668cf459f3fe827dff5f1746422155c4f6/src/J2N.TestFramework/Util].

Collection Formatting

J2N has a {{StringFormatter}} class that can be used to format collections, as follows.
{code:c#}
// IFormattable
public string ToString(IFormatProvider provider)
{
    return string.Format(provider, "{0}", this);
}

public override string ToString()
{
    return ToString(StringFormatter.CurrentCulture);
}
{code}
Nullable Keys

Use the attached {{NullableKey<T>}} and {{NullableKeyDictionary<TKey, TValue>}}
implementations. {{NullableKeyDictionary<TKey, TValue>}} is intended to be a base class
for all nullable key dictionary implementations. The backing dictionary (the one we are trying
to mimic) is then passed in through its constructor.

    Remaining Estimate: 60h
     Original Estimate: 60h

> Make Collections from Lucene.Net.Support into a 1st Class Feature
> -----------------------------------------------------------------
>
>                 Key: LUCENENET-616
>                 URL: https://issues.apache.org/jira/browse/LUCENENET-616
>             Project: Lucene.Net
>          Issue Type: Improvement
>          Components: Lucene.Net Core
>    Affects Versions: Lucene.Net 4.8.0
>            Reporter: Shad Storhaug
>            Assignee: Shad Storhaug
>            Priority: Major
>             Fix For: Lucene.Net 4.8.0
>
>         Attachments: NullableKey.cs, NullableKeyDictionary.cs
>
>   Original Estimate: 60h
>  Remaining Estimate: 60h
>
> The collection types in {{Lucene.Net.Support}} were originally sourced to support Lucene.Net
itself. While they were made public, they were not considered to be features that would be
used by anyone except for advanced users.
> However, it has become clear by user reports that some parts of Lucene's design require
specialized collections that don't exist in the .NET Framework in order to properly function
(see [LUCENENET-612|https://issues.apache.org/jira/projects/LUCENENET/issues/LUCENENET-612]
and [LUCENENET-615|https://issues.apache.org/jira/projects/LUCENENET/issues/LUCENENET-615].
.NET users are generally not familiar with these specialized collection types and assume that
when {{IDictionary<TKey, TValue>}} is required by an API that using {{Dictionary<TKey,
TValue>}} is their best choice. We need to improve documentation and increase visibility
of the specialized collection types in order to help them along.
> Some of the existing collections are composed of other nested collection objects and
should be replaced with a lower-level implementation, if possible. Since these are breaking
API changes, they should be done before the official release of Lucene.Net 4.8.0.
> Additionally, many parts of Lucene.NET expect the collections to:
>  * Be structurally equal
>  * Format their contents in the ToString() method
> The safest and most thorough way to achieve this is to replace the usage of all built-in
.NET collections with collections that we own that implement this functionality. .NET provides
interfaces to help achieve this:
>  * {{IStructuralEquatable}}
>  * {{IFormattable}}
> But neither interface is implemented in any of the built-in collections in .NET.
> h2. *Requirements for Lucene.NET Collections*
> We must have the following collections for internal use within Lucene.NET, and each should
also be made available to end users.
>  * {{J2N.Collections.Generic.List<T>}}
>  ** Subclass of {{System.Collections.Generic.List<T>}}
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{J2N.Collections.Generic.HashSet<T>}}
>  ** Subclass of {{System.Collections.Generic.List<T>}}
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{J2N.Collections.Generic.SortedSet<T>}}
>  ** Subclass of {{System.Collections.Generic.SortedSet<T>}}
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{J2N.Collections.Generic.Dictionary<TKey, TValue>}}
>  ** Same interface as {{System.Collections.Generic.Dictionary<TKey, TValue>}}
>  ** Supports null keys
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{J2N.Collections.Generic.SortedDictionary<TKey, TValue>}}
>  ** Same interface as {{System.Collections.Generic.SortedDictionary<TKey, TValue>}}
>  ** Supports null keys
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{J2N.Collections.Generic.LinkedDictionary<TKey, TValue>}}
>  ** Same interface as {{System.Collections.Generic.Dictionary<TKey, TValue>}}
>  ** Preserves insertion order across adds/deletes
>  ** Supports null keys
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{J2N.Collections.Generic.PriorityQueue<T>}}
>  ** Migrate {{Lucene.Net.Support.PriorityQueue<T>}} implementation, but clean up
the API for .NET (AddAll() > AddRange(), etc.)
>  * {{J2N.Collections.Generic.IdentityDictionary<TKey, TValue>}} (See {{Lucene.Net.Support.IdentityHashMap<TKey,
TValue>}})
>  ** Same interface as {{System.Collections.Generic.Dictionary<TKey, TValue>}}
>  ** Supports null keys
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{J2N.Collections.Generic.IdentityHashSet<T>}} (See {{Lucene.Net.Support.IdentityHashSet<T>}})
>  ** Subclass of {{System.Collections.Generic.HashSet<T>}}
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{ConcurrentHashSet<T>}} - (See LUCENENET-634)
> We should have the following collections for internal use within Lucene.NET.
>  * {{J2N.Collections.Generic.LinkedHashSet<T>}}
>  ** Same interface as {{System.Collections.Generic.List<T>}}
>  ** Preserves insertion order across adds/deletes
>  ** Implements {{IStructuralEquatable}}
>  ** Implements {{IFormattable}}
>  * {{J2N.Runtime.CompilerServices.ConditionalWeakTable<TKey, TValue>}} (See LUCENENET-636)
> h3. *Special Cases*
> There are a few uses where the above collections won't suffice because they use low-level
methods that don't exist in .NET collections. However, these are closed systems and we don't
have to worry about supporting end users in these cases.
>  * {{Lucene.Net.TestFramework.Analysis.MockCharFilter}} > Use C5's {{TreeDictionary<TKey,
TValue}}
>  * {{Lucene.Net.Grouping.AbstractGroupFacetCollector}} > Use C5's {{TreeSet<T>}}
>  * {{Lucene.Net.Highlighter.PostingsHighlight.PostingsHighlighter}} > Use C5's {{TreeSet<T>}}
> Note that C5's {{TreeSet<T>}} does not currently support {{ISet<T>}}, and
we will need to push our implementation of it back to them to support this change.
> h3. *Structural Equality*
> J2N has {{J2N.Collections.Generic.ListEqualityComparer<T>}}, {{J2N.Collections.Generic.SetEqualityComparer<T>}},
{{J2N.Collections.Generic.DictionaryEqualityComparer<TKey, TValue>}}, and {{J2N.Collections.StructuralEqualityComparer<T>}}
that implement the structural equality behavior for each type of collection. These types should
be passed in through each collection's constructor, as each has a {{Default}} and an {{Aggressive}}
implementation.
> {{Default}} is more efficient and should be used in all closed scenarios within Lucene.NET
(where the collection is not passed in from outside of the class). {{Aggressive}} is designed
to interoperate with built-in .NET collections.
> See [these minimal examples of implementations|https://github.com/NightOwl888/J2N/tree/a50d41668cf459f3fe827dff5f1746422155c4f6/src/J2N.TestFramework/Util].
> h3. *Collection Formatting*
> J2N has a {{StringFormatter}} class that can be used to format collections, as follows.
> {code:c#}
> // IFormattable
> public string ToString(IFormatProvider provider)
> {
>     return string.Format(provider, "{0}", this);
> }
> public override string ToString()
> {
>     return ToString(StringFormatter.CurrentCulture);
> }
> {code}
> h3. *Nullable Keys*
> Use the attached {{NullableKey<T>}} and {{NullableKeyDictionary<TKey, TValue>}}
implementations. {{NullableKeyDictionary<TKey, TValue>}} is intended to be a base class
for all nullable key dictionary implementations. The backing dictionary (the one we are trying
to mimic) is then passed in through its constructor.
> h3. *Integration*
> After the collections have been implemented, we can eliminate the usage of {{Lucene.Net.Support.Collections.Equals()}},
{{Lucene.Net.Support.Collections.GetHashCode(), and {{Lucene.Net.Support.Collections.ToString()}}.

> Usage in of {{Lucene.Net.Support.Collections.Equals()}} and {{Lucene.Net.Support.Collections.GetHashCode()
in Lucene.Net can be changed to the equality comparer matching the collection type.
> For string formatting, the usage can simply be changed to {{collection.ToString()}} in
production code, and we can use in {{collection.ToString(J2N.Text.StringFormatter.InvariantCulture)}}
to patch tests.
> For types where collections are passed from the outside that may contain nested collections,
we should either use {{Aggressive}} structural comparison, or document that if nested collections
are needed, the end user should use collections that implement {{IStructuralEquatable}}, such
as those from J2N.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message