lucenenet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Connie Yau <conn...@microsoft.com>
Subject RE: ICU4NET 64bit?
Date Sat, 12 Nov 2016 02:13:15 GMT
When you execute `dotnet build` in the current .NET Core toolchain uses csc.dll which in turn
calls vbcscompiler (Roslyn) to compile the C# code into IL.  This also applies to .xprojs
when you run `msbuild Lucene.Net.Portable.sln`, it'll call `dotnet build` under the covers
which then calls csc.dll.  dotnet.exe and msbuild.exe understand how to process project.jsons
and csproj/xproj files so that they can pass the appropriate arguments to csc.exe for compilation.

The issue I was mentioning is that when I run `dotnet build --runtime win7-x86`, the output
binaries will be x64 binaries because my .NET Core SDK is x64. This can be worked around by
installing .NET Core SDK x86 and running `dotnet build` with that instead.  In future releases
of the .NET Core tooling, we switch to msbuild, which is why that issue (https://github.com/dotnet/cli/issues/3387)
wasn't fixed.

Icu-dotnet has RuleBasedCollator, so that shouldn't be a blocker anymore once I move to using
icu-dotnet.

Is it better to solve this issue creating a separate PR against the master branch that switches
to using icu-dotnet?

Connie


-----Original Message-----
From: Shad Storhaug [mailto:shad@shadstorhaug.com] 
Sent: Friday, November 11, 2016 6:22 AM
To: Connie Yau <conniey@microsoft.com>
Cc: Itamar Syn-Hershko <itamar@code972.com>; dev@lucenenet.apache.org
Subject: RE: ICU4NET 64bit?

Hi Connie,

Maybe this is a naïve question, but do we need to use MSBuild anymore? I mean, doesn't Roslyn
have the capability to compile both the .NET 4.6 and .NET Core stacks? And if we used Roslyn
exclusively for builds, would it help with solving the bitness issue?

Also, ICU4Net is missing the RuleBasedCollator, which is a dependency for the last missing
piece of Lucene.Net.Analysis.Common (Lucene.Net.Analysis.Collation). Using ICU4NET on 32 bit
builds, we won't be able to add this missing piece to the Lucene.Net.Analysis.Common package.
Note that most of the work has already been completed for Lucene.Net.Analysis.Collation on
this branch: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FChristopherHaws%2Flucenenet%2Ftree%2Fanalysis-work-2%2Fsrc%2FLucene.Net.Analysis.Common%2FCollation&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709110996071&sdata=AOZcWZOLMs5bQ19WAj7GZMF7UrtaYgisa7Y%2Bi8TNqvk%3D&reserved=0

I am not sure if Lucene.Net.Analysis.Collation is critical for the Lucene.Net.Analysis.Common
package release - in the .NET world there is no RuleBasedCollator, so it will most likely
not be missed if this extension for it is not there. But, it would be nice to include it if
we can. Perhaps providing Lucene.Net.Analysis.Collation to the 64 bit release and excluding
it for 32 bit using conditional compilation is an option, although it seems like an uncommon
thing to do.

Another package (important for Lucene.Net release) that is dependent on ICU's (BreakIterator)
is Lucene.Net.Highlighter, which is (patiently) awaiting the resolution of the ICU dependency
before it is ported.

Status
=====

List of (known) sections remaining to port/complete for release:

1. Lucene.Net.Analysis.Collation (Namespace in Lucene.Net.Analysis.Common) 2. Lucene.Net.Highlighter
3. Lucene.Net.Store (Namespace in Lucene.Net.Misc - Native (C++) directories for Unix and
Windows - important??) 4. Lucene.Net.Spatial (currently WIP) 5. Lucene.Net.QueryParsers.Flexible
(Namespace in Lucene.Net.QueryParser)

Other useful packages not yet ported that are not critical for release:

1. Lucene.Net.Analysis.ICU
2. Lucene.Net.Analysis.Kuromoji
3. Lucene.Net.Analysis.Morfologik
4. Lucene.Net.Analysis.Phonetic
5. Lucene.Net.Analysis.SmartCn
6. Lucene.Net.Analysis.UIMA
7. Lucene.Net.Demo

Lucene.Net.Analysis.ICU also depends on icu.net. Although Itamar said that Lucene.Net.Analysis.ICU
is not critical to port for release, would it make sense to port it sooner rather than later
to make sure we have all of the support we need in icu.net?

Thanks,
Shad Storhaug (NightOwl888)


-----Original Message-----
From: Connie Yau [mailto:conniey@microsoft.com]
Sent: Friday, November 11, 2016 6:01 AM
To: Itamar Syn-Hershko; dev@lucenenet.apache.org
Subject: RE: ICU4NET 64bit?

Hi,

I apologise for the delayed response.  In the .NET Core migration<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Flucenenet%2Fpull%2F191%2F&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=c65gRVNTMchVpOCh8Cef1iz69S0TzJsOdYEh4WEl84g%3D&reserved=0>
I am working on, with the current .NET Core toolchain, it compiles for x64 even if a 32-bit
runtime is specified. :( (It’s a known issue<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fcli%2Fissues%2F3387&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=nouDNzHkONT4xy0wA5bYJSnLYzpBPP4828IT7CsbzWc%3D&reserved=0>
they are not going to fix because of the migration to msbuild.) As a result, the only way
I could get the Lucene.Net.Analysis.Common library running was to change it to use 64-bit
version of icu.net<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.myget.org%2Ffeed%2Ficu-dotnet%2Fpackage%2Fnuget%2Ficu.net-x64&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=kOaNxMGNyVgctWSJ7KRDLai2ugYC8Wkh1y5yzyyJTZg%3D&reserved=0>
that I compiled from the sillsdev/icu-dotnet repository<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsillsdev%2Ficu-dotnet&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=v2%2BRqQNZW%2FSJT5DIJ01ffJs0EnuH0xXUuDzCND%2BUXUI%3D&reserved=0>.

As an aside, I can create a separate PR which uses x86 version of icu-dotnet that I’ve replaced
ICU4NET with as a solution for this issue.

Connie

From: itamar.synhershko@gmail.com [mailto:itamar.synhershko@gmail.com] On Behalf Of Itamar
Syn-Hershko
Sent: Thursday, November 10, 2016 12:44 PM
To: dev@lucenenet.apache.org
Cc: Connie Yau <conniey@microsoft.com>
Subject: Re: ICU4NET 64bit?

Ok, let's see what ICU support Shad and Connie come up with. Worst case we can release all
analyzers requiring the ICU packages as a separate package and remove the ICU dependency from
the more commonly used parts.

--

Itamar Syn-Hershko
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcode972.com&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=LpwVcwgjpFB9SBWyiMMBLSMyE4gUW8BSKpilw23yw5Y%3D&reserved=0<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcode972.com%2F&data=01%7C01%7Cconniey%40microsoft.com%7C876a41dab5fd4e99406008d409aa4498%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=hQQo6v997y%2Bdt1fJvs7k3Q6zV1lZjBQHyVXIDqGhuCs%3D&reserved=0>
| @synhershko<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fsynhershko&data=01%7C01%7Cconniey%40microsoft.com%7C876a41dab5fd4e99406008d409aa4498%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=4H05YsiyAsrOceyRCLuMgtqlOavX8%2FeL%2BUvz8%2FdU2fE%3D&reserved=0>
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Wed, Nov 9, 2016 at 11:53 PM, Roethinger, Alexander <aroethinger@affili.net<mailto:aroethinger@affili.net>>
wrote:
Good morming!

I can confirm that this is an issue. Spent two hours the other night trying to deploy my solution
on a Windows Server 2012 VM to no avail. I kept on getting the known Assembly-Loading exceptions
despite setting PATH variable or copying the native DLLs to the Windows directory.

Forking Lucene.Net.Analysis.Common in order to manually remove the Thai-Analyzer and Wordbreaker
is not an option as it defies the advantage of your release packages as well as other compilation
issues.

I finally ended up simply removing the assembly references to ICU4NET and ICU4NET.Extensions
from my projects that use Lucene.Net.Analysis.Common and voila, the code still runs and I
could deploy without needing the native DLLs.

While I know that my application would crash if I were to access the Thai analyzer, since
I'm not using it, it seemed safe to proceed that way.

In any case it would be great to get rid of ICU4NET dependency, either by moving the Thai-Analyzer
and WordBreaker to a separate assembly (which only solves the problem partially) or some true
CLR alternative.

Cheers
Alexander




-----Ursprüngliche Nachricht-----
Von: itamar.synhershko@gmail.com<mailto:itamar.synhershko@gmail.com> [mailto:itamar.synhershko@gmail.com<mailto:itamar.synhershko@gmail.com>]
Im Auftrag von Itamar Syn-Hershko
Gesendet: Donnerstag, 10. November 2016 05:09
An: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>
Cc: Connie Yau <conniey@microsoft.com<mailto:conniey@microsoft.com>>
Betreff: Re: ICU4NET 64bit?

I've had several reports on ICU4NET dependencies mostly on cloud environments - probably due
to the native assemblies that are shipped with it. Let's make sure we are using a solution
that runs on the CLR and work well.

--

Itamar Syn-Hershko
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcode972.com&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=LpwVcwgjpFB9SBWyiMMBLSMyE4gUW8BSKpilw23yw5Y%3D&reserved=0<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcode972.com&data=01%7C01%7Cconniey%40microsoft.com%7C876a41dab5fd4e99406008d409aa4498%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=YkdTdD8FbJ6kjuriq8oEJEIHZEdWCmLaOPJTWQTWILs%3D&reserved=0>
| @synhershko <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fsynhershko&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=AciVwx7K%2B6B87%2FDWjZnUjNPeqaJwx%2F89QbZ8%2Fe0NCmg%3D&reserved=0<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fsynhershko&data=01%7C01%7Cconniey%40microsoft.com%7C876a41dab5fd4e99406008d409aa4498%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=4H05YsiyAsrOceyRCLuMgtqlOavX8%2FeL%2BUvz8%2FdU2fE%3D&reserved=0>>
Freelance Developer & Consultant Lucene.NET committer and PMC member

On Thu, Nov 3, 2016 at 11:55 AM, Laimonas Simutis <laimis@gmail.com<mailto:laimis@gmail.com>>
wrote:

> Alexander,
>
> I am pretty sure that is the same code that you linked. I just created 
> a nuget package with all the ICU4C dlls included so that we could 
> reference them in Lucene port via nuget without having to include c 
> libs somewhere else.
>
> I will need to go back and see where I ended up with the 64 bit port.
> Although as Shad mentions, is this still relevant since for the long 
> term the port are going away from ICU4NET?
>
>
>
> On Thu, Nov 3, 2016 at 10:04 AM, Shad Storhaug 
> <shad@shadstorhaug.com<mailto:shad@shadstorhaug.com>>
> wrote:
>
> > Alexander,
> >
> > Actually, Connie Yau from Microsoft is working on porting to .NET 
> > core, and one of the things that is happening is that she is 
> > switching the dependency from ICU4NET to icu.net<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ficu.net&data=01%7C01%7Cconniey%40microsoft.com%7C876a41dab5fd4e99406008d409aa4498%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=iDTVkPtaza84wfQvIP1WUi2U%2FStVLCUvD6LCs3hYSiE%3D&reserved=0>.
> >
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
> > hub.com%2Fsillsdev%2Ficu-dotnet&data=02%7C01%7Cconniey%40microsoft.c
> > om%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011
> > db47%7C1%7C0%7C636144709111006079&sdata=v2%2BRqQNZW%2FSJT5DIJ01ffJs0
> > EnuH0xXUuDzCND%2BUXUI%3D&reserved=0<https://na01.safelinks.protectio
> > n.outlook.com/?url=https%3A%2F%2Fna01.safelinks.protec&data=02%7C01%
> > 7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f98
> > 8bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=gLBrp
> > ri7Q7bT1BG9Q1zgXfqwZ%2ByZKu%2FSM%2B7U8lLeip8%3D&reserved=0
> > tion.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsillsdev%2Ficu-dotn
> > et&data=01%7C01%7Cconniey%40microsoft.com%7C876a41dab5fd4e99406008d4
> > 09aa4498%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=r8itnCstaCa0FO
> > L6mrSjWa2xYEPFMN0Sat9EQGFdpng%3D&reserved=0>
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Flucenenet%2Fpull%2F191&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=OEzuuiBgZLPUGfLsaMaJ6bRFa5UKxfOsI%2FeZ5BMyNKM%3D&reserved=0<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fna01.safelinks&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=2sSzbRZdT0POme2gxcmuT1KSqHvl8V2Rid5A9BmED2s%3D&reserved=0.
> > protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fluce
> > nenet%2Fpull%2F191&data=01%7C01%7Cconniey%40microsoft.com%7C876a41da
> > b5fd4e99406008d409aa4498%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdat
> > a=4L4penTOgo4T0EjS35aviV0m6cK63hxdAyk5JYhP%2FSg%3D&reserved=0>
> >
> > So at this point, making changes to ICU4NET is moot. However, I am 
> > not sure what that means in terms of 64 bit compatibility. Both of 
> > these packages are just wrappers around the ICU4C library.
> >
> > Connie, perhaps you can share what you know? If Alexander is willing 
> > to update that package to work with 64 bit (and fix the issues with 
> > auto-loading), that would be great.
> >
> >
> > Thanks,
> > Shad Storhaug (NightOwl888)
> >
> > From: Roethinger, Alexander
> > [mailto:aroethinger@affili.net<mailto:aroethinger@affili.net>]
> > Sent: Thursday, November 3, 2016 9:37 PM
> > To: dev@lucenenet.apache.org<mailto:dev@lucenenet.apache.org>
> > Subject: ICU4NET 64bit?
> >
> > Dear Devs,
> >
> > can anyone point me to the source-code (if available) for the 
> > "ICU4NET packaged together with ICU4C 55.1 32-bit release"
> > package available on Nuget and provided by Laimonas Simutis?
> >
> > Or is this the same code as in
> > "Simple wrapper for ICU4C (https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsite.icu-project.org%2F&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=dczijNKi8O9XQBYNxlDSSYqpo1pc3gjizdhn36JMk2Q%3D&reserved=0)<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsite.icu-project.org%2F)&data=01%7C01%7Cconniey%40microsoft.com%7C876a41dab5fd4e99406008d409aa4498%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=OlmEp4NxmBWbGIlQ3MW7PsEGyU4oWB79LT31mJATHNM%3D&reserved=0>."
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fniaher%2Ficu4net&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=7DsF8ufyaitRhzsJimjIt2BeR6RAqsElnOwGEjvMVKQ%3D&reserved=0<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fna01.safelinks.protection&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=9K3xPGl6EoWMw4NXE7M1cIP1KV5zzvNJ7F%2FwN0GIU%2Fc%3D&reserved=0.
> > outlook.com/?url=https%3A%2F%2Fgithub.com%2Fniaher%2Ficu4net&data=01
> > %7C01%7Cconniey%40microsoft.com%7C876a41dab5fd4e99406008d409aa4498%7
> > C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=vurBYCkwS1BiswMx2vg2%2B4
> > gLu92ReatRBS7A3Fgni2w%3D&reserved=0>
> > ?
> >
> >
> > I would like to build a 64bit (resp. AnyCPU) version using the 64bit 
> > DLLs available at ICU-Homepage.
> > I believe Laimonas had mentioned in a post that he apparently had 
> > already successfully tested a 64bit version.
> >
> > Are there plans to provide 64bit support for ICU4NET?
> > (I know there's a discussion going regarding the dependency on
> > ICU4NET)
> >
> > Thanks
> > Alexander
> >
>

Mime
View raw message