Figure out better way for dealing with CPU arches

[h-vetinari]

While debugging #22, I found out that the upstream build-refactor that I tried to follow in #17 changed things so that upstream always compiles for both AVX2 & default profiles, and then switches at runtime using:
https://github.com/facebookresearch/faiss/blob/v1.6.5/faiss/python/loader.py#L27-L39

Of course we could mimick that, but given the infrastructure added in conda/conda#9930 by @isuruf, I'm thinking we could do better. Not sure if this is ready for primetime yet -- as in, does CF infra support this already?

I see the following todos (restricted to this feedstock):

• Figure out if we want to build for AVX2, resp. other profiles
• If yes, decide trade-off between packaging size impact for packaging several (probably negligible) and multiplication of CI jobs
• Patch the upstream loader depending on outcome
  • If we keep the upstream approach, add windows detection (how?) and upstream the patch

CC @beckermr @mbargull @jakirkham

[isuruf]

I'd suggest going with upstream and shipping both. archspec based packages are not ready yet.

[h-vetinari]

Thanks @isuruf for that update! Is there any place to track the archspec-based approach so I can subscribe (or possibly help) and test/apply it once done?

[h-vetinari]

Fortunately, there's now an issue to track progress on this: conda-forge/conda-forge.github.io#1261

[ngam]

I'm thinking we could do better

In what way? Size and unnecessary complexity?

I am often tempted to think about this in general like the noarch situation --- one package for all, but maybe that's the naive way to think about this 😅

[h-vetinari]

I'm thinking we could do better

In what way? Size and unnecessary complexity?

Maximize performance & minimize binary size, by building different package variants (e.g. for the x86-v2 - x86-v4 levels) and having each user automatically download the highest-supported binary according to their CPU, via the __archspec virtual package.

one package for all, but maybe that's the naive way to think about this 😅

Doing that doubles (or triples, if building for AVX512 as well) build time & binary size for no other reason other than that we cannot (currently) do better infrastructurally, and don't want to completely forgo the performance benefits that newer CPU arches have via-à-vis SSE4.