Intel Clear Linux ignores Zen's AVX2, FMA, BMI in Math library for Phyton, C/C++, SQL, ....

Page 2 - Seeking answers? Join the AnandTech community: where nearly half-a-million members share solutions and discuss the latest tech.

moinmoin

Senior member
Jun 1, 2017
988
755
106
Ironically though, Clear Linux still runs AMD stupid fast even with Intel's shenanigans.

It's not ironic but a consequence of most Linux distributions aiming for the largest possible install base. We are slowly seeing support for 32bit i386 to i586 targets being reduced. Most common is still the now decade old 64bit i686 target. Everything newer, including any AVX support, is locked behind targets like "haswell" so usual Linux distributions (including all its software unless specifically compiled with other targets) don't profit of them. Intel started Clear Linux specifically to showcase how much performance common distributions are leaving lying dormant that way, so of course newer AMD chips are profiting of them as well.
 

Nothingness

Golden Member
Jul 3, 2013
1,931
246
106
Everything newer, including any AVX support, is locked behind targets like "haswell" so usual Linux distributions (including all its software unless specifically compiled with other targets) don't profit of them.
You are wrong: recent glibc have dedicated paths for AVX and this is deployed in all recent distros. And that's why I believe that even though the dl_platform stuff should be fixed, there likely is no impact for typical users.
 
Last edited:

moinmoin

Senior member
Jun 1, 2017
988
755
106
You are wrong: recent glibc have dedicated paths for AVX and this is deployed in all recent distros. And that's why I don't believe that even though the dl_platform stuff should be fixed, there likely is no impact for typical users.
Do you know if those paths are per feature (so the can be activate once the CPU claims support for it), or more generic? If the former, what's the point of all the code quoted in OP, wouldn't that be completely redundant?
 

Nothingness

Golden Member
Jul 3, 2013
1,931
246
106
Do you know if those paths are per feature (so the can be activate once the CPU claims support for it), or more generic? If the former, what's the point of all the code quoted in OP, wouldn't that be completely redundant?
They are per feature as far as I can see:

https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_64/multiarch/strchr.c;h=9ff0b63b6de60e72c7bbfb931bec3c9828d26ea7;hb=02d35f0646050157d47f22f2f74347431d5ccbdd

The strchr function uses features to select the implementation that matches the features found on the CPU at runtime. These features are detected when the library is loaded.

The code in the original post is here to load full libraries tuned for a specific dl_platform. The features are used in a library to change the function used at runtime.
 
  • Like
Reactions: moinmoin

moinmoin

Senior member
Jun 1, 2017
988
755
106
They are per feature as far as I can see:

https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_64/multiarch/strchr.c;h=9ff0b63b6de60e72c7bbfb931bec3c9828d26ea7;hb=02d35f0646050157d47f22f2f74347431d5ccbdd

The strchr function uses features to select the implementation that matches the features found on the CPU at runtime. These features are detected when the library is loaded.

The code in the original post is here to load full libraries tuned for a specific dl_platform. The features are used in a library to change the function used at runtime.
Thanks for the corrections and clarification.

Any reason why those specific libraries couldn't include such feature paths as well instead this what seems like a band aid approach?
 

Glo.

Diamond Member
Apr 25, 2015
3,178
1,513
136
It's not ironic but a consequence of most Linux distributions aiming for the largest possible install base. We are slowly seeing support for 32bit i386 to i586 targets being reduced. Most common is still the now decade old 64bit i686 target. Everything newer, including any AVX support, is locked behind targets like "haswell" so usual Linux distributions (including all its software unless specifically compiled with other targets) don't profit of them. Intel started Clear Linux specifically to showcase how much performance common distributions are leaving lying dormant that way, so of course newer AMD chips are profiting of them as well.
Clear Linux Project is targetting also the widest, largest install base, possible.
 

Glo.

Diamond Member
Apr 25, 2015
3,178
1,513
136
lol, they fail at the package manager already.
But they have plans in improving this Distro. There is a good reason why Linux community considers Clear Linux, if Intel will fix some things, as the next flagship distro. Yes, it might replace Fedora as a go-to flaghsip distro(but Im sure it will not replace MX Linux and Manjaro and Ubuntu in Popularity).

This is work in progress project, that still is in its infancy.
 

moinmoin

Senior member
Jun 1, 2017
988
755
106
But they have plans in improving this Distro. There is a good reason why Linux community considers Clear Linux, if Intel will fix some things, as the next flagship distro. Yes, it might replace Fedora as a go-to flaghsip distro(but Im sure it will not replace MX Linux and Manjaro and Ubuntu in Popularity).

This is work in progress project, that still is in its infancy.
Then they are not covering the largest possible install base but only planning to eventually. There are many parts that must be finished before a Linux distribution can be easily recommended to any users without knowing details about their usage.

Since you mentioned Fedora (which I happen to use as my main system for years), Fedora is about the lowest maintenance quasi rolling release Linux distribution there is, pretty much always being on the newest versions of kernel and software without anything breaking (at least in my case, anecdotal evidence and all that..). Other popular distributions don't compare to it, Clear Linux doesn't (yet) compare to them.
 

Glo.

Diamond Member
Apr 25, 2015
3,178
1,513
136
Then they are not covering the largest possible install base but only planning to eventually. There are many parts that must be finished before a Linux distribution can be easily recommended to any users without knowing details about their usage.

Since you mentioned Fedora (which I happen to use as my main system for years), Fedora is about the lowest maintenance quasi rolling release Linux distribution there is, pretty much always being on the newest versions of kernel and software without anything breaking (at least in my case, anecdotal evidence and all that..). Other popular distributions don't compare to it, Clear Linux doesn't (yet) compare to them.
Because Intel is targetting the widest, largest Install base, is the exact reason why they are doing what they are doing with it. Performance, security, stability, are three pillars on which this distro, is supposed to be built.

I still would not recommend it for anyone to use it, its too much in its infancy. But from all of the distros I've used it shows signs of highest potential. The only two other distros that have similar potential, from what I have tested over past... 9 months are Pop!_OS and Manjaro.
 

beginner99

Diamond Member
Jun 2, 2009
4,209
623
126
How does this matters? Doesn't numpy and co. by default ship anyway only with SSE and you need to make your custom build to get AVX support? Or alternatively get a precompiled version of the libs for example through conda?
 

Thala

Senior member
Nov 12, 2014
800
177
116
Thanks for the corrections and clarification.

Any reason why those specific libraries couldn't include such feature paths as well instead this what seems like a band aid approach?
The point is they dont have to. You just compile the library for a very specific target avoiding all the dynamic code path decisions at runtime plus you reduce footprint apparently. It also has advantages with regards to verification.
So no, its not a band-aid but an optimization.
 

moinmoin

Senior member
Jun 1, 2017
988
755
106
Performance, security, stability, are three pillars on which this distro, is supposed to be built.
So they got performance, two more to go.

The point is they dont have to. You just compile the library for a very specific target avoiding all the dynamic code path decisions at runtime plus you reduce footprint apparently. It also has advantages with regards to verification.
So no, its not a band-aid but an optimization.
The very specific target optimization then turns out to be very picky about when it is usable. How is that any advantage over the working dynamic code path decisions at runtime that actually check what features any given (future) CPU actually supports and offer respective optimizations?

This reminds me of websites checking for specific browsers and locking out all the rest instead simply checking for supported feature and allowing any browser in once it does.
 

Abwx

Diamond Member
Apr 2, 2011
9,037
789
126
The point is they dont have to. You just compile the library for a very specific target avoiding all the dynamic code path decisions at runtime plus you reduce footprint apparently. It also has advantages with regards to verification.
So no, its not a band-aid but an optimization.
From Wiki :


As of 2019, MKL, which remains the choice of many pre-compiled Mathematical applications on Windows (such as NumPy, SymPy, and MATLAB), still significantly underperforms on AMD CPUs with equivalent instruction sets .

 

Nothingness

Golden Member
Jul 3, 2013
1,931
246
106
From Wiki :


As of 2019, MKL, which remains the choice of many pre-compiled Mathematical applications on Windows (such as NumPy, SymPy, and MATLAB), still significantly underperforms on AMD CPUs with equivalent instruction sets .

Thala was talking about the libm, not the Intel math library.

Why should Intel spend money on making their software run fast on AMD CPUs? That's the same as icc. Would you expect AMD to make sure their dedicated tools work well on Intel CPUs? Blame the software companies that use Intel tools without checking performance on AMD platforms (BTW NumPy can be compiled with OpenBLAS).
 

ASK THE COMMUNITY