OSX Test and Monitor Test GA tests are failing for all Pythons

[valeriupredoi]

Last night's OSX tests have all died for all Python with a multitude of problems - it seems fiona is at it again, and a lot of FLAKE8 problems. Diffing the envs between passed and failed identical tests, I find these packages have changed:

<     + curl                             7.82.0  h9f20792_0                   conda-forge/osx-64                         149kB
---
>     + curl                             7.81.0  hf45b732_0                   conda-forge/osx-64                         148kB
53c53
<     + expat                             2.4.7  h96cf925_0                   conda-forge/osx-64                         150kB
---
>     + expat                             2.4.6  h96cf925_0                   conda-forge/osx-64                         149kB
61c61
<     + fontconfig                      2.13.96  h676cef8_1                   conda-forge/osx-64                         282kB
---
>     + fontconfig                      2.13.96  h10f422b_0                   conda-forge/osx-64                         297kB
123c123
<     + libcurl                          7.82.0  h9f20792_0                   conda-forge/osx-64                         327kB
---
>     + libcurl                          7.81.0  hf45b732_0                   conda-forge/osx-64                         327kB
216c216
<     + python                           3.10.2  h1dd9edd_4_cpython           conda-forge/osx-64                          13MB
---
>     + python                           3.10.2  h1dd9edd_3_cpython           conda-forge/osx-64                          13MB
219c219
<     + python-stratify               0.2.post0  py310h7f5fb2b_1              conda-forge/osx-64                         161kB
---
>     + python-stratify               0.2.post0  py310h81f86ea_1              conda-forge/osx-64

It is possible the new Python build is causing this (I can't think of any other package from that list that may cause such mayhem). I'll try pin it see if we get anywhere, if not, am just gonna turn off the tests. @zklaus @bvreede you guys have any ideas?

[zklaus]

Not completely sure, but there was a hiccup with fontconfig, see conda-forge/fontconfig-feedstock#53. That would hit us via

flowchart LR
    fontconfig --> poppler[conda-forge/poppler-feedstock#124] --> gdal --> fiona
    fontconfig --> cairo[conda-forge/cairo-feedstock#62] --> poppler

Loading

If that's right, conda-forge/poppler-feedstock#124 should solve the issue. Let's wait and see.

[valeriupredoi]

brilliant, cheers, Klaus, gonna test that in Core! If it is that, we'll just sit tight and wait - a fonts package wreaking so much havoc though?

[valeriupredoi]

@zklaus you were right, it is blithering fontconfig - I ran a check with it pinned at the previous build https://github.com/ESMValGroup/ESMValCore/runs/5481211780?check_suite_focus=true

[zklaus]

Ok, so that seems to have done the trick. Mostly. One issue remains: on osx (and arm64), Python 3.7 is no longer supported by conda-forge (see here). So that version will not be repaired.

Should we:

• Drop 3.7 altogether
• Drop 3.7 for osx
• Attempt some pinning trickery for osx

@ESMValGroup/esmvaltool-coreteam @ESMValGroup/technical-lead-development-team ?

[schlunma]

I would go for option 1 or 2, with a slight preference on option 1 (drop 3.7 altogether). Do we need another release candidate for that?

We are running the recipe tests with python>3.7, so in principle this shouldn't affect our tests.

[zklaus]

My vote also goes to option 1. Another release candidate? Hm. Strictly speaking...

[schlunma]

If we do an rc6 then it would be great to drop 3.7 today and release rc6 today or tomorrow, so that @remi-kazeroni can do the test with that. There is no need to do the tests with rc5.

[zklaus]

@valeriupredoi, any opinion?

[valeriupredoi]

Hang on, on the bus to work, will reply in more detail, but for now 1 and/or 2 should do the trick 🍺 Dr Valeriu Predoi. Computational scientist NCAS-CMS University of Reading Department of Meteorology Reading RG6 6BB United Kingdom

…

On Thu, 10 Mar 2022, 10:09 Klaus Zimmermann, ***@***.***> wrote: @valeriupredoi <https://github.com/valeriupredoi>, any opinion? — Reply to this email directly, view it on GitHub <#2584 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG5EFI3DXHR6RR5L42TJMYTU7HC5XANCNFSM5QJPYL7Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[remi-kazeroni]

If we do an rc6 then it would be great to drop 3.7 today and release rc6 today or tomorrow, so that @remi-kazeroni can do the test with that. There is no need to do the tests with rc5.

I leave it up to you. But given the time constraints with Mistral replacement, I think we can only afford one last round of testing. I can use rc5 or rc6, as you tell me.