Ensure correct Manifest.toml is used in tests

[JoshuaLampert]

I noticed there is still another problem with merging the main and test project. Looking, e.g., at this CI log, we can see that when running the tests not the same downgraded package versions are used than after doing the downgrade (e.g. Aqua.jl is downgraded correctly to v0.8.0, but tests report to use v0.8.14). The action creates the correct Manifest.toml, but it is not correctly picked up by julia-runtests. I also tried copying the Manifest.toml first to test/, but this doesn't change anything. Then I tried to copy the Manifest.toml to test/ and not using julia-runtest, but rather manually executing the tests by using this code in the action yaml file:

      - uses: julia-actions/julia-downgrade-compat@v2
        with:
          skip: LinearAlgebra,SparseArrays,Statistics,Test
          projects: ., test
          mode: forcedeps
      - name: Lock test manifest to downgraded one
        run: cp Manifest.toml test/Manifest.toml
      - name: Run tests with locked versions
        run: |
          julia --project=test --color=yes -e '
            import Pkg
            Pkg.develop(Pkg.PackageSpec(path=pwd()))
            Pkg.instantiate()
            Pkg.status()
            include("test/runtests.jl")
          '

This looks good and the correct versions are used, see here. However, I think this solution is not very elegant and I would prefer to rely on julia-actions/julia-buildpkg and julia-actions/julia-runtest instead of manually running tests. So I asked AI to help and this PR is what it came up with. I'm not sure if this works, but I will try to set up a test running the downgrade action against this PR to check. I put this into draft mode until I figured out if this fixes the issue.

Update: Seems to work fine, but only on julia v1.12 because it uses Pkg.Types.workspace_resolve_hash, which was introduced in v1.12.

Update 2: Using project_resolve_hash to try to make it work also on julia v1.10 does not seem to work, see here.

[Krastanov]

Thanks for looking into this! I currently am experiencing CI issues that seem related to what you are fixing. My Downgrade CI fails in the presence of [workspace] directives in Project.toml, so I am setting up this very ugly workaround https://github.com/QuantumSavory/QuantumSavory.jl/pull/339/changes#diff-36163f22e54cd640fae014b8c015b1b5ea47a9991866121692386946a449bf4aR25

Commenting mostly to express gratitude for this work (while getting subscribed to notifications on this)

[JoshuaLampert]

It may well be that [workspaces] are not supported yet by this action. We also have issues with [sources], see #38. I neither use sources nor workspaces in my own projects. So I never really tested them. However, looking at your downgrade.yaml, you can improve it by adding projects: .,test to ensure the main project and test project are merged and downgraded correctly. You should also add (edit: to julia-runtest)

      with:
          allow_reresolve: false
          force_latest_compatible_version: false

to make sure Pkg.jl does not reresolve and ruin all your downgrades versions again. Probably this does not fix your issue, but maybe you are lucky.😅
If the issue persists, please open an issue about the workspaces issue.

[JoshuaLampert]

I asked on the Julia Slack for help regarding a solution on julia pre v1.12, but did not get a satisfying answer. Therefore, I suggest to document in the README.md suggesting my workaround running the tests manually without using julia-runtest for julia versions before v1.12 and merge this PR anyway because it already improves the current situation by making it properly work on julia v1.12+.

[JoshuaLampert]

I added some documentation about this issue in the README. So this PR is ready from my point of view, @ranocha, @ChrisRackauckas.

[ChrisRackauckas]

@ParamThakkar123 this is likely what you needed.

[ParamThakkar123]

Yeah. Thanks @ChrisRackauckas , Will try this out 🫡

[JoshuaLampert]

Can we get a new version? 🙂

[ParamThakkar123]

+1 🚀

[ChrisRackauckas]

yup released.