Improve CUTEst benchmarks #1323

ChrisRackauckas · 2025-08-06T09:31:49Z

This is @arnavk23's commits rebased to latest master and setup as a PR to master.

…ks are run" This reverts commit 5229720. Since the sandbox is run as a user and not root, packages cannot be installed due to insufficient permissions.

…ndling - Add chunked processing (50 problems per chunk) to manage memory usage - Implement comprehensive error handling with try/catch blocks - Add time limits (300s per problem) to prevent hanging - Force garbage collection between chunks to reduce memory pressure - Add detailed progress logging with chunk and problem tracking - Handle both problem loading and solving failures gracefully - Apply improvements to all CUTEst benchmark files: * CUTEst_bounded.jmd (666 + 244 problems) * CUTEst_unbounded.jmd (285 + 114 problems) * CUTEst_quadratic.jmd (252 problems) * CUTEst_unconstrained.jmd (286 problems) This resolves CI memory issues (ProcessSignaled(9)) while maintaining comprehensive testing of all CUTEst problem sets.

- Reduce chunk size from 5 to 3 problems per chunk - Lower variable limit from 100 to 50 variables per problem - Reduce maxiters from 1e6 to 1000 iterations - Keep maxtime at 60 seconds per problem - Add aggressive problem size filtering These changes should prevent ProcessSignaled(9) OOM errors in CI while still testing a substantial number of CUTEst problems.

thazhemadam · 2025-08-13T12:25:20Z

Yes. I've tried updating the rootfs image and it seems to no longer complain about gfortran being missing.¹
I just need to clean it up a bit and will push to this branch.

https://buildkite.com/julialang/scimlbenchmarks-dot-jl/builds/3668#01989c83-a549-49b7-a131-776cf2e975ad ↩

benchmarks/OptimizationCUTEst/CUTEst_bounded.jmd

benchmarks/OptimizationCUTEst/CUTEst_quadratic.jmd

benchmarks/OptimizationCUTEst/CUTEst_safe_solvers.jmd

benchmarks/OptimizationCUTEst/CUTEst_unbounded.jmd

benchmarks/OptimizationCUTEst/CUTEst_unconstrained.jmd

ChrisRackauckas · 2025-08-14T14:34:45Z

@arnavk23 it looks like it gets stuck. How long did that test set take locally? 2 days?

arnavk23 · 2025-08-14T14:36:03Z

No, it was done in around 3 hours.

ChrisRackauckas · 2025-08-14T15:00:57Z

Did you run this same problem? All 6 succeeded? I don't see how because it had forward diff before and that is guaranteed to fail over the binaries

arnavk23 · 2025-08-15T09:06:02Z

build_benchmark.sh -

# Check for CUTEst benchmarks and verify gfortran
if [[ "${1}" == *OptimizationCUTEst* ]]; then
	echo "--- :hammer: Setting up CUTEst environment"
	echo "Checking gfortran availability..."
	which gfortran || echo "gfortran not found in PATH"
	gfortran --version || echo "gfortran not working"
	export FC=gfortran
	export F77=gfortran
	export F90=gfortran
	echo "Fortran compiler environment variables set"
fi

ChrisRackauckas · 2025-08-15T09:37:23Z

Okay it has completely stalled with


[ Info: Problem 3/17: HS35
--
  | ┌ Warning: common maxiters argument is currently not used by Feasibility
  | │
  | │ Subject to:
  | │
  | │ Nonlinear
  | │ . Set number of iterations via optimizer specific keyword arguments.
  | └ @ OptimizationMOI /cache/julia-buildkite-plugin/depots/5b300254-1738-4989-ae0a-f4d2d937f953/packages/OptimizationMOI/aKPgG/src/OptimizationMOI.jl:85
  | [ Info: ✓ Solved HS35 with Ipopt - Status: Success
  | [ Info: Completed chunk, memory usage cleaned up
  | [ Info: Processing chunk 2/6: problems 4-6
  | [ Info: Problem 4/17: HS106
  | ┌ Warning: common maxiters argument is currently not used by Feasibility
  | │
  | │ Subject to:
  | │
  | │ Nonlinear
  | │ . Set number of iterations via optimizer specific keyword arguments.
  | └ @ OptimizationMOI /cache/julia-buildkite-plugin/depots/5b300254-1738-4989-ae0a-f4d2d937f953/packages/OptimizationMOI/aKPgG/src/OptimizationMOI.jl:85

and now it's 3 days at that same spot. I think it's safe to say it's not printing anything else. Here's what we can do. We can merge this as it's now at least a major step forward in that it is actually able to run again, but it's still the same problem that it originally had so it's not complete. At least this makes it easier to work on though.

ChrisRackauckas · 2025-08-15T09:38:13Z

@arnavk23 and you get 6 PDFs from that?

arnavk23 · 2025-08-15T09:50:06Z

@ChrisRackauckas I was able to get with the original AutoForwardDiff code only.

arnavk23 · 2025-08-15T09:51:18Z

@ChrisRackauckas I am still trying to see why this is stalling.

ChrisRackauckas · 2025-08-15T09:53:17Z

@ChrisRackauckas I was able to get with the original AutoForwardDiff code only.

I don't see how. It's not possible for forwarddiff the Fortran binaries, so you couldn't've been differentiating these examples.

arnavk23 · 2025-08-15T10:12:34Z

@ChrisRackauckas I saw you are doing this on Julia 10.10, try it instead on Julia 10.9

ChrisRackauckas · 2025-08-15T10:27:27Z

SciML benchmarks only runs on LTS forward. Current LTS is v10.10. But that also won't change this, fundamentally it's impossible for ForwardDiff to differentiate a fortran binary, it's not even possible in theory.

arnavk23 · 2025-08-15T10:36:31Z

@ChrisRackauckas , I meant the current one with FiniteDiff. Was looking through the code.

ChrisRackauckas · 2025-08-15T11:11:53Z

Can you show the PDFs built with it?

arnavk23 · 2025-08-15T13:17:54Z

@ChrisRackauckas It is working, taking some time. Got unconstrained. Further it is your decision as to if I continue on this or not.
CUTEst_unconstrained.pdf

ChrisRackauckas · 2025-08-15T13:23:08Z

That one ran before, but the issue there is most of them give failures.

alonsoC1s and others added 30 commits August 6, 2025 05:20

Setting up CUTEst benchmarks dir structure

785a9d7

Unconstrained problem benchmarks

313ac1c

Fixing problem initialization

2c7dc3a

Starting the analysis

c4a198a

Added preliminary benchmarks for eq/ineq problems with free vars

810e402

ci: install gfortran when the CUTEst optimization benchmarks are run

c2babbd

fixed variable use in benchmarks

64b0e85

Removing Manifest to fix version errors

bd4c96c

Using NLPModels from General Registry

b0f1c35

revert: "ci: install gfortran when the CUTEst optimization benchmar…

b3f7ae1

…ks are run" This reverts commit 5229720. Since the sandbox is run as a user and not root, packages cannot be installed due to insufficient permissions.

ci: use rootfs image with openmodelica and gfortran pre-installed

82df232

Add SciMLBenchmarks to the manifest and the footer

3b2acff

split into benchmark groups

d344044

Update CUTEst_bounded.jmd

ca5ef66

Use OptimizationNLPModels from branch

b91dc7b

stats pkg

2d27f55

use julia 1.10 resolved manifest

0333356

omjulia

b417914

Update Manifest.toml

292f48f

manifest

7631605

formatted using JuliaFormatter

1e9a8a6

removing deprecated CUTEst.select

0554ab0

Update Project.toml

465f149

safe_solvers

a2cd0e4

Update update.jl

c06bb8e

Update make.jl

b8bc556

Update pages.jl

39f6a7d

Update SciMLBenchmarks.jl

1aeac3d

ci: use new test rootfs image

14e37f2

thazhemadam force-pushed the cutest_local branch from c808547 to ead6b8a Compare August 13, 2025 12:50

ci: sign treehashes

4b2d4d9

thazhemadam force-pushed the cutest_local branch from ead6b8a to 4b2d4d9 Compare August 13, 2025 12:51