buildOctavePackage: add passthru tests

[pbsds]

While revieweing a octavePackages PR, an ecosystem I'm not familiar with, I was puzzled to find there was no real install test.
While buildOctavePackage does run octave-cli --eval "pkg build ...", it isn't before the user calls octave.withPackages that octave-cli --eval "pkg install ..." is run.
This PR adds a sniff test that does just that to all octave packages. This test si most often only needed when bumping and adding octave packages, although it would be nice to somehow do this during installCheckPhase instead.

While the self approach here works, this builder ought to be refactored to use lib.extendMkDerivation instead, but I leave that for the octave maintainers.

Testing:

Builds fine, albeit with some warnings here and there:

• octavePackages.audio.tests -> /nix/store/gfsfcffvg139g1mxvb0klwlyx2gdjix7-octave-10.1.0-env
• octavePackages.bim.tests -> /nix/store/5qnxx078760j600wj5j934pg1qi6vnjp-octave-10.1.0-env
• octavePackages.bsltl.tests -> /nix/store/di46mpwws75v9limprr669ap1kdcqg6p-octave-10.1.0-env
• octavePackages.cgi.tests -> /nix/store/ydmv60i8sw1ka6blgbb7i0b00jpa06ba-octave-10.1.0-env
• octavePackages.communications.tests -> /nix/store/fgyvys3llwga8dffkbmvvfd4j2bmgir7-octave-10.1.0-env
• octavePackages.control.tests -> /nix/store/jmsx2h17py4if2sxks90kzxk8yva179y-octave-10.1.0-env
• octavePackages.database.tests -> /nix/store/n7750sl8ycq80jddjsk7jfhrynshk7hh-octave-10.1.0-env
• octavePackages.dataframe.tests -> /nix/store/n47kcw9akf3g1rrvkg43qvhdspypgqxz-octave-10.1.0-env
• octavePackages.dicom.tests -> /nix/store/mgkbc1bgn54y5lzv152x0zrkmv5g8n3g-octave-10.1.0-env
• octavePackages.divand.tests -> /nix/store/dxpmqirr60qcj810n89lq4q6fylwbrvz-octave-10.1.0-env
• octavePackages.doctest.tests -> /nix/store/wh5nhndxlfg8ahf2dhsi4k7s4q24kn4x-octave-10.1.0-env
• octavePackages.fpl.tests -> /nix/store/3mx9s9dbvcx20m0m4hm8mgfrvl2f8c4j-octave-10.1.0-env
• octavePackages.fuzzy-logic-toolkit.tests -> /nix/store/ysx68g7wpy4fbz21q9x9zdgb4yj9li13-octave-10.1.0-env
• octavePackages.ga.tests -> /nix/store/9p6mgsv3166il24s4bfpxbaqy7pr3np3-octave-10.1.0-env
• octavePackages.general.tests -> /nix/store/bhjm2ar1n379c2dg4k7wmby721qi3w9b-octave-10.1.0-env
• octavePackages.generate_html.tests -> /nix/store/r33k2qna874ncyf0rzfvmxwbkl3sal4k-octave-10.1.0-env
• octavePackages.geometry.tests -> /nix/store/3pxrh2wb9wlv69kvrrg6jjpgwxy2rjqr-octave-10.1.0-env
• octavePackages.image-acquisition.tests -> /nix/store/r0ywp3kq967ay64y6hhmpxkqjy8n69q9-octave-10.1.0-env
• octavePackages.image.tests -> /nix/store/pssi79pdvp0k14z1lbvsnlh46ygq78ya-octave-10.1.0-env
• octavePackages.instrument-control.tests -> /nix/store/61m4n8ii3qsnbgbd0zzc8x0bk72cx5vv-octave-10.1.0-env
• octavePackages.interval.tests -> /nix/store/g78z6ah6xxjrnvz2fafn5nx9fm2pgnyv-octave-10.1.0-env
• octavePackages.io.tests -> /nix/store/dw8lav38xn5wp4h88q926b3krxvfczdn-octave-10.1.0-env
• octavePackages.linear-algebra.tests -> /nix/store/wriqa38xf9vfnrhhv6slfpc6k5arv5cm-octave-10.1.0-env
• octavePackages.lssa.tests -> /nix/store/wwhj069fgnazbmwyakqkrhbrfdkfvwhc-octave-10.1.0-env
• octavePackages.matgeom.tests -> /nix/store/di1dp02ds6lvlm8ifwmsmxafki0my5j3-octave-10.1.0-env
• octavePackages.miscellaneous.tests -> /nix/store/pbh2khwbs3asxr98ww3dadynyp4w5gjl-octave-10.1.0-env
• octavePackages.msh.tests -> /nix/store/g609r2iy0skgcv2kginkxqpp6kcvk8nb-octave-10.1.0-env
• octavePackages.mvn.tests -> /nix/store/y84dhsf3h4kz0vs729lvncjbs75rlls3-octave-10.1.0-env
• octavePackages.nan.tests -> /nix/store/gm7ma33w5v67379wjvghx6lmpk39r4a4-octave-10.1.0-env
• octavePackages.ncarray.tests -> /nix/store/xdqn7cd3a7q0aj0f42l2ifskhn1k9c4m-octave-10.1.0-env
• octavePackages.netcdf.tests -> /nix/store/4rzqbs8i1hkhsgl9q67laf5srz25gwh5-octave-10.1.0-env
• octavePackages.nurbs.tests -> /nix/store/gnsx5i5c6ky8y5s5j8izlgjz74prp1ys-octave-10.1.0-env
• octavePackages.octclip.tests -> /nix/store/bd723m3w1ffig5a1cnvaf1wkhnc66hfv-octave-10.1.0-env
• octavePackages.octproj.tests -> /nix/store/pi19581cqja36m3zlh09l6mpk4yynq34-octave-10.1.0-env
• octavePackages.optics.tests -> /nix/store/ig9f7adp173c2qngkclp01iv388ivxsk-octave-10.1.0-env
• octavePackages.optiminterp.tests -> /nix/store/249dq2sadln4bgknylaxzc13yg4lcbbp-octave-10.1.0-env
• octavePackages.quaternion.tests -> /nix/store/amw45jgwqqf7bw55qwc8bsjzx9khlbyj-octave-10.1.0-env
• octavePackages.signal.tests -> /nix/store/q8banm7pkgzka9ygpzqivn44gn88iqpd-octave-10.1.0-env
• octavePackages.sockets.tests -> /nix/store/74xb0m3ykpxny8grbz2a0c84bp5x8qk2-octave-10.1.0-env
• octavePackages.splines.tests -> /nix/store/1yk0p5yxl46b2bl1mlmiw32s2d0a5vxm-octave-10.1.0-env
• octavePackages.statistics.tests -> /nix/store/r161lfb3cg4ajk3ky06gnlr94y1g7n1l-octave-10.1.0-env
• octavePackages.stk.tests -> /nix/store/djf5x4jsi0yx2jsw3jrs3cnzn8mc7v44-octave-10.1.0-env
• octavePackages.strings.tests -> /nix/store/f8hhf2p6ykrchjnc8nngsbbscagkhiyl-octave-10.1.0-env
• octavePackages.struct.tests -> /nix/store/y8svqkjsfxv75s7nxim9wk1qba72xrvm-octave-10.1.0-env
• octavePackages.symbolic.tests -> /nix/store/f3r2brimsfymqk28fl44cf9ddsw34w3x-octave-10.1.0-env
• octavePackages.tsa.tests -> /nix/store/rvga4qc13dcrf913mjrmq4nr4gb6rk54-octave-10.1.0-env
• octavePackages.video.tests -> /nix/store/h6m2d90qjviq3qv89mnrysbasmg55zbi-octave-10.1.0-env
• octavePackages.windows.tests -> /nix/store/qx79vshxqm1hr5d7chqzzc751iqcq75x-octave-10.1.0-env
• octavePackages.zeromq.tests -> /nix/store/ws42x3gpqvfrqm7bvbffcphf0snb994i-octave-10.1.0-env

Build failure:

• octavePackages.arduino.tests
• octavePackages.financial.tests
• octavePackages.gsl.tests
• octavePackages.mapping.tests
  • fixed by octavePackages.mapping: 1.4.2 -> 1.4.3 #406300
• octavePackages.queueing.tests

No eval (the majority because they're marked broken):

• octavePackages.data-smoothing.tests
• octavePackages.econometrics.tests
• octavePackages.fem-fenics.tests
• octavePackages.fits.tests
• octavePackages.level-set.tests
• octavePackages.ltfat.tests
• octavePackages.ocl.tests
• octavePackages.optim.tests
• octavePackages.parallel.tests
• octavePackages.sparsersb.tests
• octavePackages.tisean.tests
• octavePackages.vibes.tests
• octavePackages.vrml.tests
• octavePackages.writeRequiredOctavePackagesHook.tests

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• Nixpkgs 25.11 Release Notes (or backporting 24.11 and 25.05 Nixpkgs Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
• NixOS 25.11 Release Notes (or backporting 24.11 and 25.05 NixOS Release notes)
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[KarlJoad]

@pbsds, this is a good addition, but won't really catch much. From my experience, Octave packages usually fail either when they are compiled (if they are compiled) and repackaged into the octavePackage's tarball output or during use. This install test is a good first smoke test, but will not capture breaking issues in the final environment itself. I think we should add another passthru test that runs the package's unit tests.

We can only run the unit test in an Octave environment (octave.withPackages) because the tests must be run through Octave after the package is loaded.

During my maintenance of the octavePackages set, I have put together a little Bash script to pull a r-ryantm auto-update and run the package's unit test, which builds the updated package (buildOctavePackage), makes an Octave environment (octave.withPackages), and runs the package's unit tests, putting the results in a log file for inspection.

nix-shell -I "nixpkgs=$(pwd)" -p "octave.withPackages (os: with os; [ ${pkg} ])" \
          --command "octave-cli --eval 'pkg test ${pkg}'" > "./test-logs/${pkg}-test.log"

The --command portion starts Octave, loads the requested package, then runs its unit tests.

Do you think you could add the above --command stuff to your PR too?

[pbsds]

I could use some help, since i don't know any matlab nor octave.

• octavePackages.audio.tests.testOctavePkgTests -> /nix/store/x461wg9crrk3njsfx5al0bswv5sz3paj-octave-10.1.0-audio-2.0.5-pkg-test
• octavePackages.bim.tests.testOctavePkgTests -> /nix/store/4i5balxh4zy3gv7vg3812s773igzj990-octave-10.1.0-bim-1.1.6-pkg-test
• octavePackages.bsltl.tests.testOctavePkgTests -> /nix/store/3ijlicw576cy53sk886cdwfi6n6la31h-octave-10.1.0-bsltl-1.3.1-pkg-test
• octavePackages.cgi.tests.testOctavePkgTests -> /nix/store/ssmnxbhqpr0gybb090k6m6szjxk2zgk3-octave-10.1.0-cgi-0.1.2-pkg-test
• octavePackages.communications.tests.testOctavePkgTests -> /nix/store/fay5r0kylj9mbc5n2dzk4zg7bl0zr6r2-octave-10.1.0-communications-1.2.7-pkg-test
• octavePackages.control.tests.testOctavePkgTests -> /nix/store/2l1bhm9brs519q40svp8k917vspm7rcf-octave-10.1.0-control-3.6.1-pkg-test
• octavePackages.database.tests.testOctavePkgTests -> /nix/store/ch3f4vxfb1vf7553s4rmdl3hzrv3xa0a-octave-10.1.0-database-2.4.4-pkg-test
• octavePackages.dataframe.tests.testOctavePkgTests -> /nix/store/bfdzqigwn1vp1w5wxrgsqp5qpdid9sxi-octave-10.1.0-dataframe-1.2.0-pkg-test
• octavePackages.dicom.tests.testOctavePkgTests -> /nix/store/22hj1bddivswz89gg3jws6alg6ka5wsl-octave-10.1.0-dicom-0.6.1-pkg-test
• octavePackages.divand.tests.testOctavePkgTests -> /nix/store/f1ms1n0jd0qnhw5ks35vl5m9ri20sm86-octave-10.1.0-divand-1.1.2-pkg-test
• octavePackages.doctest.tests.testOctavePkgTests -> /nix/store/072h66q91m5n5c6qd423jf8zchwpr9b7-octave-10.1.0-doctest-0.8.0-pkg-test
• octavePackages.fpl.tests.testOctavePkgTests -> /nix/store/1aijwg5vzdirbk0b348xzwip9za7mqxv-octave-10.1.0-fpl-1.3.5-pkg-test
• octavePackages.fuzzy-logic-toolkit.tests.testOctavePkgTests -> /nix/store/ffb4yzysdvvzyyri6fk3z5wycbryc28l-octave-10.1.0-fuzzy-logic-toolkit-0.6.1-pkg-test
• octavePackages.ga.tests.testOctavePkgTests -> /nix/store/zkvhlimp0bkwm776b8kb3s9pnglnma38-octave-10.1.0-ga-0.10.4-pkg-test
• octavePackages.general.tests.testOctavePkgTests -> /nix/store/vjw5dm3bah5pwdapbqq56yij3wgqzcbs-octave-10.1.0-general-2.1.3-pkg-test
• octavePackages.generate_html.tests.testOctavePkgTests -> /nix/store/27byq3j0i6ay9866rj99dphlqrqxsy78-octave-10.1.0-generate_html-0.3.3-pkg-test
• octavePackages.geometry.tests.testOctavePkgTests -> /nix/store/qm1rmk9gi6xd2ki6sirxrq9c49s4krzj-octave-10.1.0-geometry-unstable-2021-07-07-pkg-test
• octavePackages.image-acquisition.tests.testOctavePkgTests -> /nix/store/qb5gi411r14f0qba0m4zrgiaa98rbmj5-octave-10.1.0-image-acquisition-0.3.0-pkg-test
• octavePackages.image.tests.testOctavePkgTests -> /nix/store/8xk6cv5rrp9my2zakv3lkzyq5d2l65pn-octave-10.1.0-image-2.16.1-pkg-test
• octavePackages.instrument-control.tests.testOctavePkgTests -> /nix/store/n2bmlgq07i0bix1isnd9cfbr2jg3bz5h-octave-10.1.0-instrument-control-0.9.5-pkg-test
• octavePackages.interval.tests.testOctavePkgTests -> /nix/store/6gqvqfhldf6n7pzadbh0g4sxfrxhv295-octave-10.1.0-interval-3.2.1-pkg-test
• octavePackages.io.tests.testOctavePkgTests -> /nix/store/k52dw1g7h2n2hc7dzppv418av2xcw6ww-octave-10.1.0-io-2.7.0-pkg-test
• octavePackages.linear-algebra.tests.testOctavePkgTests -> /nix/store/kai1h3pg5gj09ig58zrkjx43iia6b4sy-octave-10.1.0-linear-algebra-2.2.3-pkg-test
• octavePackages.lssa.tests.testOctavePkgTests -> /nix/store/n3yjgw9i164kgg9gb404849asycr2hip-octave-10.1.0-lssa-0.1.4-pkg-test
• octavePackages.matgeom.tests.testOctavePkgTests -> /nix/store/qbzsdwx7grwwddn51vnd77x2qfifkiz2-octave-10.1.0-matgeom-1.2.4-pkg-test
• octavePackages.miscellaneous.tests.testOctavePkgTests -> /nix/store/qn2v46zkvx00b9qpaxsikn1ad8qiq0qi-octave-10.1.0-miscellaneous-1.3.1-pkg-test
• octavePackages.msh.tests.testOctavePkgTests -> /nix/store/zhfjlfmmxwwgpq43dad0f4lnn1vzpwqk-octave-10.1.0-msh-1.0.12-pkg-test
• octavePackages.mvn.tests.testOctavePkgTests -> /nix/store/hz9cb4fkm54d9ynqagph75kwjrz85ghg-octave-10.1.0-mvn-1.1.0-pkg-test
• octavePackages.nan.tests.testOctavePkgTests -> /nix/store/hfznv5cpjs1mhs29mnmw6vcff73542s5-octave-10.1.0-nan-3.7.0-pkg-test
• octavePackages.ncarray.tests.testOctavePkgTests -> /nix/store/lch0msk36miayprzkpqi379q3qrxga0k-octave-10.1.0-ncarray-1.0.6-pkg-test
• octavePackages.netcdf.tests.testOctavePkgTests -> /nix/store/h0dzmzr2nibsgl4lyxdb1ji809d1mv9x-octave-10.1.0-netcdf-1.0.18-pkg-test
• octavePackages.nurbs.tests.testOctavePkgTests -> /nix/store/crx2lhkb936ycyqji3dvm19m73za6ld4-octave-10.1.0-nurbs-1.4.4-pkg-test
• octavePackages.octclip.tests.testOctavePkgTests -> /nix/store/2lxrmz5ps73ilgblchhblwpkf2y20sfq-octave-10.1.0-octclip-2.0.3-pkg-test
• octavePackages.octproj.tests.testOctavePkgTests -> /nix/store/bhihjyadc5qjcwfq4c9zr18afa6sb8vj-octave-10.1.0-octproj-3.0.2-pkg-test
• octavePackages.optics.tests.testOctavePkgTests -> /nix/store/rpvywxbj6y6alxb6b82dksxfiwrn095p-octave-10.1.0-optics-0.1.4-pkg-test
• octavePackages.optiminterp.tests.testOctavePkgTests -> /nix/store/y7pfb6yabih8sz1clad2ji8h8zgam2hi-octave-10.1.0-optiminterp-0.3.7-pkg-test
• octavePackages.quaternion.tests.testOctavePkgTests -> /nix/store/crlidgqqib71j6q7sc3biz48gcrgcfr3-octave-10.1.0-quaternion-2.4.0-pkg-test
• octavePackages.signal.tests.testOctavePkgTests -> /nix/store/4vwyz7wjgcnpcsav93k1a9cv0hb2ynqk-octave-10.1.0-signal-1.4.6-pkg-test
• octavePackages.sockets.tests.testOctavePkgTests -> /nix/store/di8j18q1brhc0qwdkncqlx3zaw03115x-octave-10.1.0-sockets-1.4.1-pkg-test
• octavePackages.splines.tests.testOctavePkgTests -> /nix/store/8xwi4z0a483ksj8rq4q1dmdfbamzjbdf-octave-10.1.0-splines-1.3.5-pkg-test
• octavePackages.statistics.tests.testOctavePkgTests -> /nix/store/i8hyn03bk3pnmgcn8kib5q8nlj0bklc1-octave-10.1.0-statistics-1.7.0-pkg-test
• octavePackages.stk.tests.testOctavePkgTests -> /nix/store/24azbi7cfbv4dvddpxr7y72d35542vzs-octave-10.1.0-stk-2.8.1-pkg-test
• octavePackages.strings.tests.testOctavePkgTests -> /nix/store/ircxy6z04im43fn5dbyxyafzpq1wrisq-octave-10.1.0-strings-1.3.1-pkg-test
• octavePackages.struct.tests.testOctavePkgTests -> /nix/store/pnsykqvfabz3h6d6kkx628fsdl7z4f2v-octave-10.1.0-struct-1.0.18-pkg-test
• octavePackages.symbolic.tests.testOctavePkgTests -> /nix/store/sgffdn5f9xh2kpb81daf20jgslc9c8lh-octave-10.1.0-symbolic-3.2.1-pkg-test
• octavePackages.tsa.tests.testOctavePkgTests -> /nix/store/zza9bjr3lswi6frq5vj9ahwxr03fgswq-octave-10.1.0-tsa-4.6.3-pkg-test
• octavePackages.video.tests.testOctavePkgTests -> /nix/store/ica9nn6xvj4j2xr1jv2j111qy5v8267d-octave-10.1.0-video-2.1.3-pkg-test
• octavePackages.windows.tests.testOctavePkgTests -> /nix/store/247c9ljyyqx47l4klh1sl2pxl61c4yax-octave-10.1.0-windows-1.6.5-pkg-test
• octavePackages.zeromq.tests.testOctavePkgTests -> /nix/store/5vnc5jznida4cnq9d527v36y5n0gddi0-octave-10.1.0-zeromq-1.5.5-pkg-test

All tests "pass", but the problem is that the exit code does not reflect wether the test passed or not.

Failure Summary:
  ..0-env/share/octave/octave_packages/audio-2.0.5/midicallback.m  pass    1/2
                                                                   FAIL    1
  ..0-env/share/octave/octave_packages/audio-2.0.5/midicontrols.m  pass    0/1
                                                                   FAIL    1
  ..1.0-env/share/octave/octave_packages/audio-2.0.5/mididevice.m  pass    0/1
                                                                   FAIL    1
  ...0-env/share/octave/octave_packages/audio-2.0.5/mididevinfo.m  pass    0/1
                                                                   FAIL    1
  ...1.0-env/share/octave/octave_packages/audio-2.0.5/midiflush.m  pass    1/2
                                                                   FAIL    1
Summary:

  PASS                               62
  FAIL                                5

Is there any flag i can add or need i grep for something like " FAIL *0$"?

[KarlJoad]

I don't use Matlab or Octave enough to know the exact effects anymore. I have not done any signal processing work in quite some time, so I haven't touched these tools for nearly 3 years.

Honestly, I have no idea which would be better. The documentation for Octave's pkg command is pretty sparse, particularly for the test subcommand. Also, I cannot find a package that is not marked as broken that has failing tests (mostly because I'm too lazy to go find one) and check what the $? of the octave-cli invocation is.

My recommendation is to use grep on a FAIL line. Also, don't just grep for the final fail, but also make sure we match against the individual file failures. That way we get a non-zero exit code when we match and we can see exactly which files failed their tests.

[pbsds]

Details

test successes:

• octavePackages.bim.tests.testOctavePkgTests -> /nix/store/qdpqsfclmma3ka9nf591nr9dg5q888ss-octave-10.1.0-bim-1.1.6-pkg-test
• octavePackages.bsltl.tests.testOctavePkgTests -> /nix/store/m5hz4l40f7lg2r4plkgbp1c02minlrg7-octave-10.1.0-bsltl-1.3.1-pkg-test
• octavePackages.cgi.tests.testOctavePkgTests -> /nix/store/v6pmsjzivlvl92ny9p96ni5pdlrwpc77-octave-10.1.0-cgi-0.1.2-pkg-test
• octavePackages.communications.tests.testOctavePkgTests -> /nix/store/63734fkcyhf8ijx783gx3m3sxjfx1dpr-octave-10.1.0-communications-1.2.7-pkg-test
• octavePackages.dataframe.tests.testOctavePkgTests -> /nix/store/vam6zgy00j1yk800gpw06sm8zfahyfaq-octave-10.1.0-dataframe-1.2.0-pkg-test
• octavePackages.dicom.tests.testOctavePkgTests -> /nix/store/kzl5g367b1xx4pl65l7m5d9qpj2hcl5h-octave-10.1.0-dicom-0.6.1-pkg-test
• octavePackages.divand.tests.testOctavePkgTests -> /nix/store/nf298kb9s5a7qqpfgkmd478jhysnb95n-octave-10.1.0-divand-1.1.2-pkg-test
• octavePackages.doctest.tests.testOctavePkgTests -> /nix/store/65lxg752j8gw61fs0h4vf92mbdb4p1q6-octave-10.1.0-doctest-0.8.0-pkg-test
• octavePackages.fpl.tests.testOctavePkgTests -> /nix/store/1jr4blqqgnn2imw8b39jnszpr211n4dg-octave-10.1.0-fpl-1.3.5-pkg-test
• octavePackages.fuzzy-logic-toolkit.tests.testOctavePkgTests -> /nix/store/8n975ikvq8rdxg3alym4mpjm8fqpkq1x-octave-10.1.0-fuzzy-logic-toolkit-0.6.1-pkg-test
• octavePackages.ga.tests.testOctavePkgTests -> /nix/store/nhjvyd3ggg54y0i1s29hx60kqkwr0v40-octave-10.1.0-ga-0.10.4-pkg-test
• octavePackages.general.tests.testOctavePkgTests -> /nix/store/5705iw3r3mplxqibldqcrnkr8c4b339x-octave-10.1.0-general-2.1.3-pkg-test
• octavePackages.generate_html.tests.testOctavePkgTests -> /nix/store/kfb2qi8ywa3vjwr55w4d372pdn6kkrnm-octave-10.1.0-generate_html-0.3.3-pkg-test
• octavePackages.io.tests.testOctavePkgTests -> /nix/store/8wmyrgxiy2302liapcfsgf8pqqj327ic-octave-10.1.0-io-2.7.0-pkg-test
• octavePackages.linear-algebra.tests.testOctavePkgTests -> /nix/store/vxidxfz6x362jq21gd59cb0xdg50szzl-octave-10.1.0-linear-algebra-2.2.3-pkg-test
• octavePackages.matgeom.tests.testOctavePkgTests -> /nix/store/nfpw7shfdaf76hyzfhlvsynzs5g203wd-octave-10.1.0-matgeom-1.2.4-pkg-test
• octavePackages.miscellaneous.tests.testOctavePkgTests -> /nix/store/fxdhd7jcxqizxqmx04vdim5v78amx8vf-octave-10.1.0-miscellaneous-1.3.1-pkg-test
• octavePackages.mvn.tests.testOctavePkgTests -> /nix/store/rzx02snvs3jldb8qgf2bcvdazza5s7yn-octave-10.1.0-mvn-1.1.0-pkg-test
• octavePackages.netcdf.tests.testOctavePkgTests -> /nix/store/qhjbdlik7srbfi50sk674i9y07ai060z-octave-10.1.0-netcdf-1.0.18-pkg-test
• octavePackages.nurbs.tests.testOctavePkgTests -> /nix/store/s0rjdp9y6m0k7hxvq1k9j7gj192zbi1i-octave-10.1.0-nurbs-1.4.4-pkg-test
• octavePackages.octclip.tests.testOctavePkgTests -> /nix/store/rf1lpq1fwbc4hw4gm06g6z1b8s7d0x47-octave-10.1.0-octclip-2.0.3-pkg-test
• octavePackages.octproj.tests.testOctavePkgTests -> /nix/store/7wn6r4ddjx0f3qc0v53bbmvavlmcp47y-octave-10.1.0-octproj-3.0.2-pkg-test
• octavePackages.optics.tests.testOctavePkgTests -> /nix/store/wmxbw3wwq61k6b3r701xjl1sy4gn47qz-octave-10.1.0-optics-0.1.4-pkg-test
• octavePackages.optiminterp.tests.testOctavePkgTests -> /nix/store/kzs40clifx5swk4pf4sbqv7scb68dwlx-octave-10.1.0-optiminterp-0.3.7-pkg-test
• octavePackages.quaternion.tests.testOctavePkgTests -> /nix/store/8lvg1ymf0mc81f6knawxx307rif8fkwp-octave-10.1.0-quaternion-2.4.0-pkg-test
• octavePackages.sockets.tests.testOctavePkgTests -> /nix/store/kbb2lgq1d4qiqp4hqv8xr8h0dl6l28gd-octave-10.1.0-sockets-1.4.1-pkg-test
• octavePackages.splines.tests.testOctavePkgTests -> /nix/store/604x6qmrxl9vhy2m246xqfkrxi3clj5n-octave-10.1.0-splines-1.3.5-pkg-test
• octavePackages.strings.tests.testOctavePkgTests -> /nix/store/mlv9sd4hwhk1amfajzzzk2hwv8yi8wpk-octave-10.1.0-strings-1.3.1-pkg-test
• octavePackages.struct.tests.testOctavePkgTests -> /nix/store/sazg5aymf6zm0179xhs59001w9hi3kwd-octave-10.1.0-struct-1.0.18-pkg-test
• octavePackages.tsa.tests.testOctavePkgTests -> /nix/store/xcxzs2q1l28g522xjq2kgdi7phblh47v-octave-10.1.0-tsa-4.6.3-pkg-test
• octavePackages.windows.tests.testOctavePkgTests -> /nix/store/yi1q83p3sizz82q6q17aszkl0ly50ija-octave-10.1.0-windows-1.6.5-pkg-test
• octavePackages.zeromq.tests.testOctavePkgTests -> /nix/store/6pjizdxi0192dzjqfd194i3x5gvkmv8q-octave-10.1.0-zeromq-1.5.5-pkg-test

test failures:

• octavePackages.audio.tests.testOctavePkgTests
• octavePackages.control.tests.testOctavePkgTests
• octavePackages.database.tests.testOctavePkgTests
• octavePackages.geometry.tests.testOctavePkgTests
• octavePackages.image-acquisition.tests.testOctavePkgTests
• octavePackages.image.tests.testOctavePkgTests
• octavePackages.instrument-control.tests.testOctavePkgTests
• octavePackages.interval.tests.testOctavePkgTests
• octavePackages.lssa.tests.testOctavePkgTests
• octavePackages.msh.tests.testOctavePkgTests
• octavePackages.nan.tests.testOctavePkgTests
• octavePackages.ncarray.tests.testOctavePkgTests
• octavePackages.signal.tests.testOctavePkgTests
• octavePackages.statistics.tests.testOctavePkgTests
• octavePackages.stk.tests.testOctavePkgTests
• octavePackages.symbolic.tests.testOctavePkgTests
• octavePackages.video.tests.testOctavePkgTests

[pbsds]

I used tee and a >(...) subshell to not hide the logs

[pbsds]

I added a retcode check

Details

test successes:

• octavePackages.bim.tests.testOctavePkgTests -> /nix/store/rvm4zzmb41gfvda04fzacfbav1kkr7a0-octave-10.1.0-bim-1.1.6-pkg-test
• octavePackages.bsltl.tests.testOctavePkgTests -> /nix/store/khldlkk16g6f9iyxwpzi5mrlxfy66n25-octave-10.1.0-bsltl-1.3.1-pkg-test
• octavePackages.cgi.tests.testOctavePkgTests -> /nix/store/k742s20zh6zmhlm0jsn0af7wlij265iq-octave-10.1.0-cgi-0.1.2-pkg-test
• octavePackages.communications.tests.testOctavePkgTests -> /nix/store/38kpxq4p7bl619grvlmgl15h0d54l6vr-octave-10.1.0-communications-1.2.7-pkg-test
• octavePackages.dataframe.tests.testOctavePkgTests -> /nix/store/7svdgvbhd04dl2j89g18n8mi21bfhj41-octave-10.1.0-dataframe-1.2.0-pkg-test
• octavePackages.dicom.tests.testOctavePkgTests -> /nix/store/9g4wpr0j49hghjzn054lpl00zlg936nx-octave-10.1.0-dicom-0.6.1-pkg-test
• octavePackages.divand.tests.testOctavePkgTests -> /nix/store/rja901hkzlkyrqrvg7qbdf3d4jabj9my-octave-10.1.0-divand-1.1.2-pkg-test
• octavePackages.doctest.tests.testOctavePkgTests -> /nix/store/5dxnn1hrc184zlpvy00cs10szwlqmgpd-octave-10.1.0-doctest-0.8.0-pkg-test
• octavePackages.fpl.tests.testOctavePkgTests -> /nix/store/4ixfranhmz824rlzqa34l2snf6xqbnsl-octave-10.1.0-fpl-1.3.5-pkg-test
• octavePackages.fuzzy-logic-toolkit.tests.testOctavePkgTests -> /nix/store/d91hadx59bv0cw0a7z8zdx48inha8bji-octave-10.1.0-fuzzy-logic-toolkit-0.6.1-pkg-test
• octavePackages.ga.tests.testOctavePkgTests -> /nix/store/2jwhq7iv54aqzshg4m8xq3h8plxrihfn-octave-10.1.0-ga-0.10.4-pkg-test
• octavePackages.general.tests.testOctavePkgTests -> /nix/store/bjd00kl6m86fz9cp70ai6jni6jfijw2g-octave-10.1.0-general-2.1.3-pkg-test
• octavePackages.generate_html.tests.testOctavePkgTests -> /nix/store/29z3q919nzrkd2afzr7i9c325fk8h966-octave-10.1.0-generate_html-0.3.3-pkg-test
• octavePackages.io.tests.testOctavePkgTests -> /nix/store/qpj69a8swl3rzs039fkyf77krb3nbqf6-octave-10.1.0-io-2.7.0-pkg-test
• octavePackages.linear-algebra.tests.testOctavePkgTests -> /nix/store/x28qgln5fww84z8sjr5ynnh4h9yxhhz7-octave-10.1.0-linear-algebra-2.2.3-pkg-test
• octavePackages.matgeom.tests.testOctavePkgTests -> /nix/store/f7r4rvv6czh56phmf67ihn7ki7p0j7fa-octave-10.1.0-matgeom-1.2.4-pkg-test
• octavePackages.miscellaneous.tests.testOctavePkgTests -> /nix/store/1vp5vpi3fgbw5bwf409vsz1q9n7qvwv4-octave-10.1.0-miscellaneous-1.3.1-pkg-test
• octavePackages.mvn.tests.testOctavePkgTests -> /nix/store/882fwfw2yda00qv021ih45agx0iqimcm-octave-10.1.0-mvn-1.1.0-pkg-test
• octavePackages.netcdf.tests.testOctavePkgTests -> /nix/store/zdba74gxzh5xf4vnmhv36wjf81zy48jj-octave-10.1.0-netcdf-1.0.18-pkg-test
• octavePackages.nurbs.tests.testOctavePkgTests -> /nix/store/jcm76br2zv1f4pmx2jvv63mqwd6pk1wi-octave-10.1.0-nurbs-1.4.4-pkg-test
• octavePackages.octclip.tests.testOctavePkgTests -> /nix/store/6phn6qz154i3861gswx50bwaa76kf2nh-octave-10.1.0-octclip-2.0.3-pkg-test
• octavePackages.octproj.tests.testOctavePkgTests -> /nix/store/ylllvmg0via7i99rf121znwq4dr9k4b9-octave-10.1.0-octproj-3.0.2-pkg-test
• octavePackages.optics.tests.testOctavePkgTests -> /nix/store/407nl9b2j85z22p163xsiy94rd4sczrz-octave-10.1.0-optics-0.1.4-pkg-test
• octavePackages.optiminterp.tests.testOctavePkgTests -> /nix/store/a7yq31rp93a37fwz269f7qra3qc2kjr7-octave-10.1.0-optiminterp-0.3.7-pkg-test
• octavePackages.quaternion.tests.testOctavePkgTests -> /nix/store/fxb9gwljpqyidl1v21y5782fl9v08bbp-octave-10.1.0-quaternion-2.4.0-pkg-test
• octavePackages.sockets.tests.testOctavePkgTests -> /nix/store/jcgbc9g54s49d82xq19ifxjzx45f1pdx-octave-10.1.0-sockets-1.4.1-pkg-test
• octavePackages.splines.tests.testOctavePkgTests -> /nix/store/sydyfjl35z4snh45bjk79n5vxqps42iq-octave-10.1.0-splines-1.3.5-pkg-test
• octavePackages.strings.tests.testOctavePkgTests -> /nix/store/ksx4g1axic4lgqkk2551winz3sdk6bsy-octave-10.1.0-strings-1.3.1-pkg-test
• octavePackages.struct.tests.testOctavePkgTests -> /nix/store/ib8pmcjwyga08s3qq9mv04g5421796q7-octave-10.1.0-struct-1.0.18-pkg-test
• octavePackages.tsa.tests.testOctavePkgTests -> /nix/store/d2c2fjr9b5fxaaw571a299v5h2jxmj56-octave-10.1.0-tsa-4.6.3-pkg-test
• octavePackages.windows.tests.testOctavePkgTests -> /nix/store/dp5rwkr9li4w5346ibl3fyrs9p0l566j-octave-10.1.0-windows-1.6.5-pkg-test
• octavePackages.zeromq.tests.testOctavePkgTests -> /nix/store/39zr8n4ccyxi94hkpk95p1h13v9kw4n8-octave-10.1.0-zeromq-1.5.5-pkg-test

test failures:

• octavePackages.audio.tests.testOctavePkgTests
• octavePackages.control.tests.testOctavePkgTests
• octavePackages.database.tests.testOctavePkgTests
• octavePackages.geometry.tests.testOctavePkgTests
• octavePackages.image-acquisition.tests.testOctavePkgTests
• octavePackages.image.tests.testOctavePkgTests
• octavePackages.instrument-control.tests.testOctavePkgTests
• octavePackages.interval.tests.testOctavePkgTests
• octavePackages.lssa.tests.testOctavePkgTests
• octavePackages.msh.tests.testOctavePkgTests
• octavePackages.nan.tests.testOctavePkgTests
• octavePackages.ncarray.tests.testOctavePkgTests
• octavePackages.signal.tests.testOctavePkgTests
• octavePackages.statistics.tests.testOctavePkgTests
• octavePackages.stk.tests.testOctavePkgTests
• octavePackages.symbolic.tests.testOctavePkgTests
• octavePackages.video.tests.testOctavePkgTests

[7c6f434c]

Looks good (but I have only used plain Octave, and it has been some time)

[doronbehar]

Thanks for the PR @pbsds! I was too thinking of doing something like this in the past, but never had the will power to do so. @KarlJoad Do you think it would be possible to use add (by default) a checkPhase that would check that would run a pkg install?

[pbsds]

If constructing an octave env or checkPhase proves too difficult we could pivot to a two-stage build where the second stage has access to the octave.withPackages env and simply outputs a symlink to the first stage store path

[KarlJoad]

@doronbehar

The buildPhase of buildOctavePackage just uses octave to build the add-on package (compiling C/C++/Fortran) and copying .m files to their final location, before tar-ing them up into Octave's preferred installation format, giving us the output tarball we get in $out.

Do you think it would be possible to use add (by default) a checkPhase that would check that would run a pkg install?

That would be possible, but like I said earlier in this thread, performing a pkg install does not actually reveal any useful information. The pkg install step literally just un-tar's the octavePackages $out tarball and copies the contents. If we did do this, then we would either see:

1. Fail before the pkg install because the code that needs compilation (C/C++/Fortran) cannot be compiled. CI would catch this.
2. Fail after the pkg install because there are errors in the compiled artifact or the interpreted Octave/Matlab code. Nix would not catch this, and this is the one we face more often than not.

Ideally, if we added a checkPhase to buildOctavePackage, we would have it not only install the package, but also run pkg test. That what I would have liked to do. I don't immediately see a reason why such a checkPhase could not exist, but it gets quite complicated quite quickly. The tricky part lies in building the environment where Octave can find these external add-ons so that we can invoke Octave's pkg install or pkg test.

To have the checkPhase run pkg install and pkg test, we have to do the same work as when someone uses octave.withPackages. If you take a look at octave/build-env.nix and octave/wrap.sh for this, we have to do a bunch of work to tell Octave where to install the requested package and how to find the installed package. Roughly, we have to do this:

1. Wrap Octave so that OCTAVE_SITE_INITFILE points to something we can control.
2. Use the list of 3rd party octavePackages provided in the withPackages list and install them, putting them in a well-known location. This is what creates a package list that we point to with the init-file in the next step.
3. Create the site init-file that points to the list of locally installed packages. Octave makes the distinction between globally and locally installed packages, where globally-installed ones are installed by the system administrator. We only ever do local installations, since those can be overridden with the init file.

Most of that work is to work around the (frankly) baroque package setup that Octave uses. Octave has no environment variables to specify where packages are located.

[pbsds]

Of note, as stated in the op, the pkg install test did detect #406300. It may not be because of octave however it may be the rest of the stdenv machinery, seeing as the failure was that the library it tried to link to was not found.

[KarlJoad]

@pbsds, #406300 was found by me manually running the pkg test script I have. Whatever existed in passthru when that PR was opened did not inform us of the runtime error.

[doronbehar]

With the pkg test passthru added, I found the following failures:

/nix/store/4i39kgpnrh0qb800g3q5ja7w67w8wbxl-octave-10.1.0-ncarray-1.0.6-pkg-test.drv
/nix/store/528a2a5bzvblxqfgv12121wkmha4xkcv-octave-10.1.0-env.drv
/nix/store/6jrxaxgqbf72czv47y2h38fabsdb7f1g-octave-10.1.0-video-2.1.3-pkg-test.drv
/nix/store/872w7p4qpyp88h7bj1nq5560kzy60ckz-octave-10.1.0-interval-3.2.1-pkg-test.drv
/nix/store/9mabfg4lmp60iz584bjgc0igg3n8ngii-octave-10.1.0-image-acquisition-0.3.0-pkg-test.drv
/nix/store/9mdpaqw8i637xwv3p0fq7mcldp71l1pz-octave-10.1.0-geometry-unstable-2021-07-07-pkg-test.drv
/nix/store/f17xjdg2kzq00xz5qvw7faa8xwaic2zw-octave-10.1.0-gsl-2.1.1-pkg-test.drv
/nix/store/gx5kavqxxbikwwry5rfpx2b6g106azn4-octave-10.1.0-msh-1.0.12-pkg-test.drv
/nix/store/hpscc4df3qxi2vf72s4rk5s5zn5gg95x-octave-10.1.0-financial-0.5.3-pkg-test.drv
/nix/store/hr3wyy2q0dgw0p5zndc11188bnvjm9qr-octave-10.1.0-symbolic-3.2.1-pkg-test.drv
/nix/store/hsii3pzkgrd3diy82h8rcmmaiymf7wai-octave-10.1.0-mapping-1.4.2-pkg-test.drv
/nix/store/i3z0b11flk43zygw4dyzj84g5pgqads6-octave-10.1.0-audio-2.0.5-pkg-test.drv
/nix/store/jqifw5fgw8ixrv4phlzmzjydkl32l3rn-octave-10.1.0-arduino-0.10.0-pkg-test.drv
/nix/store/k8i5b5icp301ay2c44jadg086z2nhgjp-octave-10.1.0-database-2.4.4-pkg-test.drv
/nix/store/kfhws65ap29f2g53p369bv33540p5yd6-octave-10.1.0-env.drv
/nix/store/l187whcckkbp51xiqxwxcdj34m3czl7d-octave-10.1.0-env.drv
/nix/store/mgag7d0x6ch0fzrsrr4zrpg8apgvwiiw-octave-10.1.0-stk-2.8.1-pkg-test.drv
/nix/store/na4kxiq1h55ykzpgqggqallrfv48jgw6-octave-10.1.0-env.drv
/nix/store/nzhppa2kdajsdamprq54wjz4nsah6f8j-octave-10.1.0-image-2.16.1-pkg-test.drv
/nix/store/pslhgz1q5q6zx50dka2c8j3w73lxvhlh-octave-10.1.0-control-3.6.1-pkg-test.drv
/nix/store/pvqwm7kl1x5r1a80fa6gwd2chskm0nwh-octave-10.1.0-lssa-0.1.4-pkg-test.drv
/nix/store/vb0rq6karzcdxynh8509nv231wq07npb-octave-10.1.0-signal-1.4.6-pkg-test.drv
/nix/store/vcy3r60z4bzgjl7frxvw9j6v5sxqkbs2-octave-10.1.0-instrument-control-0.9.5-pkg-test.drv
/nix/store/wfag95gzb8z6bp2jrlgh18yzxbq9p0zd-octave-10.1.0-queueing-1.2.7-pkg-test.drv
/nix/store/wk2s6g4903c4q50g91fpj80v68kix2j3-octave-10.1.0-statistics-1.7.0-pkg-test.drv
/nix/store/ydf2mn9a86hsx6zwv9i0gh41gbmyvyj2-octave-10.1.0-nan-3.7.0-pkg-test.drv
/nix/store/ysx5d8w5w8960c3c8krj28pn4i6iawg6-octave-10.1.0-env.drv

Not sure exactly what to which package do the -env failures belong to... Also, not sure how whether we should mark these packages as broken due to these failures.

[pbsds]

Any blockers?

@pbsds, #406300 was found by me manually running the pkg test script I have. Whatever existed in passthru when that PR was opened did not inform us of the runtime error.

Yes, i did not indent to claim credit. It was written in reference to the OP, but a s/did detect/does detect/ phrasing was still called for.

[pbsds]

I pushed a named change to make it easier to see which package the -env failures are for

[doronbehar]

Any blockers?

I'm just not sure whether we should mark these packages as broken or not. If we'll fail to reach a conclusion we can merge this as is..

[KarlJoad]

Marking these packages as broken just because they failed their unit tests may be a bit heavy-handed. In the past, there have been packages which fail their built-in unit tests for one reason or another, and the previous version was broken in the same way.

We should look at the test logs to see what the reported failures are before we make a decision.

[doronbehar]

OK you convinced me. I'll merge this now.