Tests failing on Julia 1.11 (and probably 1.12)

[ablaom]

Edit. It seems test pass on 1.10 but fail on 1.11. Multiple failures are documented below. As noted, it seems like the issue is the use of unstable RNGs in testing.

Test Summary: | Pass  Total  Time
class_counts  |   12     12  2.4s
Test Summary:  | Pass  Total  Time
table_wrappers |    6      6  2.8s
Test Summary:    | Pass  Total  Time██████████████████████████████████▍                           |  ETA: 0:00:00
generic_resample |   14     14  9.4s
testing checkbalance: Test Failed at /Users/anthony/MLJ/Imbalance/test/extras.jl:16
  Expression: c.output == "1: ▇▇▇▇▇▇ 10 (11.1%) \n0: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 90 (100.0%) \n"
   Evaluated: "1: ▇▇▇▇▇ 9 (9.9%) \n0: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 91 (100.0%) \n" == "1: ▇▇▇▇▇▇ 10 (11.1%) \n0: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 90 (100.0%) \n"

Stacktrace:
 [1] macro expansion
   @ ~/.julia/juliaup/julia-1.11.2+0.x64.apple.darwin14/share/julia/stdlib/v1.11/Test/src/Test.jl:679 [inlined]
 [2] macro expansion
   @ ~/MLJ/Imbalance/test/extras.jl:16 [inlined]
 [3] macro expansion
   @ ~/.julia/juliaup/julia-1.11.2+0.x64.apple.darwin14/share/julia/stdlib/v1.11/Test/src/Test.jl:1704 [inlined]
 [4] top-level scope
   @ ~/MLJ/Imbalance/test/extras.jl:4
Test Summary:          | Fail  Total  Time
extras                 |    1      1  3.6s
  testing checkbalance |    1      1  3.6s
ERROR: LoadError: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken.
in expression starting at /Users/anthony/MLJ/Imbalance/test/runtests.jl:46
ERROR: Package Imbalance errored during testing

julia> VERSION
v"1.11.2"

[ablaom]

@EssamWisam Can you reproduce?

[ablaom]

Also (after marking previous as broken):

Tomek Undersampler |    3      3  0.6s
Progress: 100%|███████████████████████████████████████████████████████████████████████████████████| Time: 0:00:00
   class: 0
cluster undersampler: Test Failed at /Users/anthony/MLJ/Imbalance/test/undersampling/cluster_undersample.jl:38
  Expression: x_n ≈ X_c0_under[:, 2]
   Evaluated: [3.091, 3.233] ≈ [3.092, 3.35]

Stacktrace:
 [1] macro expansion
   @ ~/.julia/juliaup/julia-1.11.2+0.x64.apple.darwin14/share/julia/stdlib/v1.11/Test/src/Test.jl:679 [inlined]
 [2] macro expansion
   @ ~/MLJ/Imbalance/test/undersampling/cluster_undersample.jl:38 [inlined]
 [3] macro expansion
   @ ~/.julia/juliaup/julia-1.11.2+0.x64.apple.darwin14/share/julia/stdlib/v1.11/Test/src/Test.jl:1704 [inlined]
 [4] top-level scope
   @ ~/MLJ/Imbalance/test/undersampling/cluster_undersample.jl:4
cluster undersampler: Test Failed at /Users/anthony/MLJ/Imbalance/test/undersampling/cluster_undersample.jl:53
  Expression: sum((X_under[y_under .== 2, :])') ≈ sum((kmeans(X_c0, n; maxiter = 100, rng = MersenneTwister(122))).centers)
   Evaluated: 94.09151666666668 ≈ 93.90754047619049

[ablaom]

And another:

Tomek Undersampler |    3      3  0.7s
Progress: 100%|███████████████████████████████████████████████████████████████████████████████████| Time: 0:00:00
   class: 0
cluster undersampler: Test Failed at /Users/anthony/MLJ/Imbalance/test/undersampling/cluster_undersample.jl:53
  Expression: sum((X_under[y_under .== 2, :])') ≈ sum((kmeans(X_c0, n; maxiter = 100, rng = MersenneTwister(122))).centers)
   Evaluated: 94.09151666666668 ≈ 93.90754047619049

Stacktrace:
 [1] macro expansion
   @ ~/.julia/juliaup/julia-1.11.2+0.x64.apple.darwin14/share/julia/stdlib/v1.11/Test/src/Test.jl:679 [inlined]
 [2] macro expansion
   @ ~/MLJ/Imbalance/test/undersampling/cluster_undersample.jl:53 [inlined]
 [3] macro expansion
   @ ~/.julia/juliaup/julia-1.11.2+0.x64.apple.darwin14/share/julia/stdlib/v1.11/Test/src/Test.jl:1704 [inlined]
 [4] top-level scope
   @ ~/MLJ/Imbalance/test/undersampling/cluster_undersample.jl:4
Test Summary:          | Pass  Fail  Broken  Total  Time
Cluster Undersampler   |    2     1       1      4  4.9s
  cluster undersampler |    1     1       1      3  4.8s
  materializer         |    1                    1  0.0s
ERROR: LoadError: Some tests did not pass: 2 passed, 1 failed, 0 errored, 1 broken.
in expression starting at /Users/anthony/MLJ/Imbalance/test/runtests.jl:116
ERROR: Package Imbalance errored during testing

[ablaom]

This may not be the issue, but I see there isMersenneTwister in testing. One should only use StableRNG from the StableRNGs.jl package for testing, as behaviour of MersenneTwister and other RNGs is not guaranteed to be stable over julia versions.

[ablaom]

I just found that tests pass on v"1.10.10". So might be worth checking out the RNG issue first.

[ablaom]

Here's a telling code comment:

Imbalance.jl/test/undersampling/cluster_undersample.jl

Line 10 in 82d7e8b

# changing RNG is not recommended

[EssamWisam]

This may not be the issue, but I see there isMersenneTwister in testing.

Do I understand correctly that this comment is outdated.

As noted, it seems like the issue is the use of unstable RNGs in testing.

Compared to this one?

[ablaom]

Yes. My best guess is that this is an RNG issue after all. We should replace all RNG's in testing with StableRNGs.StableRNG(seed) because MersenneTwister, Xoshiro, etc can change (do change!) with Julia versions.

Even if this is not the issue. We should be doing this anyway.

[ablaom]

@EssamWisam Any progress here?

[EssamWisam]

@ablaom thank you for the reminder. Scheduling for this week.

[EssamWisam]

I fixed this; it was mostly rng problem 60faf3a

I forgot that dev is our main branch and pushed their directly.