Tracking: @test_broken tests audit and status

# Broken Tests Summary - Current State

This document summarizes tests that are currently marked as `@test_broken` in the Catalyst test suite, organized by category and root cause.

**Last updated:** 2026-02-10

---

## Summary Statistics

- **Total `@test_broken` occurrences:** 54
- **Upstream MTK issues:** ~19
- **Upstream Symbolics issues:** ~10
- **Conservation law issues:** ~10
- **Catalyst internal issues:** ~3
- **Other (extensions, visualization, misc):** ~12

---

## Tests Blocked by MTK Issues

### Misformatted Events Not Throwing Errors ([MTK #4167](https://github.com/SciML/ModelingToolkit.jl/issues/4167))

Some malformed event syntax doesn't produce errors.

| File | Lines | Description |
|------|-------|-------------|
| [events.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/reactionsystem_core/events.jl) | 160 | Misformatted events don't yield errors |

### SteadyStateProblem Issues ([MTK #4174](https://github.com/SciML/ModelingToolkit.jl/issues/4174))

SteadyStateProblem solving fails in certain cases.

| File | Lines | Description |
|------|-------|-------------|
| [coupled_equation_crn_systems.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/reactionsystem_core/coupled_equation_crn_systems.jl) | 177 | SteadyStateProblem solving fails |

### Rational Parameter Equivalence ([MTK #4163](https://github.com/SciML/ModelingToolkit.jl/issues/4163))

Rational numbers with equivalent values but different representations aren't considered equal.

| File | Lines | Description |
|------|-------|-------------|
| [coupled_equation_crn_systems.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/reactionsystem_core/coupled_equation_crn_systems.jl) | 400 | `2//10` vs equivalent rational |

### Higher-Order Differentials ([MTK #4186](https://github.com/SciML/ModelingToolkit.jl/issues/4186))

Cannot generate systems with higher-order differentials.

| File | Lines | Description |
|------|-------|-------------|
| [coupled_equation_crn_systems.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/reactionsystem_core/coupled_equation_crn_systems.jl) | 563 | Entire test block broken |

### IV Mismatch Error Handling ([MTK #4092](https://github.com/SciML/ModelingToolkit.jl/issues/4092))

Error handling for independent variable mismatch doesn't work as expected.

| File | Lines | Description |
|------|-------|-------------|
| [reactionsystem.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/reactionsystem_core/reactionsystem.jl) | 669 | `@test_throws` as broken |

### save_idxs for JumpProblems

`save_idxs` argument doesn't work for jump problems.

| File | Lines | Description |
|------|-------|-------------|
| [mtk_structure_indexing.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/upstream/mtk_structure_indexing.jl) | 215-228 | Various `save_idxs` forms fail |

### Problem Indexing ([MTK #4187](https://github.com/SciML/ModelingToolkit.jl/issues/4187))

| File | Lines | Description |
|------|-------|-------------|
| [mtk_structure_indexing.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/upstream/mtk_structure_indexing.jl) | 319 | Entire test block broken |

### NonlinearProblem with Vector Variables

| File | Lines | Description |
|------|-------|-------------|
| [mtk_problem_inputs.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/upstream/mtk_problem_inputs.jl) | 342 | Vector variable handling in NonlinearProblem |

### mtkcompile Equation Elimination for SDEs

`mtkcompile` doesn't eliminate equations for SDEs like it does for ODEs.

| File | Lines | Description |
|------|-------|-------------|
| [component_based_model_creation.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/compositional_modelling/component_based_model_creation.jl) | 105, 118, 135, 255 | Equation count after simplification |

### Solution Interpolation with Tuple ([SciMLBase #711](https://github.com/SciML/SciMLBase.jl/issues/711))

| File | Lines | Description |
|------|-------|-------------|
| [mtk_structure_indexing.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/upstream/mtk_structure_indexing.jl) | 249 | `sol(0.0; idxs = (XY,Y))` fails |

---

## Tests Blocked by Symbolics Issues

### Symbolic Simplification ([Symbolics.jl #1739](https://github.com/JuliaSymbolics/Symbolics.jl/issues/1739))

Expressions that should simplify to zero don't always do so.

| File | Lines | Description |
|------|-------|-------------|
| [network_properties.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/network_analysis/network_properties.jl) | 410, 460, 464, 477-478, 485 | Rate equations don't simplify to zero |
| [coupled_equation_crn_systems.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/reactionsystem_core/coupled_equation_crn_systems.jl) | 807-810 | Equation equality comparisons |

---

## Conservation Law Issues (NonlinearSystem `remake`)

These tests involve `remake` of `NonlinearProblem`s with `remove_conserved = true`. The conservation law parameter Gamma and eliminated species values don't update correctly.

| File | Lines | Description |
|------|-------|-------------|
| [conservation_laws.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/network_analysis/conservation_laws.jl) | 345 | Entire test block - ODE/SDE remake with conservation laws |
| [conservation_laws.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/network_analysis/conservation_laws.jl) | 503 | Entire test block - NonlinearSystem warnings and remake |
| [conservation_laws.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/network_analysis/conservation_laws.jl) | 550, 555, 560, 565, 596, 597, 601, 602 | Individual assertions within the block |

---

## Catalyst Internal Issues

### Metadata in Completed System (#1353)

| File | Lines | Description |
|------|-------|-------------|
| [reactionsystem.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/reactionsystem_core/reactionsystem.jl) | 982 | `getmetadata` on completed system |

### DSL Independent Variable Test

| File | Lines | Description |
|------|-------|-------------|
| [dsl_options.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/dsl/dsl_options.jl) | 525 | `@reaction_network ivstest` block - MTK bug in collect_vars! |

### Transport Reaction Macro with Interpolation

| File | Lines | Description |
|------|-------|-------------|
| [spatial_reactions.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/spatial_modelling/spatial_reactions.jl) | 105 | `@transport_reaction dZ $species3` syntax |

---

## Extension/Visualization Issues

### BifurcationKit + Conservation Laws

Awaiting response from maintainer.

| File | Lines | Description |
|------|-------|-------------|
| [bifurcation_kit.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/extensions/bifurcation_kit.jl) | 20, 152 | Entire test blocks broken |

### Makie Plot Internals Changed

The interface for accessing Makie plot internals has changed.

| File | Lines | Description |
|------|-------|-------------|
| [lattice_simulation_plotting.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/extensions/lattice_simulation_plotting.jl) | 36, 45, 76 | Plot value access |

### Latexify Issues

Various latexify output differences.

| File | Lines | Description |
|------|-------|-------------|
| [latexify.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/visualisation/latexify.jl) | 169, 190, 198 | LaTeX output mismatches |

---

## Other Issues

### Jump Event Comparison

| File | Lines | Description |
|------|-------|-------------|
| [events.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/reactionsystem_core/events.jl) | 438 | Solution comparison with events in jumps - may need test redesign |

### Rare SDE Failure

| File | Lines | Description |
|------|-------|-------------|
| [simulate_SDEs.jl](https://github.com/SciML/Catalyst.jl/blob/master/test/simulation_and_solving/simulate_SDEs.jl) | 279 | Rare non-reproducible SDE failure |

---

## Previously Broken Tests Now Fixed

The following tests from the earlier MTK v10/v11 migration were fixed:

- **hybrid_models.jl:244** - Discrete events in hybrid models
- **simulate_jumps.jl:195-196** - RSSA dependency graph structure
- **reactionsystem.jl:208** - MassActionJump internal structure
- **reactionsystem.jl:520** - Jump system with interpolated species
- **symbolic_stoichiometry.jl:179** - VariableRateJump assembly
- **events.jl:336-339, 363-365** - Solution parameter access after events (MTK #4030) - NOW FIXED
- **dsl_options.jl:663** - Integer symtype inference - test removed/was never broken
- **simulate_jumps.jl:249,265** - Integer type preservation for jumps (MTK #4170) - FIXED by `c314ed5f7`
- **events.jl:145,236** - ReactionSystem equality with events - FIXED by switching to `Catalyst.isequivalent`
- **dsl_options.jl:1609,1629** - isequal for systems with events - FIXED by switching to `Catalyst.isequivalent`
- **custom_crn_functions.jl:201-203** - Events equality after deepcopy - FIXED by switching to `Catalyst.isequivalent`/`continuous_events_equal`/`discrete_events_equal`
- **reactionsystem_serialisation.jl:358,360** - Serialised system equality with events - FIXED by switching to `Catalyst.isequivalent`
- **custom_crn_functions.jl:218-219** - Expanded event affects were `nothing` - FIXED by correcting `expand_registered_functions` to properly pass affect eqs, handle `affect_neg`, and preserve `discrete_parameters`

---

## Removed/Commented Out Tests

The following tests that were previously `@test_broken` have been fully commented out:

- **latexify.jl:178** - Changed to comment, not `@test_broken`
- **latexify.jl:279, 282** - Entire test block commented out (extend/compose issues with System+ReactionSystem)

---

## Recommendations

### Medium Priority
1. Investigate conservation law `remake` issues for NonlinearProblems
2. File/track Catalyst #1353 for metadata issue
3. Update Makie plot tests for new internals

### Low Priority
4. Latexify output differences (cosmetic)
5. BifurcationKit integration (awaiting upstream)


File	Lines	Description
network_properties.jl	410, 460, 464, 477-478, 485	Rate equations don't simplify to zero
coupled_equation_crn_systems.jl	807-810	Equation equality comparisons

File	Lines	Description
conservation_laws.jl	345	Entire test block - ODE/SDE remake with conservation laws
conservation_laws.jl	503	Entire test block - NonlinearSystem warnings and remake
conservation_laws.jl	550, 555, 560, 565, 596, 597, 601, 602	Individual assertions within the block

Uh oh!

Tracking: @test_broken tests audit and status #1407

Description

Broken Tests Summary - Current State

Summary Statistics

Tests Blocked by MTK Issues

Misformatted Events Not Throwing Errors (MTK #4167)

SteadyStateProblem Issues (MTK #4174)

Rational Parameter Equivalence (MTK #4163)

Higher-Order Differentials (MTK #4186)

IV Mismatch Error Handling (MTK #4092)

save_idxs for JumpProblems

Problem Indexing (MTK #4187)

NonlinearProblem with Vector Variables

mtkcompile Equation Elimination for SDEs

Solution Interpolation with Tuple (SciMLBase #711)

Tests Blocked by Symbolics Issues

Symbolic Simplification (Symbolics.jl #1739)

Conservation Law Issues (NonlinearSystem remake)

Catalyst Internal Issues

Metadata in Completed System (#1353)

DSL Independent Variable Test

Transport Reaction Macro with Interpolation

Extension/Visualization Issues

BifurcationKit + Conservation Laws

Makie Plot Internals Changed

Latexify Issues

Other Issues

Jump Event Comparison

Rare SDE Failure

Previously Broken Tests Now Fixed

Removed/Commented Out Tests

Recommendations

Medium Priority

Low Priority

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Conservation Law Issues (NonlinearSystem `remake`)