Unconditional constraint biases distribution, ignoring vsc.dist preference

[silobadTT]

Hey,

I am observing that an unconditional not_inside constraint on a random variable (r_fmt_in) is causing its vsc.dist preference to be ignored, resulting in a heavily skewed distribution.

To isolate this, I created two test classes:

• RandOpsA: Applies a dist for a uniform distribution over many formats, but also has a hard, unconditional constraint that r_fmt_in cannot be an MX format.
• RandOpsB: The control case, which is identical but lacks the unconditional not_inside constraint.

My expectation is that RandOpsA should produce a uniform distribution among the remaining non-MX formats. However, the results show a significant bias towards Float32. This suggests the solver is finding an "easy" solution path (likely by satisfying a downstream constraint first and working backward) and is not honoring the dist preference, even when the r_disable_mx variable and if_then logic are removed from the equation. I am following the ongoing issue #242 , which is why I included the solve_order constraint to ensure that the variables are getting 'swizzled/randomized'.

---

Code to Reproduce

This self-contained script profiles both classes and prints the resulting distributions.

pyvsc_test.py

RandOpsA:

Distribution for: r_disable_mx
    - 0 : 158   ( 52.67%) ##########################
    - 1 : 142   ( 47.33%) #######################

Distribution for: r_fmt_in
    - Float32   : 152   ( 50.67%) #########################
    - Fp8e5m2   : 18    (  6.00%) ###
    - Int16     : 18    (  6.00%) ###
    - Int32     : 18    (  6.00%) ###
    - Int8      : 18    (  6.00%) ###
    - Float16   : 17    (  5.67%) ##
    - Fp8e4m3   : 15    (  5.00%) ##
    - Tf32      : 15    (  5.00%) ##
    - Uint8     : 15    (  5.00%) ##
    - Float16_b : 14    (  4.67%) ##

RandOpsB (without 'if_then' constraint):

Distribution for: r_disable_mx
    - 0 : 158   ( 52.67%) ##########################
    - 1 : 142   ( 47.33%) #######################

Distribution for: r_fmt_in
    - Fp8e5m2   : 22    (  7.33%) ###
    - MXFp6e2m3 : 21    (  7.00%) ###
    - Int16     : 20    (  6.67%) ###
    - Fp8e4m3   : 19    (  6.33%) ###
    - Float16_b : 18    (  6.00%) ###
    - MXFp4e2m1 : 18    (  6.00%) ###
    - Tf32      : 18    (  6.00%) ###
    - Uint8     : 18    (  6.00%) ###
    - Int8      : 17    (  5.67%) ##
    - MXFp8e4m3 : 17    (  5.67%) ##
    - MXFp8e5m2 : 17    (  5.67%) ##
    - MXInt8    : 16    (  5.33%) ##
    - Float16   : 14    (  4.67%) ##
    - MXFp6e3m2 : 14    (  4.67%) ##
    - MXInt2    : 14    (  4.67%) ##
    - Float32   : 13    (  4.33%) ##
    - Int32     : 13    (  4.33%) ##
    - MXInt4    : 11    (  3.67%) #

The isolated case is misleading; I can obviously take the MX formats out of the vsc.dist() constraint, but the end-goal is to have the constraint look as follows with an if_then statement:

@vsc.constraint
  def c_disable_mx_op_in(self):
    vsc.solve_order(self.r_disable_mx, self.r_fmt_in)
    with vsc.if_then(self.r_disable_mx == 1):
      self.r_fmt_in.not_inside(vsc.rangelist(
        FormatType.MXInt8,
        FormatType.MXInt4,
        FormatType.MXInt2,
        FormatType.MXFp8e5m2,
        FormatType.MXFp8e4m3,
        FormatType.MXFp6e3m2,
        FormatType.MXFp6e2m3,
        FormatType.MXFp4e2m1,
      ))

... but the issue that I am describing persists and I wanted to make the directed case as simple as possible

[alwilson]

Currently the partsel swizzler uses the dist constraints to do a random, weighted selection of the possibilities. If that value isn't satisfiable given other constraints, as in your example with the not_inside, then that constraint is tossed, which leaves the solver to find a low effort solution of 0.

I think there are a few ways the swizzler could be better here. It should probably test if the selected dist value is satisfiable, retry so many times, and end with a random bit pattern constraint as a fail safe to prevent bias. There could be some sort of pass or exploration step early on to eliminate dist values that are impossible, but that's pretty expensive without adding caching to pyvsc.

[silobadTT]

I've observed the same behaviour when my input value range is defined as a vsc.rangelist(). Does this behaviour of "tossing" constraints due to conflicts introduced by not_inside() extend to vsc.rangelist() as well?

[alwilson]

It would extend to any constraint that excludes values in the distribution. Currently pyvsc doesn't do much to explore the solution space for a variable before randomizing it. It does factor in constraints that establish clear min/max bounds, like x < 50. It probably could use rangelist() to exclude impossible values or ranges from the distribution, but I'm not sure it does.

For distribution constraints pyvsc uses the distribution itself and whatever min/max it was able to glean for that field to generate a weighted random value constraint. It later tries to apply that constraint with the solver and if it fails it throws it out.

For most fields w/o a dist constraint each individual bit gets a constrained to a random value, and all those bits get checked against the solver in a random order. For distribution constraints it currently only tries once. If the constraints are such that it's improbable to select a valid constraint, then that field doesn't get that random bit constraint or swizzling treatment. My current idea to alleviate this is to try N random samples against the solver, and if those all fail then randomize it as if it didn't have a dist constraint. At least that way you get a more uniform distribution as a fail safe.

[silobadTT]

Yes, a fix like this would greatly improve the reusability of randomizable objects. For example, I have a single class for a base hardware operation (A) that I want to reuse in different sequences, like A -> B -> C or A -> D -> C. Each use case requires slightly different constraints on A. Instead of defining a separate class for each variation, the ideal approach is to layer on external constraints where needed, such as, 'in this specific sequence, do not allow MX input formats for A.'

[alwilson]

@silobadTT Alright, I've added a PR that should do what I described. I'm interested if that goes far enough to fix the issues you've seen. The weights you have in your example are all a value of 1, so the fallback works well there. The somewhat uniform distribution of the fallback should become more obvious as those weights change.

[silobadTT]

I've validated this change locally across my complex operation sequences and can confirm it's working as intended. The randomization distribution is significantly improved. My brief testing shows a ~20% performance regression, which is an acceptable and expected trade-off for the more robust constraint solving