Improvements to parallel assignment #3748

d-torrance · 2025-04-23T16:28:42Z

We add various parallel assignment features, e.g:

Parallel assignment for mutable hash tables and lists

i1 : x = new MutableHashTable;

i2 : y = new MutableList;

i3 : (x.foo, y#0) = (1, 2)

o3 = (1, 2)

o3 : Sequence

i4 : peek x

o4 = MutableHashTable{foo => 1}

i5 : peek y

o5 = MutableList{2}

Partial parallel assignment

i6 : (x,,y) = (1,2,3)

o6 = (1, , 3)

o6 : Sequence

i7 : x

o7 = 1

i8 : y

o8 = 3

Parallel method installation

i9 : X = new Type of HashTable;

i10 : foo = method();

i11 : (foo X, X + X) := (x -> 1, (x,y) -> 2)

o11 = (FunctionClosure[stdio:11:19-11:25], FunctionClosure[stdio:11:27-11:37])

o11 : Sequence

i12 : (x, y) = (new X, new X)

o12 = (X{}, X{})

o12 : Sequence

i13 : foo x

o13 = 1

i14 : x + y

o14 = 2

Parallel augmented assignment

Edit: I decided against (x,y,z) += 1 being equivalent to (x,y,z) += (1,1, 1).

i15 : (x,y,z) = (1,2,3)

o15 = (1, 2, 3)

o15 : Sequence

i16 : (x,y,z) += (4,5,6)

o16 = (5, 7, 9)

o16 : Sequence

i17 : x

o17 = 5

i18 : y

o18 = 7

i19 : z

o19 = 9

Fixes precedence issues with assignment

See #3733

i25 : (X >> X) := (x,y) -> 5

o25 = FunctionClosure[stdio:25:12-25:22]

o25 : FunctionClosure

i26 : (x,y) = (new X, new X)

o26 = (X{}, X{})

o26 : Sequence

i27 : x >> y

o27 = 5

TODO:

Finish tests
Add documentation
Update >> docs to mention that we need parentheses for installing methods

Closes: #1774

mahrud · 2025-04-23T20:08:56Z

I don't love (foo X, X + X) := (x -> 1, (x,y) -> 2) ...

d-torrance · 2025-04-24T00:50:34Z

Yeah I don't expect anyone would use parallel assignment much for method installation. It just seems like something that ought to work if we want = and := to work in general. Also, it lets us put parentheses around a single expression for grouping, solving the >> method installation issue.

mahrud · 2025-04-25T00:43:35Z

I'm gonna unsubscribe because there are a lot of force pushes but the notifications don't show what has actually changed. @ me when it is ready.

This will allow us to figure out the symbol later for augmented/parallel assignment. Also merge "unary" and "postfix" members of Symbol struct. A symbol can't have both anyway, and since we're storing the symbol and not the specific function, we wouldn't be sure which we want.

Then we can use this info for augmented/parallel assignment.

This info is now saved in the lhs Code object

Now contains a CodeSequence of the elements on the left hand side instead of symbol information. This will enable us to support other types of code in the left hand side.

The left hand side is now a sequence of Code objects rather than a list of symbols. Also add support for (x) = y (behaves just like x = y).

e.g., assignment to mutable hash tables and lists and installing methods

There was likely a typo in the file, and the last input likely was supposed to be b and not a. This apparently was "fixed" at some point, as this is the behavior in Macaulay2 1.24.11: i1 : (b) = 1:x o1 = 1 : (x) o1 : Sequence i2 : b o2 = x o2 : Symbol However, this seems incorrect, as (b) is not a sequence in Macaulay2, but just b itself surrounded by parentheses for grouping.

bugs/dan/0-parallel-assignment

mahrud · 2025-04-30T15:05:33Z

M2/Macaulay2/d/evaluate.d

-    if nlhs == 1 then value = seq(value);
+    if nlhs == 1 then (
+	when value is a:Sequence do (
+	    -- unless y is a sequence of length 1, then it behaves like x = y#0


Just to make sure, (x) = (1,2,3) should still be an error, correct?

In the current draft, it's equivalent to x = (1,2,3). But for consistency, you're right -- it probably should be an error. I'll push a new version here in a moment.

@d-torrance @mahrud Is this agreed on, should I take a separate look?

M2/Macaulay2/d/evaluate.d

mahrud · 2025-04-30T21:50:05Z

One more thing, either for now or a future PR: would be great if this kind of parallel assignment also worked:

i1 : for (a,b) in {(1,2), (3,4)} list a
stdio:1:4:(3): error: expected symbol

d-torrance · 2025-05-01T01:11:28Z

Ooh yeah that would be nice! I played around with it for a bit and it would require a lot of pretty low-level changes to binding and whatnot. Let's hold off for a future PR.

mahrud · 2025-05-01T02:41:11Z

M2/Macaulay2/d/common.d

-export setuppostfix(e:SymbolClosure,fn:unop):void := (
-     unopNameList = unopNameListCell(fn,e.symbol,unopNameList);
-     e.symbol.postfix = fn;
-     );


I understand what this commit does, but was it necessary? I see the simplification, but also seems like not a bad idea to be able to differentiate postfix vs unary operators in the interpreter.

Once we're done parsing, there's really no difference between prefix and postfix unary operators. We just stuff the corresponding function in the unaryCode object and call it a day. And a unary operator can't be both prefix and postfix anyway -- it has to be one or the other. So it didn't seem necessary to have two separate fields in the Symbol struct.

Also, with this PR I'm proposing the unaryCode objects store the operator symbol itself (for potential use by parallel/augmented assignment) instead of the function that gets called. And if the symbol has both a prefix and postfix function that might get called, we'd have to do some extra stuff to figure out which one to use.

It should still be possible to differentiate between prefix and postfix by looking at the parseinfo member of the corresponding Word object.

mahrud · 2025-05-01T02:42:57Z

M2/Macaulay2/d/parse.d

@@ -210,7 +210,6 @@ export augmentedAssignmentCode := {+
    oper:Symbol,
    lhs:Code,
    rhs:Code,
-    info:Symbol, -- variable name or operator


Is there any reason to keep this for debugging purposes?

This was a hack that I came up when adding augmented assignment. There were two unrelated cases where we needed some information about a symbol that wasn't being stored in the Code object:

The variable name for globalMemoryReferenceCode and threadMemoryReferenceCode, which currently just store the frame index of the variable.

The binary/unary operator for binaryCode and unaryCode, which currently just store the function associated with the operator.

With this PR, the proposal is that these various Code objects now store the symbols themselves. So the symbol that was in info is still there, but now inside lhs.

mahrud · 2025-05-01T02:46:50Z

M2/Macaulay2/m2/integers.m2

@@ -116,6 +116,7 @@ scan({symbol +=, symbol -=, symbol &=, symbol |=, symbol ^^=},

 store = method()
 store(AtomicInt, ZZ) := atomicStore
+(<- AtomicInt) := store


Why is this method defined this way?

M2/Macaulay2/packages/Macaulay2Doc/operators/assignment.m2

mahrud · 2025-05-01T02:51:30Z

M2/Macaulay2/packages/Macaulay2Doc/operators/assignment.m2

-	       { "If the left hand side has more elements than the right hand side, then the extra symbols on the left side are given the value ", TO "null", "." },
-	       { "If the left hand side has fewer elements than the right hand side, then the last symbol on the left hand side is given
-     		    as value a sequence containing the trailing elements of the right hand side." 
-		    },
-	       { "If the right hand side is not a sequence, then it is assigned to the first symbol on the left, and the remaining symbols are assigned the
-		    value ", TO "null", "."
-		    }
+	       { "the number of expressions must match the number of variables." }


When was this documented behavior removed? It actually seems preferable to me.

No idea. I was surprised to learn about it!

It probably wouldn't be too much trouble to restore it.

mahrud

The amount of changes is not overwhelming, but I don't feel confident in thoroughly reviewing this before the next release. I think we should merge this after the release, so any potential bugs have some time to reveal themselves.

Also, several aspects of this touch on things that @pzinn has more experience with, so his review may also be valuable.

d-torrance · 2025-05-01T12:04:06Z

I'm totally fine with waiting for the next release.

mahrud · 2025-05-01T14:42:38Z

How much harder is it to also allow (a,b,c) = {1,2,3} and (a,b,c) = [x,y,z] etc?
(I don't think there's much point in {a,b,c} = {1,2,3}, other than maybe if you think it's good to return error if the types don't match?)

d-torrance · 2025-05-01T15:17:41Z

Adding support for the various types of visible list on the right hand side would be very simple, I think. Ensuring that the types match would be a bit trickier since we currently don't keep track of which type of parentheses we're using on the left hand side.

mahrud · 2025-05-01T15:50:34Z

Okay, I'm totally fine with {a,b,c} = {1,2,3} not working. After all the output is not a list, but the individual symbols changed. However, we need to be careful of what is the result of L = (a,b,c) = {1,2,3}. A list or a sequence?

d-torrance · 2025-05-01T16:18:28Z

I just pushed a proposal to allow other types of list on the right hand side. Currently, it always returns a sequence, since the easy way to implement this is just to extract the sequence from the list and use it.

i1 : {x, y, z} = {1, 2, 3}

o1 = (1, 2, 3)

o1 : Sequence

We don't actually give any new examples for = and := since "multiple" assignment is documented before the other uses of these operators, so it would seem out of order. But we do mention that it's possible to use it.

mahrud · 2025-05-01T16:24:29Z

Thinking about this more, I think we want L = (a,b,c) = {1,2,3} to be a list.

d-torrance · 2025-05-01T16:36:23Z

Done!

i1 : {x, y, z} = {1, 2, 3}

o1 = {1, 2, 3}

o1 : List

If we've reached the assignment stage during x ??= y, we know that x is nullish (otherwise we would have returned it earlier). But we've been running "x = (x ?? y)", even though we know that the right hand side will be y in this case.

mahrud · 2025-07-11T20:13:50Z

@d-torrance once Paul's PR is merged, shall we merge this as well?

d-torrance · 2025-07-14T18:53:41Z

Sounds good to me!

mikestillman · 2025-07-15T15:20:45Z

oops, I missed the lat part of the conversation with my last question. I should try playing with #3599, and this one #3748, myself. Perhaps also with #3737?

d-torrance force-pushed the parallel-assignment branch 2 times, most recently from fae8011 to b06c309 Compare April 24, 2025 12:11

d-torrance added the Interpreter label Apr 24, 2025

d-torrance force-pushed the parallel-assignment branch 4 times, most recently from e104064 to 2ae336d Compare April 24, 2025 19:58

d-torrance added 9 commits April 29, 2025 20:58

Save symbol instead of only frame index in global/thread mem code

3cbc8ce

Then we can use this info for augmented/parallel assignment.

Remove now-unnecessary info member from augmentedAssignmentCode

5556bf0

This info is now saved in the lhs Code object

Update members of parallelAssignmentCode struct

24bab7f

Now contains a CodeSequence of the elements on the left hand side instead of symbol information. This will enable us to support other types of code in the left hand side.

Rewrite evaluation of parallel assignment

e86a205

The left hand side is now a sequence of Code objects rather than a list of symbols. Also add support for (x) = y (behaves just like x = y).

Add support for parallel assignment of other operators

b47af5a

e.g., assignment to mutable hash tables and lists and installing methods

Add augmented parallel assignment

fbe6760

Add unit tests for parallel assignment

4975655

d-torrance force-pushed the parallel-assignment branch from 2ae336d to fffd641 Compare April 30, 2025 01:20

d-torrance marked this pull request as ready for review April 30, 2025 01:20

d-torrance requested a review from mahrud April 30, 2025 01:20

mahrud reviewed Apr 30, 2025

View reviewed changes

bugs/dan/0-parallel-assignment Show resolved Hide resolved

d-torrance added the waiting for review label Apr 30, 2025

mahrud reviewed Apr 30, 2025

View reviewed changes

d-torrance force-pushed the parallel-assignment branch 2 times, most recently from bb0cc07 to 89b8108 Compare April 30, 2025 17:18

mahrud reviewed Apr 30, 2025

View reviewed changes

M2/Macaulay2/d/evaluate.d Outdated Show resolved Hide resolved

mahrud reviewed May 1, 2025

View reviewed changes

M2/Macaulay2/packages/Macaulay2Doc/operators/assignment.m2 Outdated Show resolved Hide resolved

mahrud reviewed May 1, 2025

View reviewed changes

d-torrance requested a review from pzinn May 1, 2025 12:26

d-torrance added this to the version 1.25.11 milestone May 1, 2025

d-torrance added 4 commits May 1, 2025 12:19

Document use of () for installing methods for ops w/ low precedence

e7d4812

Update parallel assignment docs

6f7a05d

We don't actually give any new examples for = and := since "multiple" assignment is documented before the other uses of these operators, so it would seem out of order. But we do mention that it's possible to use it.

Restore previous behavior of (x) = 1:y (equivalent to x = y again)

2f51db7

Parallel assignment error message updates

6c8dee4

d-torrance force-pushed the parallel-assignment branch from f2ea270 to a5a2c3b Compare May 1, 2025 16:20

Allow all types of list on the RHS of parallel assignment

8fc0a87

d-torrance force-pushed the parallel-assignment branch from a5a2c3b to 8fc0a87 Compare May 1, 2025 16:35

pzinn mentioned this pull request May 11, 2025

User-defined keywords #3599

Open

Improvements to parallel assignment #3748

Are you sure you want to change the base?

Improvements to parallel assignment #3748

Uh oh!

Conversation

d-torrance commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Parallel assignment for mutable hash tables and lists

Partial parallel assignment

Parallel method installation

Parallel augmented assignment

Fixes precedence issues with assignment

Uh oh!

mahrud commented Apr 23, 2025

Uh oh!

d-torrance commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mahrud commented Apr 25, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mahrud commented Apr 30, 2025

Uh oh!

d-torrance commented May 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mahrud left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d-torrance commented May 1, 2025

Uh oh!

mahrud commented May 1, 2025

Uh oh!

d-torrance commented May 1, 2025

Uh oh!

mahrud commented May 1, 2025

Uh oh!

d-torrance commented May 1, 2025

Uh oh!

mahrud commented May 1, 2025

Uh oh!

d-torrance commented May 1, 2025

Uh oh!

mahrud commented Jul 11, 2025

Uh oh!

d-torrance commented Jul 14, 2025

Uh oh!

mikestillman commented Jul 15, 2025

Uh oh!

Uh oh!

d-torrance commented Apr 23, 2025 •

edited

Loading

d-torrance commented Apr 24, 2025 •

edited

Loading

mahrud left a comment •

edited

Loading