Present clearer diff fo single element changes to sets

[lukehoban]

Currently, when a single item in a set has a diff, we show every item of the set having changed places.

For example - in the diff below (from pulumi/pulumi-aws#920) - there is one element actually changing (IamInstanceProfile's value is changing from an id to an ARN) - but all other elements are getting mixed up:

Previewing update (cameron):
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:cameron::beanstalk-env-order-bug::pulumi:pulumi:Stack::beanstalk-env-order-bug-cameron]
    ~ aws:elasticbeanstalk/environment:Environment: (update)
        [id=e-mvzjv6gg2f]
        [urn=urn:pulumi:cameron::beanstalk-env-order-bug::aws:elasticbeanstalk/environment:Environment::beanstalk-env-order-bug-cameron]
        [provider=urn:pulumi:cameron::beanstalk-env-order-bug::pulumi:providers:aws::default_1_27_0::6d629690-746c-4946-b638-718881ccf3f6]
      ~ settings: [
          ~ [0]: {
                  ~ name     : "VPCId" => "VPCId"
                  ~ namespace: "aws:ec2:vpc" => "aws:ec2:vpc"
                  - resource : ""
                  ~ value    : "vpc-0a79acb18cf59054f" => "vpc-0a79acb18cf59054f"
                }
          ~ [1]: {
                  ~ name     : "MaxSize" => "Subnets"
                  ~ namespace: "aws:autoscaling:asg" => "aws:ec2:vpc"
                  ~ value    : "1" => "subnet-037fba84fc1fd4664"
                }
          ~ [2]: {
                  ~ name     : "setting1" => "IamInstanceProfile"
                  ~ namespace: "aws:elasticbeanstalk:application:environment" => "aws:autoscaling:launchconfiguration"
                  ~ value    : "value1" => "arn:aws:iam::052848974346:instance-profile/beanstalk-env-order-bug-cameron-617dfaf"
                }
          ~ [3]: {
                  ~ name     : "MinSize" => "InstanceType"
                  ~ namespace: "aws:autoscaling:asg" => "aws:autoscaling:launchconfiguration"
                  ~ value    : "1" => "t3.medium"
                }
          ~ [4]: {
                  ~ name     : "IamInstanceProfile" => "MinSize"
                  ~ namespace: "aws:autoscaling:launchconfiguration" => "aws:autoscaling:asg"
                  ~ value    : "beanstalk-env-order-bug-cameron-617dfaf" => "1"
                }
          ~ [5]: {
                  ~ name     : "setting2" => "MaxSize"
                  ~ namespace: "aws:elasticbeanstalk:application:environment" => "aws:autoscaling:asg"
                  ~ value    : "value2" => "1"
                }
          ~ [6]: {
                  ~ name     : "Subnets" => "setting1"
                  ~ namespace: "aws:ec2:vpc" => "aws:elasticbeanstalk:application:environment"
                  ~ value    : "subnet-037fba84fc1fd4664" => "value1"
                }
          ~ [7]: {
                  ~ name     : "InstanceType" => "setting2"
                  ~ namespace: "aws:autoscaling:launchconfiguration" => "aws:elasticbeanstalk:application:environment"
                  ~ value    : "t3.medium" => "value2"
                }
          ~ [8]: {
                  ~ name     : "setting3" => "setting3"
                  ~ namespace: "aws:elasticbeanstalk:application:environment" => "aws:elasticbeanstalk:application:environment"
                  ~ value    : "value3" => "value3"
                }
        ]

[t0yv0]

Another user affected: pulumi/pulumi-aws#2856

Also, there is another related issue here #1756 that goes beyond just presentation, currently the user is not able to use ignoreChanges to do-what-I-mean over such set element changes.

[t0yv0]

Another repro of this with some debug details on why it's happening. #1894 - I think this is exactly the problem in pulumi/pulumi-aws#2095

[t0yv0]

Thinking about possible solutions here; one option would be to extend the CLI with special support in the detailedDiff to express "element added" vs "element removed" vs "element updated" changes irrespective of numeric index in a set.

If we do not change the CLI interface, I am thinking we could still make progress if we drop the assumption that the numeric indices [0..3] mean anything related to the program positions.

This would work as follows. Suppose we are changing from set S1: set<T> to set S2: set<T> with set element identity function f: T -> K. The diff between S1 and S2 will have:

type diff = array<changes<T>>
type change<T> = added(T) | removed(T) | changed(T, T)

The invariant is that changes only happen at the same identity:

changed x y ==> f(x) == f(y)

Then we can sort changes by f. This gives an canonical order and allocates integers to every change - these integers no longer line up with element positions in the source program before and after the change, but since Pulumi prefers sorting by f already this "almost lines up", the advantage being that there is no mixup happening if the identity function f is correctly specified: changing an element identity will have one removed entry and one added entry, and a changed entry will always correspond to an element that preserves its identity.

Unfortunately I still don't have it worked out how to naturally interpret ignoreChanges in the setting #1756 - and a complete proposal should have an answer for that. The key difficulty is that ignoring changes may be affecting element identity.