C# Dirty/Versioned Records

[wilbennett]

Now that C# 9.0 has support for records, I believe including "dirty" and version/revision support would be greatly beneficial.

Whenever records are constructed, either by calling the constructor or using object initialization, an implicit "dirty" flag should be set to false. Any instance created through the use of the "with" operator, should have its "dirty" flag set to true.

A natural extension to this is to have an implicit version/revision number.

Whenever records are constructed, either by calling the constructor or using object initialization, an implicit "version/revision" field/property should be set to 1/0. Any instance created through the use of the "with" operator, should have its "version/revision" field/property incremented.

Consider:

record Person(string firstName, string lastName);

void Main()
{
    var person = new Person("John", "Mckenzie"); // Implicit version set to 1.
    var newPerson = CheckCapitalization(person);

    if (version newPerson > version person)
        person = newPerson;
}

Person CheckCapitalization(Person person)
{
  return person.LastName.StartsWith("Mck")
     ? person with { LastName: "McK" + person.LastName.Substring(3) } // Inc version
     : person;
}

Of course, this is a trivial example that is just meant to show the concept. The real benefit is when used in a more elaborate scenario or in async contexts (e.g. Rosyln or Entity Framework).

[theunrepentantgeek]

I believe the real problem with this proposal comes with the detail - there are too many edge cases where different teams/projects/developers will want different behaviour.

For just one example, if I reset the value of a property back to the original value, should the instance be considered dirty?

I can make a disciplined logical argument that it should be considered dirty.
I can make an equally compelling argument that it should not be.
And that's the problem.

[wilbennett]

I think it doesn't make sense to introduce something that heavily depends on the context into all contexts, which is what you're proposing.

I think you misunderstand what I'm saying. I'm saying the code you write to do the update is contextual. The language only does increments regardless of the situation.

This is completely unlike anything else in C#. Assignment is just that: an assignment, users don't expect it to silently allocate object that is different from the original.

I'm not suggesting allocating new objects. The version number would be a number in memory before the actual record data, just like the size is prepended to some memory in C++ or the length of a string prepended before the string in Delphi/Object Pascal.

As far as being a niche problem, I disagree. I look at this the same as state machines. Developers, including myself, rarely implement explicit state machines for problems which would benefit by it. Instead, a lot of code is created that ends up implicitly implementing a state machine anyway. I think if you take a step back and look objectively, you will realize that most code you've written that has a collection of object instances or compositions of object instances (especially multithreaded) are doing implicit versioning (though it may not be explicitly called versioning) - only with much more code than this proposal suggests. I know I've been doing it!

Records themselves are an example of similar phenomena. I would venture to guess that a lot of developers will be using records now. They could have always created immutable objects and simple DTOs before but it was a big pain so they didn't do it. Adding this versioning support I think will have similar effect - make the pattern more accessible, reduce code size and make code more readable.

[HaloFour]

@wilbennett

I'm not suggesting allocating new objects. The version number would be a number in memory before the actual record data, just like the size is prepended to some memory in C++ or the length of a string prepended before the string in Delphi/Object Pascal.

The only way that would be possible would be if the System.Object class was changed so that the version would be the first field. That would be a suggestion for the runtime repo, not the C# repo, as C# cannot make every object automatically have this "version" field. There's also nothing to handle incrementing a version on assignment, nor do I understand why that would be desirable.

As far as being a niche problem, I disagree.

Versioning as you describe here is pretty niche, and I would also suggest that the behavior that you're describing is certainly not the implementation of versioning most people would use. Even where in my projects where I use versioning it doesn't apply to the vast, vast, vast majority of the classes I write. And where it does apply it wouldn't make sense to increment it on every modification or assignment.

[wilbennett]

The only way that would be possible would be if the System.Object class was changed so that the version would be the first field.

This is definitely a deal-breaker if there is no workaround. I don't think adding to every object is desirable (I have a workaround in mind but I don't want you guys to start throwing tomatoes at me. lol).

Versioning as you describe here is pretty niche, and I would also suggest that the behavior that you're describing is certainly not the implementation of versioning most people would use.

Yeah, maybe a different name than version is needed. The point isn't versioning in the traditional sense. The point is to control updates. The increment on assignment is not really an increment per se - more of a reconciliation. I was just addressing the concern about allowing multiple updates without conflicts. In the same vein, the increment each time is about keeping the logic simple and accounting for changes - not about full fledged versioning.

I think it may be confusing because I called it versioning but honestly I don't know what else to call it. Just keep in mind it's about data fidelity and not traditional versioning.

[theunrepentantgeek]

@wilbennett, your response doesn't address my concerns in any way.

Let me restate them another way.

If the compiler is to implement versioning and/or dirty detection, every detail of the behaviour needs to be locked down by the language design team.

So far in this thread, we've identified the following details:

• Should assigning the original value clear the dirty flag?
  (You assert no, but to my mind, given that records implement value equality, the answer should be yes - but that dramatically increases the memory overhead.)

• Should assignment increment the version of the record?
  (Given value equality semantics, I believe it shouldn't, but I think you're saying it should. I may be wrong., but it's still a question to be resolved.)

• Should version updates be friendly in async/multi-threading scenarios.
  (I believe you assert yes, but I'm concerned about the overhead of carrying around all those locks, and about the possibility of untoward race conditions)

• Should versioning be a simple monotonically increasing int, or major/minor based? With what semantics for updates?

• Should the version be serialized along with the rest of the object?
  (A simple answer is yes, because that's what some/most existing serialization frameworks will do by default - but you assert that it shouldn't.)

You wrote:

Again, it all depends on the context.

This is the problem. Different contexts require different answers to the above concerns (and to others).

Locking in a very specific set of choices in the compiler might well suit your scenarios, but it sure won't suit most of mine.

Even if we agreed that versioning is needed in a majority of the situations where records will be used (*), a compiler implemented solution would only satisfy a small minority of those situations because of the policy decisions baked in by the language design team. All the other scenarios would still need to implement their policies manually.

(*) And I actually think that versioning is needed in only a vanishingly small set of situations.

[wilbennett]

Hi @theunrepentantgeek. I'll address your last concern first.

When I say context, I'm talking about code you, the developer writes not what the compiler does. What the compiler does is simple, it increments a number when you use record.with. Again, versioning may be the wrong name/term for this. The goal is simple: Allow updating a reference to an immutable structure while preventing stale updates.

This can definitely be done without help from the compiler. It just means it won't be standard and you'll have to learn how each project handles it.

With that goal in mind, hopefully my answers to your bullet points will make more sense now:

Should assigning the original value clear the dirty flag?
(You assert no, but to my mind, given that records implement value equality, the answer should be yes - but that dramatically increases the memory overhead.)

This is about update handling not full versioning. We always increment because there was a "change". There was a record.with call, there was an increment. If there needs to be additional checks, the dev is free to do so. The only responsibility of the compiler is to say "This particular instance is a result of x number of 'modifications' from the original". What you choose to do with that information is up to you. You can choose to do an equality comparison. You can choose to just do an update based on the version if your updates are idempotent. That's what I mean by context - the environment in which you choose how you handle updates.

Should assignment increment the version of the record?
(Given value equality semantics, I believe it shouldn't, but I think you're saying it should. I may be wrong., but it's still a question to be resolved.)

Assignment doesn't increment the version. I believe what you are referring to is my response to @svick about his concern of an updater doing multiple updates causing the version to advance multiple times. If you read that carefully, you will see that it is not about an increment on assignment as it is about rolling the minor version up to the major. The adjustment would only happen if the minor version is not 0. The intent being to allow multiple updates to "roll up" as one.

Should version updates be friendly in async/multi-threading scenarios.
(I believe you assert yes, but I'm concerned about the overhead of carrying around all those locks, and about the possibility of untoward race conditions)

There are no locks, no race conditions. We are dealing with immutable structures. The version of a new immutable record is being set to the version + 1 of another immutable record on construction.

Should versioning be a simple monotonically increasing int, or major/minor based? With what semantics for updates?

This has probably been answered by now. The initial proposal is for simple increasing. If allowing multiple updates by an updater is required, then a major/minor approach can be used as in my response to @svick. The caveat being that the "roll up" of the minor version is needed on assignment.

Should the version be serialized along with the rest of the object?
(A simple answer is yes, because that's what some/most existing serialization frameworks will do by default - but you assert that it shouldn't.)

Also probably answered by now. This isn't full fledged versioning so no. This is meant to be ephemeral and only as a means to "synchronize" updates.

Hope this helps. FYI: I'm leaning towards @HaloFour's suggestion of using source generators if this can't be implemented without adding the version int to every object instance.

[svick]

What's the advantage of the proposed code over just:

person = CheckCapitalization(person);

Or maybe (assuming the assignment is not trivial):

var newPerson = CheckCapitalization(person);

if (!ReferenceEquals(newPerson, person))
    person = newPerson;

Also, why should every single record pay the performance penalty for this, when it's likely that only a small minority of them will use this feature?

[wilbennett]

Thanks for your input svick!

The primary advantage is in multithreaded or async update scenarios. See my response to @theunrepentantgeek above for an example.

As for the performance penalty, you can consider the flag as an extra 32/64 bit integer field. When doing Record.with, this integer is incremented. Expense depends on context but I do not see incrementing a integer (only on Record.with calls) as a huge expense.

Regarding the amount of code that would use this feature. I could be wrong but I suspect that if this is implemented, it will be used all the time (especially in multithreaded scenarios). Why do I think so? The entire point of records is simplicity and immutability which aids in readability and use in multithreaded scenarios. Chances are if you are using records, you have a collection of them or they serve as data by composition. Versioning only comes into play when you are modifying the records which would mean updating collections or composition containers - most likely asynchronously for performance. Having versioning will help with avoiding unnecessary updates in many contexts (like Entity Framework).

Another cheesy example:

record RatTrap(bait: CheeseType; caughtRat: Rat);

• Asynchronously present the trap to a collection of rats
• Each rat who likes the cheese tries to update the trap: UpdateTrap(ratTrap with { caughtRat = this})

The logic of updating the trap is:

   UpdateTrap(newTrap: RatTrap)
      if (version newTrap > version this.Trap)
         this.Trap = newTrap

Note the simplicity of updating "this.Trap". It doesn't matter if it's being called by a single thread or from multiple threads. The first rat to update "wins". Of course, for multiple threads, this would have to be in a protected context (e.g. lock or processing queue).

[svick]

Expense depends on context but I do not see incrementing a integer (only on Record.with calls) as a huge expense.

Incrementing an integer is indeed not a significant cost when compared with allocating a new object. But I think that increasing the size of every record by 8 bytes is.

Having versioning will help with avoiding unnecessary updates in many contexts (like Entity Framework).

In that case, I would like to hear from someone who works on EF that they would actually use this feature.

Another cheesy example

Why wouldn't ReferenceEquals be sufficient here?

[wilbennett]

Why wouldn't ReferenceEquals be sufficient here?

For simplicity I'll use integers for the reference:

this.Trap - reference is 0
Rat 1 calls UpdateTrap - Succeeds because his reference is not 0
Rat 2 calls UpdateTrap - Succeeds because his reference is not 0
etc....

With ReferenceEquals, the test would always pass.

If you change the cheese in the trap, this reference will now change.
Present the new trap to the rats.
Slow rats from the previous call could overwrite the new trap!

[wilbennett]

In that case, I would like to hear from someone who works on EF that they would actually use this feature.

It would be built into the framework itself:

• You run a query and get back a collection of versioned records (all 0/1)
• You replace some records in the collection using record.with
• You call update and EF only updates the database with the higher versioned records

[wilbennett]

This solution obviously isn't a silver bullet. With this language support, however, following a simple guideline of each updater only performing a single record.with update makes writing this kind of code simple and easy to reason about. Try implementing this logic without this feature and you will see what I mean.

One thing to keep in mind is that all the language would be doing is incrementing version numbers. How you choose to use the version is up to you.

[HaloFour]

This seems like a feature better offered by a source generator, one that can be specifically provided by Entity Framework that will work well with the policies that make sense for Entity Framework.

[Versioned]
partial record Person(string firstName, string lastName);

[HaloFour]

My reservation with this is that if the compiler handles it, the implementation is free to change as technology evolves.

The compiler always handles it. There is no such thing as a copy constructor as far as the runtime is concerned. When you define a record the compiler is already emitting the code to manually copy the fields from the prototype instance.

All the examples I've given show the versioning having meaning only for the duration of the program run.

What are the use cases for something like that? I've written versioning into my software before, but it's only been used to facilitate concurrent updates across process boundaries, e.g. a web service in front of a database. And I only care to actually increment that version when successfully persisting the entity back to the database.

[wilbennett]

What are the use cases for something like that?

The Visual Studio IDE itself. When you are writing an analyzer that needs to make changes to the edited file, you first get a version of the file (Roslyn is immutable for the most part). You make your changes, which increments the version, and then try to write it back to the editor. If the user had typed something new or deleted some characters, the version of the file will have been incremented and your changes would not be applied since your version would no longer be newer.

Again, it's been a while since I worked on it so I may be wrong on the details or they may have changed. I'm pretty sure that's the way it worked when I was doing it though.

[CyrusNajmabadi]

So, I can say that for that purpose we would be highly unlikely to use a system like this. The logic we have here is actually more complex as we have to understand a branched tree-like structure for forking. It wouldn't be a simple 'dirty/version' flag.

And, indeed, you would expect that someone could make N changes to a solution and then apply that final Solution back to the workspace. This would be allowed even if the version number was N apart.

Sorry, but this really wouldn't be a use case here.

[wilbennett]

@HaloFour I'm leaning toward your suggestion if implementing this change means updating every object instance to have a "version".

[wilbennett]

Thanks @CyrusNajmabadi. I guess I misunderstood how it works then.

[CyrusNajmabadi]

First, let me say that i find the idea of having some sort of versioning concept very interesting. It's certainly something that arises in some domains (including ones that Roslyn plays in). However, i also think this is something extremely domain specific and fiddly. Different domains do this very differently and tehre are often many knobs you need to deal with something like this.

Because of that, i don't feel like this should be something baked into either hte language or runtime.

Instead, it would make a ton more sense for this to be brought in by a source generator that can be customized to the level that people will invariably need. Because practically all parts of a record can be implemented manually, it would be totally fine to let people declare a [Versioned] record that a special source generator would tehn spit out all this custom logic for.

So i strongly approve someone pursuing this space. However, i think the right place would not be in the language or runtime. Cheers!

[wilbennett]

Thanks for your input @CyrusNajmabadi. I have a feeling this will be the sentiment of a lot of developers. Hopefully some of the comments here will change your minds though :)

[CyrusNajmabadi]

So far, the comments have pushed me more strongly towards this view. Versioning is a highly domain specific concept. You've brought up roslyn as a consumer of this, but as I've pointed out, roslyn wouldn't be using this facility for this purpose.

I think there's merit here in people codifying different versioning schemes. However, that work really works be outside the language in something like EF, or some other lib/sg.

[wilbennett]

Thanks for the feedback guys. I guess we can close this out. I'll go with @HaloFour's suggestion.

Closing thought. Regarding the major/minor approach, I just realized that the "increment" on assignment is unnecessary. As long as the "version" keyword does the "roll up", everything works.
Oh and - hold the tomatoes please - the fix for not having to modify the runtime to add a version to every object is to add a compiler access only first "version" field to each record object. Doesn't the string object do this for length?

Thanks again and happy holidays.