Ledger rule changes: documentation & consistency?

[waalge]

Someone sent me a message to inform me that a dapp what I wrote no longer works.
The breakage is due to a change in ledger rules.
Specifically the one corresponding to https://input-output-hk.github.io/cardano-js-sdk/enums/_cardano_sdk_core.TxSubmissionErrorCode.html#ConflictingInputsAndReferences

Essentially the dapp involves a CIP 68 style "updateable" state, that doubles as linked list lexicographically sorted by token name.
To insert a new item in the list, you "update" the token that will precede it.

The initial entry of the list, has blank token name (excluding CIP67 tag) .
This entry's datum contains information that controls additional behaviour of how a new item can be inserted into the list.
As a result, for any insert, the validator expects to find this token in reference inputs.

Under the change to ledger rules, this breaks for the first insert.
In fact, for every insert in which the first token is spent.

There are numerous elementary ways the dapp can be modified adhere to the new ledger rules.
But had the dapp already been deployed, that would be irrelevant.

Reading from the distance, I'm guessing significant efforts have been made to futureproof Plutus.
Its kinda all for not so much if Ledger rules can just break your dapp anyway.
I frequently make assumptions on ledger rules.
For example,

• MinAda was no longer required, then a bunch of stuff I've written would no longer work, such as

expect [_ , (hash, name, amount) ] = value |> assets.flatten

• Inputs are lexicographically sorted. (An example utilising this is a bit more involved)

It is very helpful relying on things being enforced by the ledger. And very unhelpful when these change.
From memory this is not the first time this is has happened.

As far as I know, there is no "Ledger rules we promise you can assume" list out there. Is there?

[rphair]

@lehins @WhatisRT I've looked over CIP-0084 again to see if "guarantees" about making & announcing future changes might have been anticipated and documented.

According to my own (outsider's) reading & interpretation of What merits a ledger CIP? a CIP needs to be published every time the Ledger implementation changes.

My understanding of the technical issue above isn't enough to know if it was caused by an implementation change, a bug fix, or a bug. But in response to the OP's request for the CIP process:

• Is there a source of information that developers could use to help avoid the situation @waalge is talking about?
• Could we update and/or prioritise CIP-0084 somehow to pre-empt this kind of problem? (cc @Ryun1 @Crypto2099 @perturbing)

[WhatisRT]

We've had some vague plans for a list of things that we guarantee forever, but it never really materialized because people are just too busy to work on it when we get one ask per year or less for this type of thing. It's a shame, but there isn't all that much we can do about it, it's a significant amount of work to figure out whether we can make guarantees like that with an unknown (and unknowable) future, and the guarantees we could give might be incredibly weak.

However, that doesn't mean that we couldn't try to do better. I've been in favour of slowing down the rate of change of the ledger and making the process more public for a while now, which might have helped here. Maybe it wouldn't have prevented this issue directly, but it would have been nice if there was a public forum available.

I believe this change was only discussed internally and then implemented. With small changes (which this one is) it is a relatively big overhead to write a CIP. We often have a couple of small changes like this on every HF that just clean a few things up. I don't really know what the best process here would be. Maybe a 'catchall' CIP that collects all of these things?

We're going to need some kind of process like this in the not too distant future anyway, since these now need to be consistent across multiple node implementations. There is going to be a multi-node conformance test suite, which will require manual intervention for logic changes like these, so maybe there is a future where we have a protocol that looks something like this: When we make small tweaks in the logic, we add a short description to the catchall CIP and add relevant tests. I think there being two items to do makes it less likely that this is accidentally forgotten.

[rphair]

Maybe a 'catchall' CIP that collects all of these things?

@WhatisRT I've marked this for Triage in the next meeting to air the issue & ask if the other editors would favour a CIP to collect these changes as additional sections to a "catch-all" Specification section... probably also with a Changelog since the change announcements & text updates wouldn't generally be coincident (?). I'd be available to help you write & submit that shell of a CIP with whatever such things you have on the boards so far.

[WhatisRT]

I think for the next HF we're still too early. Maybe other people already have some small tweaks they want to do, but I think for now we mostly have very high-level ideas. But maybe it would be good practice to put an empty skeleton in place just so that people know it's there. I guess we could also do such a CIP for Conway but I'm not sure if that's worth the effort.

[rphair]

thanks @WhatisRT - I'll post here when we have some feedback from the dev community at the meeting (not for about 3 weeks); https://hackmd.io/@cip-editors/111 ... and will also post any developer requests that come in on Discord where this issue was originally raised.

[ch1bo]

As far as I know, there is no "Ledger rules we promise you can assume" list out there. Is there?

@waalge The ledger specs should be, but likely they are incomplete and probably they are not approachable enough to really benefit from "more eyes on the prize". I have been in various conversations of making the ledger test suites more accessible to other implementations in course of cardano-blueprint (also related IntersectMBO/cardano-ledger#4892) and more often than not those conversations gravitate away from formal methods and identify a good old "list of test vectors" as a more accessible way about creating (and maintaining!) a correct ledger implementation.

[waalge]

Im not sure what the coherent ask is here.
Something like

is there a way that on a ledger change, the new ledger logic reruns all existing txs to see if they still work?

Naively, this would probably not be insightful. On increasing fees, say, almost all previous txs would fail. Thered need to be something more.
Im hazy on what apparatus is involved in the process of changing ledger rules. Is there already some

$$Fix : txs --> txs$$

Best guess of fixing txs obeying old ledger rules to new ledger rules.

Then we could spot the dapps that potentially will stop working.

In this case, Fix may have dropped a ref input in case of inclusion in inputs.

[waalge]

@ch1bo please correct me. My parsing of what you said is really concerned with understanding what ledger rules are currently. Or maybe also in the near future. Very welcome.

But its another thing to "behaviour we promise not to change, like, ever"

[rphair]

Further discussion presented here, particularly how such process improvements would be desirable (perhaps required) for "node diversity": https://forum.cardano.org/t/managing-a-shared-ledger-new-cip-like-process/145575 cc @yHSJ

[WhatisRT]

@waalge if it's not a problem, could you provide the script that was bricked? It might help with finding the best solution here. Any documentation would also be helpful. If it's not public you could send it to me via email, it's on my profile.

[ch1bo]

@ch1bo please correct me. My parsing of what you said is really concerned with understanding what ledger rules are currently. Or maybe also in the near future. Very welcome.

@waalge I wrote up my thoughts on the forum thread. Sorry for pulling things between here and there, but could not resist the excellent question posted by @yHSJ there - also, we want to revive the forum a bit :)