Automatic state change detection (#774)

* Add automatic state change detection with ContractState wrapper

- Wrap contract state access in ContractState<T, idx> template with .get() (const) and .mut() (marks dirty) accessors
- Move all contract state fields into nested struct StateData for each contract
- Add needsCleanup() const method to containers
- Add using declaration to unhide const operator() overload in QpiContextProcedureCall
- Use .get() instead of .mut() for const proposal method calls
- Update contracts.md: describe StateData, state.get()/state.mut() accessors,
  and dirty state digest recomputation at end of tick
- Update contracts_proposals.md: update all code examples to use
  state.get().proposals / state.mut().proposals
- Update README.md: mention StateData and accessor pattern
- Update EmptyTemplate.h: add empty StateData with usage comments

Author: fnordspace

doc/contracts.md

@@ -26,12 +26,13 @@ A contract also never gets access to uninitialized memory (all memory is initial
 
 Each contract is implemented in one C++ header file in the directory `src/contracts`.
 
-A contract has a state struct, containing all its data as member variables.
+A contract is defined as a struct inheriting from `ContractBase`. All persistent data of the contract must be declared inside a nested `struct StateData`.
 The memory available to the contract is allocated statically, but extending the state will be possible between epochs through special `EXPAND` events.
 
-The state struct also includes the procedures and functions of the contract, which have to be defined using special macros such as `PUBLIC_PROCEDURE()`, `PRIVATE_FUNCTION()`, or `BEGIN_EPOCH()`.
-Functions cannot modify the state, but they are useful to query information with the network message `RequestContractFunction`.
-Procedures can modify the state and are either invoked by special transactions (user procedures) or internal core events (system procedures).
+The contract struct also includes the procedures and functions of the contract, which have to be defined using special macros such as `PUBLIC_PROCEDURE()`, `PRIVATE_FUNCTION()`, or `BEGIN_EPOCH()`.
+The state is accessed through a `ContractState` wrapper named `state`: use `state.get()` for read-only access and `state.mut()` for write access. Calling `state.mut()` marks the state as dirty for automatic change detection. The digest of dirty contract states is recomputed at the end of each tick, so that only contracts whose state actually changed incur the cost of rehashing.
+Functions can only read the state (via `state.get()`), making them useful to query information with the network message `RequestContractFunction`.
+Procedures can modify the state (via `state.mut()`) and are either invoked by special transactions (user procedures) or internal core events (system procedures).
 
 Contract developers should be aware of the following parts of the Qubic protocol that are not implemented yet in the core:
 - Execution of contract procedures will cost fees that will be paid from its contract fee reserve.
@@ -163,7 +164,7 @@ In order to make the function available for external requests through the `Reque
 `[INPUT_TYPE]` is an integer greater or equal to 1 and less or equal to 65535, which identifies the function to call in the `RequestContractFunction` message (`inputType` member).
 If the `inputSize` member in `RequestContractFunction` does not match `sizeof([NAME]_input)`, the input data is either cut off or padded with zeros.
 
-The contract state is passed to the function as a const reference named `state`.
+The contract state is accessed through a `ContractState` wrapper named `state`. In functions, only `state.get()` is available, returning a const reference to the `StateData`. For example, `state.get().myField` reads a field from the state.
 
 Use the macro with the postfix `_WITH_LOCALS` if the function needs local variables, because (1) the contract state cannot be modified within contract functions and (2) creating local variables / objects on the regular function call stack is forbidden.
 With these macros, you have to define the struct `[NAME]_locals`.
@@ -201,8 +202,7 @@ In order to make the function available for invocation by transactions, you need
 `REGISTER_USER_PROCEDURE` has its own set of input types, so the same input type number may be used for both `REGISTER_USER_PROCEDURE` and `REGISTER_USER_FUNCTION` (for example there may be one function with input type 1 and one procedure with input type 1).
 If the `inputSize` member in `Transaction` does not match `sizeof([NAME]_input)`, the input data is either cut off or padded with zeros.
 
-The contract state is passed to the procedure as a reference named `state`.
-And it can be modified (in contrast to contract functions).
+The contract state is accessed through a `ContractState` wrapper named `state`. In procedures, both `state.get()` (read-only) and `state.mut()` (read-write) are available. Use `state.mut()` when modifying state fields, e.g. `state.mut().myField = newValue`. Calling `state.mut()` marks the state as dirty so that its digest is recomputed at the end of the tick.
 
 Use the macro with the postfix `_WITH_LOCALS` if the procedure needs local variables, because creating local variables / objects on the regular function call stack is forbidden.
 With these macros, you have to define the struct `[NAME]_locals`.
@@ -252,8 +252,7 @@ System procedures 1 to 5 have no input and output.
 The input and output of system procedures 6 to 9 are discussed in the section about [management rights transfer](#management-rights-transfer).
 The system procedure 11 and 12 are discussed in the section about [contracts as shareholder of other contracts](contracts_proposals.md#contracts-as-shareholders-of-other-contracts)
 
-The contract state is passed to each of the procedures as a reference named `state`.
-And it can be modified (in contrast to contract functions).
+The contract state is accessed through a `ContractState` wrapper named `state`, the same as in user procedures. Use `state.get()` for reading and `state.mut()` for modifying state fields.
 
 For each of the macros above, there is a variant with the postfix `_WITH_LOCALS`.
 These can be used, if the procedure needs local variables, because creating local variables / objects on the regular function call stack is forbidden.
@@ -572,7 +571,7 @@ https://github.com/qubic/core/issues/574
 
 It is prohibited to locally instantiate objects or variables on the function call stack. This includes loop index variables `for (int i = 0; ...)`.
 Instead, use the function and procedure definition macros with the postfix `_WITH_LOCALS` (see above).  
-In procedures you alternatively may store temporary variables permanently as members of the state.
+In procedures you alternatively may store temporary variables permanently as members of `StateData` (accessed via `state.mut()`).
 
 Defining, casting, and dereferencing pointers is forbidden.
 The character `*` is only allowed in the context of multiplication.

doc/contracts_proposals.md

@@ -10,7 +10,7 @@ There are some general characteristics of the proposal voting:
 - Each proposal has a type and some types of proposals commonly trigger action (such as setting a contract state variable or transferring QUs to another entity) after the end of the epoch if the proposal is accepted by getting enough votes.
 - The proposer entity can have at most one proposal at a time. Setting a new proposal with the same seed will overwrite the previous one.
 - Number of simultaneous proposals per epoch is limited as configured by the contract. The data structures storing the proposal and voting state are stored as a part of the contract state.
-- In this data storage, commonly named `state.proposals`, and the function/procedure interface, each proposal is identified by a proposal index.
+- In this data storage, commonly named `proposals` (a member of `StateData`, accessed via `state.get().proposals` or `state.mut().proposals`), and the function/procedure interface, each proposal is identified by a proposal index.
 - The types of proposals that are allowed are restricted as configured by the contract.
 - The common types of proposals have a predefined set of options that the voters can vote for. Option 0 is always "no change".
 - Each vote, which is connected to each voter entity, can have a value (most commonly an option index) or `NO_VOTE_VALUE` (which means abstaining).
@@ -63,10 +63,11 @@ Features:
 
 If you need more than these features, go through the following steps anyway and continue reading the section about understanding the shareholder voting implementation.
 
-#### 1. Setup proposal storage
+#### 1. Setup proposal types and storage
 
-First, you need to add the proposal storage to your contract state.
-You can easily do this using the QPI macro `DEFINE_SHAREHOLDER_PROPOSAL_STORAGE(numProposalSlots, assetName)`.
+First, you need to add the proposal types and storage to your contract.
+Use the QPI macro `DEFINE_SHAREHOLDER_PROPOSAL_TYPES(numProposalSlots, assetName)` to define the required types,
+and declare the `proposals` field manually inside your `StateData`.
 With the yes/no shareholder proposals supported by this macro, each proposal slot occupies 22144 Bytes of state memory.
 The number of proposal slots limits how many proposals can be open for voting simultaneously.
 The `assetName` that you have to pass as the second argument is the `uint64` representation of your contract's 7-character asset name.
@@ -80,20 +81,26 @@ std::cout << assetNameFromString("QUTIL") << std::endl;
 Replace "QUTIL" by your contract's asset name as given in `contractDescriptions` in `src/contact_core/contract_def.h`.
 You will get an integer that we recommend to assign to a `constexpr uint64` with a name following the scheme `QUTIL_CONTRACT_ASSET_NAME`.
 
-When you have decided about the number of proposal slots and found out the the asset name, you can define the proposal storage similarly to this example taken from the contract QUTIL:
+When you have decided about the number of proposal slots and found out the the asset name, you can define the proposal types and storage similarly to this example taken from the contract QUTIL:
 
 ```C++
 struct QUTIL
 {
-    // other state variables ...
+    DEFINE_SHAREHOLDER_PROPOSAL_TYPES(8, QUTIL_CONTRACT_ASSET_NAME);
 
-    DEFINE_SHAREHOLDER_PROPOSAL_STORAGE(8, QUTIL_CONTRACT_ASSET_NAME);
+    struct StateData
+    {
+        // other state variables ...
 
-    // ...
+        ProposalVotingT proposals;
+
+        // ...
+    };
 };
 ```
 
-`DEFINE_SHAREHOLDER_PROPOSAL_STORAGE` defines a state object `state.proposals` and the types `ProposalDataT`, `ProposersAndVotersT`, and `ProposalVotingT`.
+`DEFINE_SHAREHOLDER_PROPOSAL_TYPES` defines the types `ProposalDataT`, `ProposersAndVotersT`, and `ProposalVotingT`.
+The `proposals` field of type `ProposalVotingT` must be declared manually inside `StateData`.
 Make sure to have no name clashes with these.
 Using other names isn't possible if you want to benefit from the QPI macros for simplifying the implementation.
 
@@ -123,19 +130,19 @@ struct QUTIL
         switch (input.proposal.data.variableOptions.variable)
         {
         case 0:
-            state.smt1InvocationFee = input.acceptedValue;
+            state.mut().smt1InvocationFee = input.acceptedValue;
             break;
         case 1:
-            state.pollCreationFee = input.acceptedValue;
+            state.mut().pollCreationFee = input.acceptedValue;
             break;
         case 2:
-            state.pollVoteFee = input.acceptedValue;
+            state.mut().pollVoteFee = input.acceptedValue;
             break;
         case 3:
-            state.distributeQuToShareholderFeePerShareholder = input.acceptedValue;
+            state.mut().distributeQuToShareholderFeePerShareholder = input.acceptedValue;
             break;
         case 4:
-            state.shareholderProposalFee = input.acceptedValue;
+            state.mut().shareholderProposalFee = input.acceptedValue;
             break;
         }
     }
@@ -175,7 +182,7 @@ struct QUTIL
 {
     // ...
 
-    IMPLEMENT_DEFAULT_SHAREHOLDER_PROPOSAL_VOTING(5, state.shareholderProposalFee)
+    IMPLEMENT_DEFAULT_SHAREHOLDER_PROPOSAL_VOTING(5, state.get().shareholderProposalFee)
 
     // ...
 }
@@ -231,7 +238,7 @@ The following elements are required to support shareholder proposals and voting:
 - `GetShareholderVotes`: Function for getting the votes of a shareholder. Usually shouldn't require a custom implementation.
 - `GetShareholderVotingResults`: Function for getting the vote results summary. Usually doesn't require a custom implementation.
 - `SET_SHAREHOLDER_PROPOSAL` and `SET_SHAREHOLDER_VOTES`: These are notification procedures required to handle voting of other contracts that are shareholder of your contract. They usually just invoke `SetShareholderProposal` or `SetShareholderVote`, respectively.
-- Proposal data storage and types: The default implementations expect the object `state.proposals` and the types `ProposalDataT`, `ProposersAndVotersT`, and `ProposalVotingT`, which can be defined via `DEFINE_SHAREHOLDER_PROPOSAL_STORAGE` in some cases.
+- Proposal types and storage: The default implementations expect the types `ProposalDataT`, `ProposersAndVotersT`, and `ProposalVotingT` (defined via `DEFINE_SHAREHOLDER_PROPOSAL_TYPES`) and the object `proposals` of type `ProposalVotingT` in `StateData` (accessed via `state.get().proposals` or `state.mut().proposals`).
 
 QPI provides default implementations through several macros, as used in the [Introduction to Shareholder Proposals](#introduction-to-shareholder-proposals).
 The following tables gives an overview about when the macros can be used.
@@ -244,7 +251,7 @@ Finally, multi-variable proposals change more than one variable if accepted. The
 
 Default implementation can be used?             | 1-var yes/no | 1-var N option | 1-var scalar | multi-var
 ------------------------------------------------|--------------|----------------|--------------|-----------
-`DEFINE_SHAREHOLDER_PROPOSAL_STORAGE`           | X            |                |              | X
+`DEFINE_SHAREHOLDER_PROPOSAL_TYPES`             | X            |                |              | X
 `IMPLEMENT_FinalizeShareholderStateVarProposals`| X            |                |              |
 `IMPLEMENT_SetShareholderProposal`              | X            |                |              |
 `IMPLEMENT_GetShareholderProposal`              | X            | X              | X            |
@@ -265,15 +272,23 @@ You may also have a look into the example contracts given in the table.
 
 #### Proposal types and storage
 
-The default implementation of `DEFINE_SHAREHOLDER_PROPOSAL_STORAGE(assetNameInt64, numProposalSlots)` is defined as follows:
+The default implementation of `DEFINE_SHAREHOLDER_PROPOSAL_TYPES(numProposalSlots, assetNameInt64)` is defined as follows:
 
 ```C++
     public:
         typedef ProposalDataYesNo ProposalDataT;
         typedef ProposalAndVotingByShareholders<numProposalSlots, assetNameInt64> ProposersAndVotersT;
         typedef ProposalVoting<ProposersAndVotersT, ProposalDataT> ProposalVotingT;
-    protected:
+```
+
+The `proposals` field must be declared manually inside `StateData`:
+
+```C++
+    struct StateData
+    {
         ProposalVotingT proposals;
+        // ...
+    };
 ```
 
 With `ProposalDataT` your have the following options:
@@ -285,7 +300,7 @@ With `ProposalDataT` your have the following options:
 The number of proposal slots linearly scales the storage and digest compute requirements. So we recommend to use a quite low number here, similar to the number of variables that can be set in your state.
 
 `ProposalVotingT` combines `ProposersAndVotersT` and `ProposalDataT` into the class used for storing all proposal and voting data.
-It is instantiated as `state.proposals`.
+It is instantiated as `proposals` inside `StateData` (accessed via `state.get().proposals` for reads and `state.mut().proposals` for writes).
 
 In order to support MultiVariables proposals that allow to change multiple variables in a single proposal, the variable values need to be stored separately, for example in an array of `numProposalSlots` structs, one for each potential proposal.
 See the contract TestExampleA to see how to support multi-variable proposals.
@@ -330,9 +345,9 @@ PRIVATE_PROCEDURE_WITH_LOCALS(FinalizeShareholderStateVarProposals)
     // Analyze proposal results and set variables:
     // Iterate all proposals that were open for voting in this epoch ...
     locals.p.proposalIndex = -1;
-    while ((locals.p.proposalIndex = qpi(state.proposals).nextProposalIndex(locals.p.proposalIndex, qpi.epoch())) >= 0)
+    while ((locals.p.proposalIndex = qpi(state.get().proposals).nextProposalIndex(locals.p.proposalIndex, qpi.epoch())) >= 0)
     {
-        if (!qpi(state.proposals).getProposal(locals.p.proposalIndex, locals.p.proposal))
+        if (!qpi(state.get().proposals).getProposal(locals.p.proposalIndex, locals.p.proposal))
             continue;
 
         locals.proposalClass = ProposalTypes::cls(locals.p.proposal.type);
@@ -341,7 +356,7 @@ PRIVATE_PROCEDURE_WITH_LOCALS(FinalizeShareholderStateVarProposals)
         if (locals.proposalClass == ProposalTypes::Class::Variable || locals.proposalClass == ProposalTypes::Class::MultiVariables)
         {
             // Get voting results and check if conditions for proposal acceptance are met
-            if (!qpi(state.proposals).getVotingSummary(locals.p.proposalIndex, locals.p.results))
+            if (!qpi(state.get().proposals).getVotingSummary(locals.p.proposalIndex, locals.p.results))
                 continue;
 
             locals.p.acceptedOption = locals.p.results.getAcceptedOption();
@@ -386,7 +401,7 @@ PUBLIC_PROCEDURE(SetShareholderProposal)
     }
 
     // try to set proposal (checks invocator's rights and general validity of input proposal), returns proposal index
-    output = qpi(state.proposals).setProposal(qpi.invocator(), input);
+    output = qpi(state.mut().proposals).setProposal(qpi.invocator(), input);
     if (output == INVALID_PROPOSAL_INDEX)
     {
         // error -> reimburse invocation reward
@@ -423,8 +438,8 @@ struct GetShareholderProposal_output
 PUBLIC_FUNCTION(GetShareholderProposal)
 {
     // On error, output.proposal.type is set to 0
-    output.proposerPubicKey = qpi(state.proposals).proposerId(input.proposalIndex);
-    qpi(state.proposals).getProposal(input.proposalIndex, output.proposal);
+    output.proposerPubicKey = qpi(state.get().proposals).proposerId(input.proposalIndex);
+    qpi(state.get().proposals).getProposal(input.proposalIndex, output.proposal);
 }
 ```
 
@@ -452,7 +467,7 @@ PUBLIC_FUNCTION(GetShareholderProposalIndices)
     {
         // Return proposals that are open for voting in current epoch
         // (output is initialized with zeros by contract system)
-        while ((input.prevProposalIndex = qpi(state.proposals).nextProposalIndex(input.prevProposalIndex, qpi.epoch())) >= 0)
+        while ((input.prevProposalIndex = qpi(state.get().proposals).nextProposalIndex(input.prevProposalIndex, qpi.epoch())) >= 0)
         {
             output.indices.set(output.numOfIndices, input.prevProposalIndex);
             ++output.numOfIndices;
@@ -465,7 +480,7 @@ PUBLIC_FUNCTION(GetShareholderProposalIndices)
     {
         // Return proposals of previous epochs not overwritten yet
         // (output is initialized with zeros by contract system)
-        while ((input.prevProposalIndex = qpi(state.proposals).nextFinishedProposalIndex(input.prevProposalIndex)) >= 0)
+        while ((input.prevProposalIndex = qpi(state.get().proposals).nextFinishedProposalIndex(input.prevProposalIndex)) >= 0)
         {
             output.indices.set(output.numOfIndices, input.prevProposalIndex);
             ++output.numOfIndices;
@@ -511,7 +526,7 @@ typedef bit SetShareholderVotes_output;
 
 PUBLIC_PROCEDURE(SetShareholderVotes)
 {
-    output = qpi(state.proposals).vote(qpi.invocator(), input);
+    output = qpi(state.mut().proposals).vote(qpi.invocator(), input);
 }
 ```
 
@@ -530,7 +545,7 @@ typedef ProposalMultiVoteDataV1 GetShareholderVotes_output;
 PUBLIC_FUNCTION(GetShareholderVotes)
 {
     // On error, output.votes.proposalType is set to 0
-    qpi(state.proposals).getVotes(input.proposalIndex, input.voter, output);
+    qpi(state.get().proposals).getVotes(input.proposalIndex, input.voter, output);
 }
 ```
 
@@ -548,7 +563,7 @@ typedef ProposalSummarizedVotingDataV1 GetShareholderVotingResults_output;
 PUBLIC_FUNCTION(GetShareholderVotingResults)
 {
     // On error, output.totalVotesAuthorized is set to 0
-    qpi(state.proposals).getVotingSummary(
+    qpi(state.get().proposals).getVotingSummary(
         input.proposalIndex, output);
 }
 ```