Remove hidden data before validation and make clean data available in DataProcessor #1279

ivarne · 2025-04-29T13:29:50Z

Now all data is cleaned for all (backend) validation if AppSettings.RemoveHiddenData == true and IInstanceDataAccessor has a new method GetCleanDataAccessor so that a DataProcessor (or UserAction) can easily get a copy of the data with hidden data removed for calculations or similar.

One key factor here is that we add a source generator to be able to work with data models more efficiently

During self review I tried to split the PR into smaller commits that stands on its own

63018b3 Move Altinn.App.Analyzer from .Core to .Api
ecf795d Make changes for more efficient removal of hidden data
- Introduce IFormDataWrapper
- Deprecate DataModelWrapper
- New helpers in IInstanceDataAccessor
1f015ef Add soure generator for IFormDataWrapper implemenations in .Analyzers
This includes two test projects.
- SourceGenerator.Tests: Use the generator directly and verify text output
- SourceGenerator.Integration.Tests: Reference the generator in build process and test the generated code
This has to be two projects, because it is really hard to debug a source generator when it runs as part of the build process.
67d2df5 Add Clean and Previous InstanceDataAccessors
This enables DataProcessors to get the clean versions of data models for calculations.
df07e13 Clean data before validation if AppSettings.RemoveHiddenData = true
6589b33 Minor fixes

Related Issue(s)

Form data: Remove hidden values #1157

Verification

Your code builds clean without any errors or warnings
Manual testing done (required)
Relevant automated test added (if you find this hard, leave it and we'll help out)
All tests run green

Documentation

User documentation is updated with a separate linked PR in altinn-studio-docs. (if applicable)

ivarne · 2025-04-29T13:33:21Z

/publish

github-actions · 2025-04-29T13:33:40Z

PR release:

⚙️ Building...
✅ Done!

src/Altinn.App.Analyzers/SourceTextGenerator/AddIndexToPathGenerator.cs

src/Altinn.App.Analyzers/SourceTextGenerator/SourceReaderUtils.cs

src/Altinn.App.Analyzers/Utils/EquatableArray.cs

src/Altinn.App.Analyzers/Utils/NanoJsonReader.cs

src/Altinn.App.Analyzers/Utils/RoslynExtensions.cs

src/Altinn.App.Core/Features/IFormDataWrapper.cs

ivarne · 2025-04-29T20:43:04Z

/publish

github-actions · 2025-04-29T20:43:26Z

PR release:

⚙️ Building...
✅ Done!

src/Altinn.App.Analyzers/SourceTextGenerator/AltinnRowIdsGenerator.cs

ivarne · 2025-05-07T07:15:02Z

/publish

github-actions · 2025-05-07T07:15:23Z

PR release:

⚙️ Building...
✅ Done!

ivarne · 2025-05-13T07:04:57Z

/publish

github-actions · 2025-05-13T07:05:18Z

PR release:

⚙️ Building...
✅ Done!

src/Altinn.App.Analyzers/IncrementalGenerator/FormDataWrapperGenerator.cs

martinothamar · 2025-05-13T09:02:15Z

src/Altinn.App.Analyzers/IncrementalGenerator/FormDataWrapperGenerator.cs

+            {
+                if (dataType.Type != JsonType.Object)
+                {
+                    continue;


This branch could emit a diagnostic, all the values in dataTypes should be objects

Yes, I definitely could add more diagnostics. Not sure I want to do that, because diagnostics from incremental source generators is pretty buggy. I wanted to have a diagnostic for not finding "ClassRef" in compilation, and added some extra just because they were easy.

Reading up on the whole "diagnostics from incremental generators" topic, it seems like there is no intention to have a create story there. Maybe we should go in the opposite direction then? Drop all diagnostics here and rather implement analyzers if we feel it is necessary?

src/Altinn.App.Analyzers/IncrementalGenerator/FormDataWrapperGenerator.cs

martinothamar · 2025-05-13T09:18:47Z

src/Altinn.App.Analyzers/ModelPathNode.cs

+    /// <summary>
+    /// Initializes a new instance of the <see cref="ModelPathNode"/> class.
+    /// </summary>
+    public ModelPathNode(


It wasn't obvious to me why cSharpName and jsonName could be empty in some cases. That is for the root node right? Maybe we could have static named constructors like CreateRoot and CreateProperty or something. Type hierarchy/union pattern is also an option

That is for the root node right?

Yes, it is only for root.

I think the proper solution would be to create two different types. Node(List<Property>) and Property(cSharpName, jsonName, node).

(Yes, I checked and that is what PolyType does have two levels)

Do you think clarity is worth the rewrite?

I think it is worth the rewrite at some point. If we do that now as part of this PR or in the future I'll leave for you to decide 😄

src/Altinn.App.Analyzers/SourceTextGenerator/RemoveGenerator.cs

martinothamar · 2025-05-13T09:39:19Z

src/Altinn.App.Analyzers/SourceTextGenerator/SourceReaderUtils.cs

+                // Arrays have never been supported in data models.
+                // Just ignore them for now.
+                // return (UnwrapNullable(arrayTypeSymbol.ElementType), arrayTypeSymbol);
+                throw new NotSupportedException(


Should this function create diagnostics instead of throwing?

Yes, probably, but since 8.0.0 we crash at runtime if anyone has an array in their data model, and nobody complains (probably because they generate models in studio.)

martinothamar · 2025-05-13T09:41:21Z

src/Altinn.App.Analyzers/SourceTextGenerator/SourceReaderUtils.cs

+                    return (UnwrapNullable(collectionTypeSymbol.TypeArguments[0]), namedTypeSymbol);
+                }
+
+                break;


Is it always ICollection<>? Feels a little brittle. We could build a set of collection symbols in an earlier stage and do set lookups instead

The proper thing to do would probably be to use whatever System.Text.Json / PolyType does.

I think we should either try to do the simple thing or rely on a library.

src/Altinn.App.Analyzers/SourceTextGenerator/SourceTextGenerator.cs

src/Altinn.App.Analyzers/Utils/EquitableDiagnostic.cs

ivarne · 2025-05-15T09:39:06Z

/publish

github-actions · 2025-05-15T09:39:29Z

PR release:

⚙️ Building...
✅ Done!

sonarqubecloud · 2025-06-17T10:35:04Z

Quality Gate failed

Failed conditions
64.37% Condition Coverage on New Code (required ≥ 65%)

See analysis details on SonarQube Cloud

ivarne · 2025-06-19T08:26:35Z

/publish

github-actions · 2025-06-19T08:26:58Z

PR release:

⚙️ Building...
✅ Done!

* Introduce IFormDataWrapper * Deprecate DataModelWrapper * New helpers in IInstanceDataAccessor

This includes two test projects. * SourceGenerator.Tests: Use the generator directly and verify text output * SourceGenerator.Integration.Tests: Reference the generator in build process and test the generated code This has to be two projects, because it is really hard to debug a source generator when it runs as part of the build process.

This enables DataProcessors to get the clean versions of data models for calculations.

… service

Backend currently have 3 failiure conditions where a data model for a reference is not found. * It does not exist in applicationmetadata * No dataElement has yet been created * The dataType has maxCount != 1 Currently the backend message is "DataType non-existing not found in applicationmetadata.json", which is more specific so I modified the shared tests.

…th CleanDataAccessor when RemoveHiddenData=true

…rmDataValidator along with NoIncrementalValidation

The generated code could not add indexes to the end

If a List backing a repeating group contains a null, that is highly likely caused by cleaning the rows

sonarqubecloud · 2025-08-06T11:46:58Z

Quality Gate failed

Failed conditions
24.8% Coverage on New Code (required ≥ 65%)
21.95% Condition Coverage on New Code (required ≥ 65%)

See analysis details on SonarQube Cloud

ivarne · 2025-10-01T20:42:24Z

Now that #1421 is merged, there is no more need for this

ivarne added backport-ignore This PR is a new feature and should not be cherry-picked onto release branches feature Label Pull requests with new features. Used when generation releasenotes labels Apr 29, 2025

github-advanced-security bot found potential problems Apr 29, 2025

View reviewed changes

ivarne force-pushed the feat/cleanDataAccessor branch from 11165d0 to d6c3aff Compare April 30, 2025 13:12

github-advanced-security bot found potential problems Apr 30, 2025

View reviewed changes

src/Altinn.App.Analyzers/SourceTextGenerator/AltinnRowIdsGenerator.cs Fixed Show fixed Hide fixed

ivarne force-pushed the feat/cleanDataAccessor branch from b878697 to 92c03b0 Compare May 7, 2025 06:24

ivarne force-pushed the feat/cleanDataAccessor branch 9 times, most recently from f0f764f to 1f015ef Compare May 12, 2025 12:54

ivarne changed the title ~~Feat/clean data accessor~~ Remove hidden data before validation and make clean data available in DataProcessor May 12, 2025

ivarne marked this pull request as ready for review May 12, 2025 13:22

ivarne requested a review from martinothamar May 12, 2025 13:31

martinothamar reviewed May 13, 2025

View reviewed changes

ivarne mentioned this pull request May 14, 2025

Use strict mocking in signing tests to ensure that we do proper Setup() #1314

Merged

ivarne linked an issue Jun 18, 2025 that may be closed by this pull request

Don't use runtime reflection to read from data models (use source generation or IL code instead) #695

Open

4 tasks

ivarne added 21 commits August 6, 2025 13:08

Move Altinn.App.Analyzer from .Core to .Api

db23807

Make changes for more efficient removal of hidden data

7665d69

* Introduce IFormDataWrapper * Deprecate DataModelWrapper * New helpers in IInstanceDataAccessor

Fix conflicting tests

195b308

Add Clean and Previous InstanceDataAccessors

180db09

This enables DataProcessors to get the clean versions of data models for calculations.

Because of the ["text"] function data accessors needs the translation…

75b113a

… service

Clean data before validation if AppSettings.RemoveHiddenData = true

0638aec

Minor fixes

d9795d4

Improve test coverage

50a8bc6

sonar issues

25ebd7b

Fix code review feedback

52cd5ff

Use v4 of json schema for $schema annotations in layouts

de9e097

Cleanup

f364467

Add more tests for the RequiredLayoutValidator to see how it works wi…

f77dd10

…th CleanDataAccessor when RemoveHiddenData=true

Add ShouldRunAfterRemovingHiddenData to IValidator and proxy into IFo…

ba1d5cb

…rmDataValidator along with NoIncrementalValidation

Add more telemetry to iinstanceDataAccessor

3ca8324

Fix bug in AddIndexToPathGenerator

6c23edd

The generated code could not add indexes to the end

Store reference to State in Component context and Obsolete DataModel

7e35a7e

Check rows for null when evaluating hidden expressions

f11ac5b

If a List backing a repeating group contains a null, that is highly likely caused by cleaning the rows

Merge artifacts

2ca64b3

ivarne force-pushed the feat/cleanDataAccessor branch from 876c618 to 2ca64b3 Compare August 6, 2025 11:40

ivarne mentioned this pull request Aug 6, 2025

Support working with clean data in dataProcessors and validators and removeWhenHidden #1421

Merged

5 tasks

ivarne closed this Oct 1, 2025

Remove hidden data before validation and make clean data available in DataProcessor #1279

Remove hidden data before validation and make clean data available in DataProcessor #1279

Uh oh!

Conversation

ivarne commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related Issue(s)

Verification

Documentation

Uh oh!

ivarne commented Apr 29, 2025

Uh oh!

github-actions bot commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR release:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ivarne commented Apr 29, 2025

Uh oh!

github-actions bot commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR release:

Uh oh!

Uh oh!

ivarne commented May 7, 2025

Uh oh!

github-actions bot commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR release:

Uh oh!

ivarne commented May 13, 2025

Uh oh!

github-actions bot commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR release:

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ivarne commented May 15, 2025

Uh oh!

github-actions bot commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR release:

Uh oh!

sonarqubecloud bot commented Jun 17, 2025

Quality Gate failed

Uh oh!

ivarne commented Apr 29, 2025 •

edited

Loading

github-actions bot commented Apr 29, 2025 •

edited

Loading

github-actions bot commented Apr 29, 2025 •

edited

Loading

github-actions bot commented May 7, 2025 •

edited

Loading

github-actions bot commented May 13, 2025 •

edited

Loading

github-actions bot commented May 15, 2025 •

edited

Loading

github-actions bot commented Jun 19, 2025 •

edited

Loading