Automatic deserialization $type

[asapostolov]

I have this property in one of my classes (this is how it's stored in RavenDB)

"Settings": {
        "$type": "System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[System.Object, mscorlib]], mscorlib",
        "lastUsedUnitId": {
              "1": "units-3368",
              "$type": "System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.String, mscorlib]], mscorlib"
          }
}

The C# property looks like this

public Dictionary<string, object> Settings { get; set; }

I assume (I think this is how this worked until recently) that lastUsedUnitId will automatically be deserialized to Disctionary<int, string> by the RavenDB deserialization process because of the $type property there. However - I find it's deserialized as JObject instead which I have to convert manually in my code to Dictionary.

Is there any way that this conversion can be done automatically? Am I missing something?

[maciejaszyk]

Hi, can you give us more information about which version of RavenDB Client and .NET you use?

I tried to reproduce it ( https://gist.github.com/maciejaszyk/ca4fecd932c322dca787ba6d673752fb ), but the test works just fine.

[ayende]

Following up again, you are explicitly using Patch with a script here, or using Patch through the session API?

[ayende]

Here is a similar issue, see: https://issues.hibernatingrhinos.com/issue/RavenDB-18414

[asapostolov]

@arekpalinski

As Oren pointed above the issue here is that your patch command explicitly uses mscorlib

I addressed this concern in the discussion above - I'm not running the test (or my regular app) an .NET Core/.NET5+ - I'm running a .NET Framework 4.8 app - you can make a .NET Framework project and try to run the test on it and confirm that it fails - it fails with me.

@ayende @arekpalinski
This is a test I created in order to research why I was getting this error with later/est versions of Raven.Client. I'm not using Patch in the code of my app - I'm just loading an entity that was previously saved with that $type declaration.

I downgraded Raven.Client in Friday to 5.2.2 (because of this issue) and my app is working fine with that version - it desterilizes the property correctly into a Dictionary, BUT the test above fails (with 5.2.2). I also tried the test with the previous stable version which I ran for a long time 5.1.4 and the test fails there too.

[asapostolov]

@arekpalinski
Here's a gist with the same test failing for .NET 5

https://gist.github.com/asapostolov/01a5e95c0168065a091f4337da359671

[ayende]

thanks, looking into this

[ayende]

Okay, that took way too long to figure out, but now I have a good handle on what is going on here.

This will not work:

"Settings": {
        "$type": "System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[System.Object, mscorlib]], mscorlib",
        "lastUsedUnitId": {
              "1": "units-3368",
              "$type": "System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.String, mscorlib]], mscorlib"
          }
}

This, on the other hand, will:

"Settings": {
        "$type": "System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[System.Object, mscorlib]], mscorlib",
        "lastUsedUnitId": {
              "$type": "System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.String, mscorlib]], mscorlib"
             "1": "units-3368", 
          }
}

The key difference between those two, the $type position in the inner dictionary.

NewtonSoft.Json requires that the "metadata" properties such as $type would be the first properties in the json in order to parse them properly.

[asapostolov]

However, editing this in the Studio would lead to the wrong sort order, which then cause the problem.

Ok, so this is the issue and it's probably limited to a few documents that I've edited through the studio which messed up the ordering.

I'll do some more tests and confirm if it's working correctly in my application code as well.

[asapostolov]

I cannot fix my "broken" documents because there's auto-format "on save" in the studio. It would be ideal if the auto-format moves "$type" as first attribute by default.

[ayende]

You might be able to do that using patch operation, or just load and save the document through the API, no?

[asapostolov]

Yes, I can patch them 1-by-1

But there's something else - my app does not have this issue with v. 5.2.2 (it deserializes the 'broken' documents correctly) while the unit test crashes with 5.2.2. There might be 2 separate issues here, but I need to think of a way to reproduce the issue that my app is having with the versions above 5.2.3

[ayende]

This isn't related to the version of RavenDB. That hasn't changed. The issue is probably reordering from the Studio.
Once that happens, the rest follows.

[asapostolov]

@ayende
I see that the issue we talked about is marked as fixed
https://issues.hibernatingrhinos.com/issue/RavenDB-18417

But the fix is very unhelpful.

It just gives a useless warning that doesn't prevent the issue from occurring.

Are you avoiding creating the fix of the JSON editor because it's too hard to change the algorithm for formatting? Or is there some other issue? I can imagine how easy might be to just skip the formatting function of the editor before saving when you detect that there is numeric key?

I also tried to fix the document with this script and it doesn't change the document if I don't change '-A' to '-a'.

from Users
where id()='users-3073-b'
update {
    this.Settings["lastUsedUnitId"] = {
        "$type": "System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.String, mscorlib]], mscorlib",
        "1": "units-14887-A"
    };
}

Which is extremely weird. Is that how patch is supposed to work? Not change the doc if the change is only in the ordering of the properties?

[ml054]

Hi @asapostolov

Parsing raw document to json is not only used for formatting documents - we modify its content (metadata) so we want to operate on object literals not on strings.

You may say: so let's write custom json serialize library to handle that case. It won't help us much, since then we break ECMA script standard (numeric props should go first). Also it might have negative performance impact (json.parse is native method).

Using patch in studio executes that on server side, where we use javascript engine to patch objects. Running patch you described above will result in props order: (1 and then type).

You can observe same approach when using google chrome network tab:

it also sort property keys to match ecma script standard.

I understand that this doesn't work well with your data structure, since serializer embeds $type property.

So adding a warning about having numeric properties seems to be helpful - it allows you prevent issues and be aware of a weird part of emca script specs. Let's assume you want to use node.js client - then you also might end with this issue - so it is better to warns user asap.

To fix your existing documents you can capture 'save document' request modify order of properties and issue this request manually via curl, postman etc.

[asapostolov]

Ok, I understand, thanks for explaining!

I think it's easier then, to rewrite my code in order to not use Dictionary<int, X> in this collection. You might consider stating somewhere that you don't support Dictionary<int, X> with $type serialization in order to prevent this from happening in the future.

Thanks alot!

[ml054]

Hi,

For following test:

 public class User
        {
            public string Id { get; set; }
            public string Name { get; set; }
            public IDictionary<string, string> TestingString { get; set; }
            public IDictionary<int, string> TestingInt { get; set; }
        }
        
        [Fact]
        public void CRUD_Operations()
        {
            using (var store = GetDocumentStore())
            {
                using (var newSession = store.OpenSession())
                {
                    newSession.Store(new User
                    {
                        Name = "user1",
                        TestingString = new Dictionary<string, string>() {
                            {"a", "b"}
                        },
                        TestingInt = new Dictionary<int, string>()
                        {
                            {5, "b"}
                        }
                    }, "users/1");

                    newSession.SaveChanges();
                }
            }
        }

We end up with document:

So we do support IDictionary. Problem starts with $type is introduced +we use patch (see https://issues.hibernatingrhinos.com/issue/RavenDB-18655) or if you start playing with document in studio (see https://issues.hibernatingrhinos.com/issue/RavenDB-18417).

That's the reason we warn about that.

[asapostolov]

Yes, you are right, thank you!