Add support to (de)serialize Jackson's JsonNode [DATAMONGO-1651]

[spring-projects-issues]

Rasheed Amir opened DATAMONGO-1651 and commented

We have some entities which are extendable. And in order to make them extedable we have a field of type com.fasterxml.jackson.databind.node.JsonNode in them; so, that we can store key & value sort of data in it and then we can do searching on base of it as well. By default it serializes a JsonNode like as following:

{
    "qXiPdBVWnFqgSTaTXEiqrgjPp" : {
        "_children" : {
            "7c6b5721-744e-44ef-94c2-cdcd16dfdb6c" : {
                "_value" : "cb3dfac3-0552-4d48-9e06-20bb688330ff",
                "_class" : "com.fasterxml.jackson.databind.node.TextNode"
            },
            "47d32e01-a274-4995-b22c-d032fc39ee2b" : {
                "_children" : [ 
                    {
                        "_value" : "7e97f0f0-2b14-46e1-8998-95163f729371",
                        "_class" : "com.fasterxml.jackson.databind.node.TextNode"
                    }, 
                    {
                        "_value" : "74cd183a-68a3-483f-88dc-ca0d273d5744",
                        "_class" : "com.fasterxml.jackson.databind.node.TextNode"
                    }, 
                    {
                        "_value" : "0a897d65-c7c0-48ad-b0d5-a46d3a836397",
                        "_class" : "com.fasterxml.jackson.databind.node.TextNode"
                    }
                ],
                "_nodeFactory" : {
                    "_cfgBigDecimalExact" : false
                },
                "_class" : "com.fasterxml.jackson.databind.node.ArrayNode"
            },
            "6fc3394b-d235-4467-8f60-b5c71c8fb4b9" : {
                "_children" : {
                    "d989c804-7d6f-45b3-90d7-4f0d63964500" : {
                        "_value" : "17a4fcc6-f4c4-4680-989b-4640721033fe",
                        "_class" : "com.fasterxml.jackson.databind.node.TextNode"
                    },
                    "bcbc9540-1a33-47ec-af0a-9ba8de572323" : {
                        "_children" : [ 
                            {
                                "_value" : "2fee7cdf-ea40-4b95-a696-a8b7e1d9df8d",
                                "_class" : "com.fasterxml.jackson.databind.node.TextNode"
                            }, 
                            {
                                "_value" : "5359075e-475e-4987-bf24-ecd7d5b06756",
                                "_class" : "com.fasterxml.jackson.databind.node.TextNode"
                            }, 
                            {
                                "_value" : "4aea693b-a49c-4b9a-9505-376ea4daa622",
                                "_class" : "com.fasterxml.jackson.databind.node.TextNode"
                            }
                        ],
                        "_nodeFactory" : {
                            "_cfgBigDecimalExact" : false
                        },
                        "_class" : "com.fasterxml.jackson.databind.node.ArrayNode"
                    }
                },
                "_nodeFactory" : {
                    "_cfgBigDecimalExact" : false
                },
                "_class" : "com.fasterxml.jackson.databind.node.ObjectNode"
            }
        },
        "_nodeFactory" : {
            "_cfgBigDecimalExact" : false
        },
        "_class" : "com.fasterxml.jackson.databind.node.ObjectNode"
    },
    …
}

And it has two issues:

1. its not searchable
2. deserialization doesn't work; and when you try to read back you get this error

org.springframework.data.mapping.model.MappingInstantiationException: Failed to instantiate com.fasterxml.jackson.databind.node.ObjectNode using constructor NO_CONSTRUCTOR with arguments 

I would like it to serialize customdata like following:

{
    "customData" : {
        "acctest_boolFalse" : false,
        "acctest_numberList" : [ 
            1.0, 
            1.2, 
            3.0
        ],
        "acctest_stringList" : [ 
            "entry1", 
            "entry2"
        ],
        "acctest_boolTrue" : true,
        "acctest_int" : {
            "subStringList" : [ 
                "entry1", 
                "entry2"
            ]
        },
        "acctest_boolList" : [ 
            true, 
            false, 
            true
        ],
        "acctest_objList" : [ 
            {
                "subStringList" : [ 
                    "entry1", 
                    "entry2"
                ]
            }, 
            {
                "subBool" : true
            }
        ],
        "acctest_hej" : "seriesString",
        "acctest_null" : null,
        "acctest_decimal" : 1.12
    }
}

One easy hack is to register a custom Converter ( e.g. convert to String) but that way searching doesn't work. I can create sample project if you need

---

1 votes, 4 watchers

[spring-projects-issues]

Christoph Strobl commented

I'm not quite sure what you're trying to do with this construct. If your data is not too complex, what about storing stuff inside a Map instead of the JsonNode. Or create dedicated domain types with the additional fields you require per usecase storing them into the very same collection using @Document(collection="..."). Anyways - mind adding the smaple and giving a little more insight to this. Thanks!

[spring-projects-issues]

Rasheed Amir commented

Thanks for quick response @Christoph.

We are building a REST API; and we allow our end users to store any sort of custom data on some selected resources & then we want them to do query on based of that custom data as well. Please look at this sample input:

customData" : {
        "acctest_boolFalse" : false,
        "acctest_numberList" : [ 
            1.0, 
            1.2, 
            3.0
        ],
        "acctest_stringList" : [ 
            "entry1", 
            "entry2"
        ],
        "acctest_boolTrue" : true,
        "acctest_int" : {
            "subStringList" : [ 
                "entry1", 
                "entry2"
            ]
        },
        "acctest_boolList" : [ 
            true, 
            false, 
            true
        ],
        "acctest_objList" : [ 
            {
                "subStringList" : [ 
                    "entry1", 
                    "entry2"
                ]
            }, 
            {
                "subBool" : true
            }
        ],
        "acctest_hej" : "seriesString",
        "acctest_null" : null,
        "acctest_decimal" : 1.12
    }

We want to give flexibility to our end users so, we can't use Map or embedded collections. Does it help to understand or should I create a sample app as well?

[spring-projects-issues]

Oliver Drotbohm commented

There are currently no plans to support Jackson types natively. Usually using types like these in a domain model feels rather weird in the first place as they tie it to a (frontend) serialization library. As Christoph suggested, a custom converter reading the property into a Map should work as well but I'd even go a step further and suggest to let Jackson convert your payload into a Map in the first place and rather store that. That way you keep your domain model free of Jackson types in the first place, get searchability and whatever you store can be rendered as JSON easily, too

[spring-projects-issues]

Rasheed Amir commented

Thanks Olivier for your input.

How should the Map look like? e.g. Map<String, Object>? or what?

I found a way that I can use Jackson codec provider. But its very strange that it doesn't work with Spring Mongodb.

I have created this sample project: https://github.com/rasheedamir/spring-mongodb-jackson in which I am using bson4jackson to create a JacksonCodecProvider. Now here is the weird thing; if you plz clone this repo and then checkout "jackson" branch and look in ToDoRepositoryTest. You can see that the first test shouldSave using spring mongodb doesn't use methods of JacksonCodecProvider to encode and decode. But if you run the other test shouldSaveUsingJacksonCodec you will see that it does invoke JacksonCodeProvider. I really don't understand why! Please put breakpoints in encode JacksonCodec class and you will find that they aren't invoke in first case but they are invoked in second case. What could be the reason?

[spring-projects-issues]

Rasheed Amir commented

Quick update using following custom converters I was able to make it work:

public class ObjectNodeReadConverter implements Converter<DBObject, ObjectNode>
{
    private final ObjectMapper objectMapper;

    public ObjectNodeReadConverter(ObjectMapper objectMapper)
    {
        this.objectMapper = objectMapper;
    }

    @Override
    public ObjectNode convert(DBObject source)
    {
        try
        {
            return objectMapper.readValue(source.toString(), ObjectNode.class);
        }
        catch (IOException e)
        {
            throw new UncheckedIOException(e);
        }
    }
}

public class ObjectNodeWriteConverter implements Converter<ObjectNode, DBObject>
{
    @Override
    public DBObject convert(ObjectNode source)
    {
        return (DBObject) JSON.parse(source.toString());
    }
}

I need to test querying but I assume it will work

[saurabhcdt]

Is this working for you? I'm having same issue. But I'm not sure on libraries/imports used. Any specific maven dependency needs to be added for this? Can you please provide complete implementation or point to code link if possible?

[ClaudioConsolmagno]

I had a similar issue recently and the following worked for me (SpringBoot 2.4):

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.bson.Document;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.convert.converter.Converter;
import org.springframework.data.convert.ReadingConverter;
import org.springframework.data.convert.WritingConverter;
import org.springframework.data.mongodb.core.convert.MongoCustomConversions;

import java.util.List;

@Configuration
public class MongoDbBotBuilderConfiguration {

    private final ObjectMapper objectMapper;

    public MongoDbBotBuilderConfiguration() {
        this.objectMapper = new ObjectMapper()
                .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
                .findAndRegisterModules();
        this.objectMapper.activateDefaultTyping(objectMapper.getPolymorphicTypeValidator());
    }

    @Bean
    public MongoCustomConversions getMongoCustomConversions() {
        return new MongoCustomConversions(
                List.of(
                        new JsonNodeToDocumentConverter(),
                        new DocumentToJsonNodeConverter())
        );
    }

    @WritingConverter
    class JsonNodeToDocumentConverter implements Converter<JsonNode, Document> {
        @Override
        public Document convert(final JsonNode source) {
            return Document.parse(source.toString());
        }
    }

    @ReadingConverter
    class DocumentToJsonNodeConverter implements Converter<Document, JsonNode> {
        @Override
        public JsonNode convert(Document source) {
            try {
                return objectMapper.readTree(source.toJson());
            } catch (JsonProcessingException e) {
                throw new RuntimeException("Unable to parse Document to JsonNode", e);
            }
        }
    }

}

[christophstrobl]

as already mentioned we're not planning to add support for JsonNode conversion.

[flyhelanman]

I had a similar issue recently and the following worked for me (SpringBoot 2.4):

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.bson.Document;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.convert.converter.Converter;
import org.springframework.data.convert.ReadingConverter;
import org.springframework.data.convert.WritingConverter;
import org.springframework.data.mongodb.core.convert.MongoCustomConversions;

import java.util.List;

@Configuration
public class MongoDbBotBuilderConfiguration {

    private final ObjectMapper objectMapper;

    public MongoDbBotBuilderConfiguration() {
        this.objectMapper = new ObjectMapper()
                .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
                .findAndRegisterModules();
        this.objectMapper.activateDefaultTyping(objectMapper.getPolymorphicTypeValidator());
    }

    @Bean
    public MongoCustomConversions getMongoCustomConversions() {
        return new MongoCustomConversions(
                List.of(
                        new JsonNodeToDocumentConverter(),
                        new DocumentToJsonNodeConverter())
        );
    }

    @WritingConverter
    class JsonNodeToDocumentConverter implements Converter<JsonNode, Document> {
        @Override
        public Document convert(final JsonNode source) {
            return Document.parse(source.toString());
        }
    }

    @ReadingConverter
    class DocumentToJsonNodeConverter implements Converter<Document, JsonNode> {
        @Override
        public JsonNode convert(Document source) {
            try {
                return objectMapper.readTree(source.toJson());
            } catch (JsonProcessingException e) {
                throw new RuntimeException("Unable to parse Document to JsonNode", e);
            }
        }
    }

}

it work for me , But make another problem , when i save a JsonNode object , mongo can not auto generate ObjectId for me

[ggalantsev]

Inspired by @ClaudioConsolmagno comment i've created BsonValue converter to handle json primitives.

import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.RequiredArgsConstructor;
import lombok.SneakyThrows;
import org.bson.BsonArray;
import org.bson.BsonBoolean;
import org.bson.BsonDecimal128;
import org.bson.BsonDocument;
import org.bson.BsonDouble;
import org.bson.BsonInt32;
import org.bson.BsonInt64;
import org.bson.BsonNull;
import org.bson.BsonString;
import org.bson.BsonValue;
import org.bson.types.Decimal128;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.convert.converter.Converter;
import org.springframework.data.convert.ReadingConverter;
import org.springframework.data.convert.WritingConverter;
import org.springframework.data.mongodb.core.convert.MongoCustomConversions;

import java.util.List;
import java.util.Map;
import java.util.stream.StreamSupport;

@Configuration
@RequiredArgsConstructor
public class MongoConversionsConfiguration {

    private final ObjectMapper objectMapper;

    @Bean
    public MongoCustomConversions getMongoCustomConversions() {
        return new MongoCustomConversions(List.of(
            new JsonNodeToBsonConverter(),
            new ListToJsonNodeConverter<>(),
            new AnyToJsonNodeConverter()
        ));
    }

    @WritingConverter
    class JsonNodeToBsonConverter implements Converter<JsonNode, BsonValue> {
        @SneakyThrows
        @Override
        public BsonValue convert(final JsonNode json) {
            switch (json.getNodeType()) {
                case OBJECT:
                    Map<String, JsonNode> stringJsonNodeMap = objectMapper.readValue(json.toString(), new TypeReference<>() {
                    });
                    BsonDocument bsonDocument = new BsonDocument();
                    stringJsonNodeMap.forEach((key, value) -> bsonDocument.append(key, convert(value)));
                    return bsonDocument;

                case ARRAY:
                    return new BsonArray(StreamSupport.stream(json.spliterator(), false).map(this::convert).toList());

                case STRING:
                    return new BsonString(json.asText());

                case NUMBER:
                    if (json.isShort() || json.isInt()) return new BsonInt32(json.asInt());
                    if (json.isLong()) return new BsonInt64(json.asLong());
                    if (json.isFloat() || json.isDouble()) return new BsonDouble(json.asDouble());
                    if (json.isBigDecimal()) return new BsonDecimal128(new Decimal128(json.decimalValue()));
                    if (json.isBigInteger()) return new BsonString(json.bigIntegerValue().toString());
                    return json.isIntegralNumber() ? new BsonInt64(json.asLong()) : new BsonDouble(json.asDouble());

                case BOOLEAN:
                    return new BsonBoolean(json.asBoolean());

                case MISSING:
                case NULL:
                    return new BsonNull();

                default:
                    return new BsonString(objectMapper.writeValueAsString(json));
            }
        }
    }

    // without this converter, List<T> converts to first element of the list
    @ReadingConverter
    class ListToJsonNodeConverter<T> implements Converter<List<T>, JsonNode> {
        @Override
        public JsonNode convert(List<T> list) {
            return objectMapper.valueToTree(list);
        }
    }

    @ReadingConverter
    class AnyToJsonNodeConverter implements Converter<Object, JsonNode> {
        @Override
        public JsonNode convert(Object any) {
            return objectMapper.valueToTree(any);
        }
    }

}

For models like:

@Document(collection = "data")
@Data
@NoArgsConstructor
@AllArgsConstructor
public class Data {
    @Id
    private String id;
    private Map<String, JsonNode> data = new HashMap<>();
}

Validated with payload:

{
  "id": "123",
  "data": {
    "stringField": "exampleString",
    "numberField": 12345,
    "longNumberField": 1234567890,
    "bigIntegerField": 92233720368547758071,
    "decimalField": 123.456,
    "bigDecimalField": 1234567890.1234567890,
    "booleanField": true,
    "nullField": null,
    "arrayField": [1, 2, 3, 4, 5],
    "objectField": {
      "nestedString": "nestedExample",
      "nestedNumber": 67890,
      "nestedLongNumber": 1234567890,
      "nestedDecimal": 123.456,
      "nestedBigDecimal": 1234567890.123456789023234234,
      "nestedBigInteger": 92233720368547758071,
      "nestedBoolean": false,
      "nestedNull": null,
      "nestedArray": [6, 7, 8, 9, 10],
      "nestadArrayWithObjects": [
        {
          "name": "object1",
          "value": 1234567890
        },
        {
          "name": "object2",
          "value": 1234567890.1234567890
        }
      ],
      "nestedObject": {
        "deepNestedString": "deepNestedExample",
        "deepNestedNumber": 111213,
        "deepNestedBoolean": true
      }
    },
    "arrayOfObjects": [
      {
        "name": "object1",
        "value": 92233720368547758071
      },
      {
        "name": "object2",
        "value": 456
      }
    ],
    "objectWithArray": {
      "name": "exampleObject",
      "values": [10, 20, 30, 40, 50]
    }
  }
}

The only thing is:

• big decimals are trimmed to decimal128 and returned in scientific notation
• big integers are wrapped In string.