How to handle PATCH-style requests

[retendo]

Hi,
I can't figure out how to work with PATCH requests, where you want to update just the specified fields of an entity.
Problems arise when I want to update an optional value and I need to distinguish between the field not being present in the json and specifically setting null as a value.
I'm using Saturn/Giraffe/Thoth.Json.Giraffe.

If there is no supported way of doing this, I thought about representing my changes in JSON Patch format. That should be easier to handle. (http://jsonpatch.com)

Any suggestions?

[MangelMaxime]

In theory, you want to set skipNullField to false like that if a field is null the encoder/decoder will not skip it.

[retendo]

I tried this, but nothing changed. Maybe I'm missing something...

Here is the flawed manual solution that I came up with:

[<CLIMutable>]
type PatchReqElement = {
    Path: string
    Value: string option
}

[<CLIMutable>]
type RawPatchReq = {
    Data: PatchReqElement list
}

type PossessionUpdateReq = {
    Level: double option
    BestBefore: DateTime option option
    PricePaid: double option option
    EditionId: EditionId option
    DeletedAt: DateTime option option
}
module PossessionUpdateReq =
    let nones =
        {
            Level = None
            BestBefore = None
            PricePaid = None
            EditionId = None
            DeletedAt = None
        }

let toPossessionUpdateReq (raw: RawPatchReq) : Result<PossessionUpdateReq, string> =
        let populateRequest (patch: PatchReqElement) req =
            match patch.Path with
            | "level" ->
                let value = patch.Value |> Option.map double |> Result.requireSome "Field 'level' can't be null"
                value |> Result.map (fun value -> { req with Level = Some value })
            | "bestBefore" ->
                let value = patch.Value |> Option.map DateTime.Parse
                { req with BestBefore = Some value } |> Ok
            | "pricePaid" ->
                let value = patch.Value |> Option.map double
                { req with PricePaid = Some value } |> Ok
            | "editionId" ->
                let value = patch.Value |> Option.map (int64 >> EditionId) |> Result.requireSome "Field 'editionId' can't be null"
                value |> Result.map (fun value -> { req with EditionId = Some value })
            | "deletedAt" ->
                let value = patch.Value |> Option.map DateTime.Parse
                { req with DeletedAt = Some value } |> Ok
            | _ -> Ok req
        let populateRequestFiltered req raw = Result.bind (populateRequest raw) req
        
        let req = List.fold populateRequestFiltered (Ok PossessionUpdateReq.nones) raw.Data
        req

Now I can send something like this:

{ "data": 
    [
        { "path": "level", "value": "0.6" },
        { "path": "pricePaid", "value": "0.8" },
        { "path": "bestBefore", "value": null },
        { "path": "editionId", "value": "1" }
    ]
}

This solution has some flaws though, as I can't rely too much on a battle tested decoding library.
It would be great if there would be a way to decode a structure like PossessionUpdateReq directly, as in:
Make non-optional fields in your data model an option and optional fields an option option, so the outer option is there to say if a field is present in the JSON or not.
If it is not -> None
If it is -> Some ( if the value was explicitly set to null -> None OR Some (...decode value...) )

Does that make sense?

[MangelMaxime]

Hum, I guess the problem comes from the fact you are using 'T option option.

And we kind of erase the option type to a really simple representation:

• If Some ..., it outputs directly the value
• If None, it uses null or the absence of the value/property field.

So when nested several option we lose some information

open Fable.Core
open Thoth.Json

let someValue : string option= Some "Maxime"
let noneValue : string option = None
let someSomeValue : string option option = Some (Some "Maxime")
let someNoneValue : string option option = Some None
let deeplyNestedValue : string option option option option option = Some (Some (Some (Some None)))

JS.console.log(Encode.Auto.toString(4, someValue)) // "Maxime"
JS.console.log(Encode.Auto.toString(4, noneValue)) // null
JS.console.log(Encode.Auto.toString(4, someSomeValue)) // "Maxime"
JS.console.log(Encode.Auto.toString(4, someNoneValue)) // null
JS.console.log(Encode.Auto.toString(4, deeplyNestedValue)) // null

I am working on Thoth.Json 5 which is already doing some changes to how it represents some types perhaps we should make a custom representation for option type in order to retain the information. It will increase the JSON size but avoid losing information.

Right now, unless you copy/paste/adapt the code of the Auto modules you can't use it to solve your problem. However, you should be able to write your own Encode.option and Decode.option to have the desired behaviour I think.

Prototype:

There is a lot of code and I didn't focus on making it pretty just wanted to provide some hint for a potential solution. I think by using some helpers etc. it could look much better ^^

open Fable.Core
open Thoth.Json


// Standard behaviour from Thoth.Json Auto modules
module Standard =
    let someValue : string option= Some "Maxime"
    let noneValue : string option = None
    let someSomeValue : string option option = Some (Some "Maxime")
    let someNoneValue : string option option = Some None
    let deeplyNestedValue : string option option option option option = Some (Some (Some (Some None)))

    JS.console.log(Encode.Auto.toString(4, someValue)) // "Maxime"
    JS.console.log(Encode.Auto.toString(4, noneValue)) // null
    JS.console.log(Encode.Auto.toString(4, someSomeValue)) // "Maxime"
    JS.console.log(Encode.Auto.toString(4, someNoneValue)) // null
    JS.console.log(Encode.Auto.toString(4, deeplyNestedValue)) // null

module Custom =

    let log x = JS.console.log x

    module Encode =

        let losslessOption (encoder : 'a -> JsonValue) =
            fun value ->
                match value with
                | Some value ->
                    Encode.object 
                        [
                            "$type$", Encode.string "option"
                            "$state$", Encode.string "Some"
                            "$value$", encoder value
                        ]

                | None ->
                    Encode.object 
                        [
                            "$type$", Encode.string "option"
                            "$state$", Encode.string "None"
                        ]

    module Decode =
        
        let losslessOption (decoder : Decoder<'value>) : Decoder<'value option> =
            Decode.field "$type$" Decode.string
            |> Decode.andThen (fun typ ->
                match typ with
                | "option" ->
                    Decode.field "$state$" Decode.string
                    |> Decode.andThen (fun state ->
                        match state with
                        | "Some" ->
                            Decode.field "$value$" decoder |> Decode.map Some

                        | "None" ->
                            Decode.succeed None
                        
                        | invalid ->
                            "Expected an object with a field `$state$` set to `Some` or `None` but instead got `" + invalid + "`"
                            |> Decode.fail 
                    )

                | invalid ->
                    "Expected an object with a field `$type$` set to `option` but instead got `" + invalid + "`"
                    |> Decode.fail 
            )

    let someValue : string option= Some "Maxime"
    let noneValue : string option = None
    let someSomeValue : string option option = Some (Some "Maxime")
    let someNoneValue : string option option = Some None
    let deeplyNestedValue : string option option option option option = Some (Some (Some (Some None)))

    Encode.toString 4 (Encode.losslessOption Encode.string someValue)
    |> log

    Encode.toString 4 (Encode.losslessOption Encode.string noneValue)
    |> log

    Encode.toString 4 (Encode.losslessOption (Encode.losslessOption Encode.string) someSomeValue)
    |> log

    Encode.toString 4 (Encode.losslessOption (Encode.losslessOption Encode.string) someNoneValue)
    |> log

    Encode.toString 4 (Encode.losslessOption (Encode.losslessOption (Encode.losslessOption (Encode.losslessOption (Encode.losslessOption Encode.string)))) deeplyNestedValue)
    |> log

    match Decode.fromString (Decode.losslessOption Decode.string) (Encode.toString 4 (Encode.losslessOption Encode.string someValue)) with
    | Ok value ->
        match value with
        | Some value ->
            printfn "Got a Some ... %A" value
        
        | None ->
            printfn "Got a None"

    | Error err ->
        JS.console.error err

    match Decode.fromString (Decode.losslessOption Decode.string) (Encode.toString 4 (Encode.losslessOption Encode.string noneValue)) with
    | Ok value ->
        match value with
        | Some value ->
            printfn "Got a Some ... %A" value
        
        | None ->
            printfn "Got a None"

    | Error err ->
        JS.console.error err

    match Decode.fromString (Decode.losslessOption (Decode.losslessOption Decode.string)) (Encode.toString 4 (Encode.losslessOption (Encode.losslessOption Encode.string) someSomeValue)) with
    | Ok value ->
        match value with
        | Some (Some value) ->
            printfn "Got a Some (Some %A)" value
        
        | Some None ->
            printfn "Got a Some None"

        | None ->
            printfn "Got a None"

    | Error err ->
        JS.console.error err

REPL demo

[retendo]

Perfect, I will try this in the next couple of days. I think it would actually be a great addition to Thoth.Json, as right now there doesn't seem to be a nice, generic and build-in way to achieve this and PATCH requests are probably not that uncommon.

[retendo]

Going the custom road seems to be the way to go.

Custom decoder/encoder:

type PatchTestReq = {
        A: string option
        B: string option
        C: string option option
        D: string option option
        E: string option option
    } with
        static member Decoder : Decoder<PatchTestReq> =
            Decode.object
                (fun get ->
                    {
                        A = get.Optional.Field "a" Decode.string
                        B = get.Optional.Field "b" Decode.string
                        C = get.Optional.Field "c" (Decode.option Decode.string)
                        D = get.Optional.Field "d" (Decode.option Decode.string)
                        E = get.Optional.Field "e" (Decode.option Decode.string)
                    }
                )
        static member Encoder =
            Encode.Auto.generateEncoder<PatchTestReq>(caseStrategy = CaseStrategy.CamelCase)

Extra coders:

extra |> Extra.withCustom PatchTestReq.Encoder PatchTestReq.Decoder

So when you send this:

{
    "b": "B",
    "d": null,
    "e": "E"
}

...you will get this in F#:

{ A = None
  B = Some "B"
  C = None
  D = Some None  // <--- when using Auto.Decoder, this would be None
  E = Some (Some "E") }

...and when you send it as the response, you will get the exact same thing back:

{
    "b": "B",
    "d": null,  // <--- when using Auto.Decoder, this would not be present anymore
    "e": "E"
}

Which means the Auto Decoder behaves a bit differently when it comes to Optionals compared to the Encoder.

[retendo]

I got more...

With the solution above it isn't possible to allow for "b" to not be present in the JSON but specifically disallow { "b": null }.

A handy extension takes care of that:

[<AutoOpen>]
module DecodingHelper =
    [<RequireQualifiedAccess>]
    module Decode =
        let some field (decoder : Decoder<'value>) : Decoder<'value option> =
            fun path outerValue ->
                match Decode.field field decoder path outerValue with
                | Ok innerValue -> Ok (Some innerValue)
                | Error err ->
                    match err with
                    | (_, BadField _) -> Ok None
                    | _ -> Error err

    type Decode.IGetters with
        member x.OptionalRequiredField field decoder =
            x.Required.Raw (Decode.some field decoder)

You can use it like this:

static member Decoder : Decoder<PatchTestReq> =
    Decode.object
        (fun get ->
            {
                A = get.Optional.Field "a" Decode.string
                B = get.OptionalRequiredField "b" Decode.string // <--- Can be absent or have a value but not explicitly null
                C = get.Optional.Field "c" (Decode.option Decode.string)
                D = get.Optional.Field "d" (Decode.option Decode.string)
                E = get.Optional.Field "e" (Decode.option Decode.string)
            }
        )