Fix: `format` does not transform `hash`-fields

[TobiasNx]

I intended to create an example for format inspired by:
https://manpages.ubuntu.com/manpages/plucky/man3/Catmandu::Fix::format.3pm.html

See example in playground

Input

{
    "number": "41",
    "numbers": [
        "42",
        "15"
    ],
    "hash": {
        "name": "Albert"
    }
}

Flux

inputFile
| open-file
| as-records
| decode-json
| fix(transformationFile)
| encode-yaml
| print
;

Fix

format("number","%-10.10d")
format("numbers[]","%-10.10d %-5.5d")
format("hash.name","%-10s: %s")

Runs into errors:

Caused by: java.util.IllegalFormatPrecisionException: 10
    at java.util.Formatter$FormatSpecifier.checkInteger (Formatter.java:3138)
        java.util.Formatter$FormatSpecifier.<init> (Formatter.java:2874)
        java.util.Formatter.parse (Formatter.java:2713)
        java.util.Formatter.format (Formatter.java:2655)
        java.util.Formatter.format (Formatter.java:2609)
        java.lang.String.format (String.java:2897)
        org.metafacture.metafix.FixMethod$16.lambda$apply$0 (FixMethod.java:242)
        org.metafacture.metafix.Value.lambda$asList$3 (Value.java:321)
        org.metafacture.metafix.Value.newArray (Value.java:167)
        org.metafacture.metafix.Value.asList (Value.java:317)
        org.metafacture.metafix.Value.asList (Value.java:297)
        org.metafacture.metafix.Value$Hash.getList (Value.java:965)
        org.metafacture.metafix.FixMethod$16.apply (FixMethod.java:241)
        org.metafacture.metafix.RecordTransformer.lambda$processFunction$20 (RecordTransformer.java:217)
        org.metafacture.metafix.RecordTransformer.lambda$processFix$22 (RecordTransformer.java:237)
        org.metafacture.metafix.RecordTransformer.lambda$transform$1 (RecordTransformer.java:136)
        org.metafacture.metafix.RecordTransformer.tryRun (RecordTransformer.java:248)
        org.metafacture.metafix.RecordTransformer.lambda$transform$2 (RecordTransformer.java:136)
        java.lang.Iterable.forEach (Iterable.java:75)
        org.metafacture.metafix.RecordTransformer.transform (RecordTransformer.java:135)
        org.metafacture.metafix.Metafix.endRecord (Metafix.java:325)
        org.metafacture.json.JsonDecoder.decode (JsonDecoder.java:297)
        org.metafacture.json.JsonDecoder.processRecord (JsonDecoder.java:247)
        org.metafacture.json.JsonDecoder.process (JsonDecoder.java:235)
        org.metafacture.json.JsonDecoder.process (JsonDecoder.java:45)
        org.metafacture.io.RecordReader.emitRecord (RecordReader.java:144)
        org.metafacture.io.RecordReader.process (RecordReader.java:132)
        org.metafacture.io.RecordReader.process (RecordReader.java:40)
        org.metafacture.io.FileOpener.process (FileOpener.java:158)
        org.metafacture.io.FileOpener.process (FileOpener.java:41)
        org.metafacture.flux.parser.StringSender.process (StringSender.java:43)
        org.metafacture.flux.parser.Flow.start (Flow.java:118)
        org.metafacture.flux.parser.FluxProgramm.start (FluxProgramm.java:168)
        org.metafacture.runner.Flux.main (Flux.java:87)
        metafacture_playground.process$process.invokeStatic (process.clj:45)
        metafacture_playground.process$process.invoke (process.clj:33)
        metafacture_playground.handler$process_request.invokeStatic (handler.clj:29)
        metafacture_playground.handler$process_request.invoke (handler.clj:27)
        metafacture_playground.handler$fn__3831.invokeStatic (handler.clj:96)
        metafacture_playground.handler/fn (handler.clj:92)
        compojure.core$wrap_response$fn__2406.invoke (core.clj:158)
        compojure.core$wrap_route_middleware$fn__2390.invoke (core.clj:128)
        compojure.core$wrap_route_info$fn__2395.invoke (core.clj:137)
        compojure.core$wrap_route_matches$fn__2399.invoke (core.clj:146)
        compojure.core$routing$fn__2414.invoke (core.clj:185)
        clojure.core$some.invokeStatic (core.clj:2701)
        clojure.core$some.invoke (core.clj:2692)
        compojure.core$routing.invokeStatic (core.clj:185)
        compojure.core$routing.doInvoke (core.clj:182)
        clojure.lang.RestFn.applyTo (RestFn.java:139)
        clojure.core$apply.invokeStatic (core.clj:667)
        clojure.core$apply.invoke (core.clj:660)
        compojure.core$routes$fn__2418.invoke (core.clj:192)
        ring.middleware.params$wrap_params$fn__3532.invoke (params.clj:67)
        metafacture_playground.handler$wrap_body_string$fn__3793.invoke (handler.clj:18)
        clojure.lang.Var.invoke (Var.java:384)
        ring.adapter.jetty$proxy_handler$fn__4050.invoke (jetty.clj:27)
        ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle (:-1)
        org.eclipse.jetty.server.handler.HandlerWrapper.handle (HandlerWrapper.java:127)
        org.eclipse.jetty.server.Server.handle (Server.java:501)
        org.eclipse.jetty.server.HttpChannel.lambda$handle$1 (HttpChannel.java:383)
        org.eclipse.jetty.server.HttpChannel.dispatch (HttpChannel.java:556)
        org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:375)
        org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:273)
        org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded (AbstractConnection.java:311)
        org.eclipse.jetty.io.FillInterest.fillable (FillInterest.java:105)
        org.eclipse.jetty.io.ChannelEndPoint$1.run (ChannelEndPoint.java:104)
        org.eclipse.jetty.util.thread.QueuedThreadPool.runJob (QueuedThreadPool.java:806)
        org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run (QueuedThreadPool.java:938)
        java.lang.Thread.run (Thread.java:829)

[blackwinter]

The Catmandu examples are not entirely portable, since the precision is not available for integers (%d). According to the Java documentation:

For character, integral, and date/time argument types and the percent and line separator conversions, the precision is not applicable; if a precision is provided, an exception will be thrown.

This is where the different host languages leak into the respective implementations. Perl's sprintf function differs in details from Java's feature set. I, for one, don't think we should - or even can - extend our compatibility requirements that far down into these implementation details.

---

Aside: The third example is wrong, it's supposed to format the hash instead of a single value hash.name. The mistake in Catmandu's documentation (name vs. hash; also number vs. numbers) becomes apparent when looking at the test instead.

[TobiasNx]

Thanks. I get a better understanding of format .

It seems that the transformation only works with strings since all values are strings:
https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-json%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=format%28%22number%22%2C%22+%25s+%22%29%0A%23+format%28%22number%22%2C%22+%25d+%22%29+%23+does+not+work+since+string+is+incoming%0A&data=%7B%0A++++%22Hello%22%3A+%22World%22%0A%7D

(I cannot change the input in the existing link... I opened an issue for this in the playground repo.)

Input:

format("number"," %s ")
# format("number"," %d ") # does not work since string is incoming

[blackwinter]

It seems that the transformation only works with strings since all values are strings:

That's correct. Another difference between Java and Perl.

[TobiasNx]

Thanks, I adjusted my example. It seems that hashes do not work in Fix, arrays and simple elements work with %s:

https://metafacture.org/playground/?flux=inputFile%0A%7C+open-file%0A%7C+as-records%0A%7C+decode-json%0A%7C+fix%28transformationFile%29%0A%7C+encode-yaml%0A%7C+print%0A%3B&transformation=%23+%23+Internally+all+values+in+Metafacture+are+strings%2C+therefore+only+%25s+for+strings+can+be+used.%0A%23+format%28%22name%22%2C%22%25-10s%22%29%0A%23+format%28%22number%22%2C%22%2510.10s%22%29%0A%23+%0A%23+%0A%23+format%28%22numbers%5B%5D%22%2C%22%25-10.10s+%25-5.5s%22%29%0Aformat%28%22hash%22%2C%22%25-10s%3A+%25-10s%22%29%0A&data=%7B%0A++++%22name%22%3A+%22Joe%22%2C%0A++++%22number%22%3A+%2241%22%2C%0A++++%22numbers%22%3A+%5B%0A++++++++%2242%22%2C%0A++++++++%2215%22%0A++++%5D%2C%0A++++%22hash%22%3A+%7B%0A++++++++%22name%22%3A+%22Albert%22%0A++++%7D%0A%7D%0A

{
    "name": "Joe",
    "number": "41",
    "numbers": [
        "42",
        "15"
    ],
    "hash": {
        "name": "Albert"
    }
}

# The first work
# # Internally all values in Metafacture are strings, therefore only %s for strings can be used.
# format("name","%-10s")
# format("number","%10.10s")
# 
# 
# format("numbers[]","%-10.10s %-5.5s")

# This does not:
format("hash","%-10s: %-10s")

When using:

format("hash","%-10s")

Output is:

hash: "{name=Albert}"

[blackwinter]

It seems that hashes do not work in Fix

Correct again. Apparently, our implementation (introduced in 4659439) isn't fully compatible with Catmandu's (yet).