Question: Why having conformance test on Mac and Linux

[nlohmann]

Are there any differences running the conformance tests on different operating systems?

[Loki-Astari]

To be honest I have not checked.
As an exercise I simply tried to make it run on both Mac an Linux in Travis. I was following the lead of the original author on this point.

[Loki-Astari]

Accidental close. Oops

[Loki-Astari]

A surprising number of differences:

> diff linux osx
< 1. Parse Validation,jsonxx,fail14,true
---
> 1. Parse Validation,jsonxx,fail14,false
950c950
< 1. Parse Validation,sajson,pass01,true
---
> 1. Parse Validation,sajson,pass01,false
952c952
< 1. Parse Validation,sajson,pass03,true
---
> 1. Parse Validation,sajson,pass03,false
987c987
< 1. Parse Validation,ujson,fail02,false
---
> 1. Parse Validation,ujson,fail02,true
989c989
< 1. Parse Validation,ujson,fail04,false
---
> 1. Parse Validation,ujson,fail04,true
993c993
< 1. Parse Validation,ujson,fail08,false
---
> 1. Parse Validation,ujson,fail08,true
1007c1007
< 1. Parse Validation,ujson,fail23,false
---
> 1. Parse Validation,ujson,fail23,true
1017,1019c1017,1019
< 1. Parse Validation,ujson,fail33,false
< 1. Parse Validation,ujson,pass01,false
< 1. Parse Validation,ujson,pass02,false
---
> 1. Parse Validation,ujson,fail33,true
> 1. Parse Validation,ujson,pass01,true
> 1. Parse Validation,ujson,pass02,true
2168c2168
< 2. Parse Double,jsoncpp,test19,true
---
> 2. Parse Double,jsoncpp,test19,false
2171,2173c2171,2173
< 2. Parse Double,jsoncpp,test21,true
< 2. Parse Double,jsoncpp,test22,true
< 2. Parse Double,jsoncpp,test23,true
---
> 2. Parse Double,jsoncpp,test21,false
> 2. Parse Double,jsoncpp,test22,false
> 2. Parse Double,jsoncpp,test23,false
2176c2176
< 2. Parse Double,jsoncpp,test26,true
---
> 2. Parse Double,jsoncpp,test26,false
2183,2188c2183,2188
< 2. Parse Double,jsoncpp,test32,true
< 2. Parse Double,jsoncpp,test33,true
< 2. Parse Double,jsoncpp,test34,true
< 2. Parse Double,jsoncpp,test35,true
< 2. Parse Double,jsoncpp,test36,true
< 2. Parse Double,jsoncpp,test37,true
---
> 2. Parse Double,jsoncpp,test32,false
> 2. Parse Double,jsoncpp,test33,false
> 2. Parse Double,jsoncpp,test34,false
> 2. Parse Double,jsoncpp,test35,false
> 2. Parse Double,jsoncpp,test36,false
> 2. Parse Double,jsoncpp,test37,false
2192c2192
< 2. Parse Double,jsoncpp,test40,true
---
> 2. Parse Double,jsoncpp,test40,false
2245c2245
< 2. Parse Double,jsonxx,test26,true
---
> 2. Parse Double,jsonxx,test26,false
2253,2257c2253,2257
< 2. Parse Double,jsonxx,test33,true
< 2. Parse Double,jsonxx,test34,true
< 2. Parse Double,jsonxx,test35,true
< 2. Parse Double,jsonxx,test36,true
< 2. Parse Double,jsonxx,test37,true
---
> 2. Parse Double,jsonxx,test33,false
> 2. Parse Double,jsonxx,test34,false
> 2. Parse Double,jsonxx,test35,false
> 2. Parse Double,jsonxx,test36,false
> 2. Parse Double,jsonxx,test37,false
3363c3363
< 2. Parse Double,yajl,test39,false
---
> 2. Parse Double,yajl,test39,true
3366c3366
< 2. Parse Double,yajl,test41,false
---
> 2. Parse Double,yajl,test41,true
3396c3396
< 2. Parse Double,yajl,test69,false
---
> 2. Parse Double,yajl,test69,true
3562,3563c3562,3563
< 3. Parse String,jvar,test1,false
< 3. Parse String,jvar,test10,false
---
> 3. Parse String,jvar,test1,true
> 3. Parse String,jvar,test10,true
3565,3570c3565,3570
< 3. Parse String,jvar,test2,false
< 3. Parse String,jvar,test3,false
< 3. Parse String,jvar,test5,false
< 3. Parse String,jvar,test6,false
< 3. Parse String,jvar,test8,false
< 3. Parse String,jvar,test9,false
---
> 3. Parse String,jvar,test2,true
> 3. Parse String,jvar,test3,true
> 3. Parse String,jvar,test5,true
> 3. Parse String,jvar,test6,true
> 3. Parse String,jvar,test8,true
> 3. Parse String,jvar,test9,true
3590,3591c3590,3591
< 3. Parse String,pjson,test10,true
< 3. Parse String,pjson,test11,true
---
> 3. Parse String,pjson,test10,false
> 3. Parse String,pjson,test11,false
3597c3597
< 3. Parse String,pjson,test9,true
---
> 3. Parse String,pjson,test9,false
3661,3665c3661,3665
< 3. Parse String,ujson,test1,false
< 3. Parse String,ujson,test10,false
< 3. Parse String,ujson,test11,false
< 3. Parse String,ujson,test2,false
< 3. Parse String,ujson,test3,false
---
> 3. Parse String,ujson,test1,true
> 3. Parse String,ujson,test10,true
> 3. Parse String,ujson,test11,true
> 3. Parse String,ujson,test2,true
> 3. Parse String,ujson,test3,true
3667,3669c3667,3669
< 3. Parse String,ujson,test6,false
< 3. Parse String,ujson,test8,false
< 3. Parse String,ujson,test9,false
---
> 3. Parse String,ujson,test6,true
> 3. Parse String,ujson,test8,true
> 3. Parse String,ujson,test9,true
4125,4126c4125,4126
< 4. Roundtrip,jsoncpp,roundtrip24,true
< 4. Roundtrip,jsoncpp,roundtrip25,true
---
> 4. Roundtrip,jsoncpp,roundtrip24,false
> 4. Roundtrip,jsoncpp,roundtrip25,false
4484c4484
< 4. Roundtrip,ujson,roundtrip05,false
---
> 4. Roundtrip,ujson,roundtrip05,true
4488,4489c4488,4489
< 4. Roundtrip,ujson,roundtrip09,false
< 4. Roundtrip,ujson,roundtrip10,false
---
> 4. Roundtrip,ujson,roundtrip09,true
> 4. Roundtrip,ujson,roundtrip10,true

[Loki-Astari]

Just a couple of the libraries though:

>diff linux osx | awk -F, '/^[<>]/ {print $2}' | sort | uniq -c
  28 jsoncpp
  14 jsonxx
  16 jvar
   6 pjson
   4 sajson
  36 ujson
   6 yajl

[nlohmann]

Wow, interesting! If the conformance is dependent on the operating system (or also: on the compiler), then may you should overwork (i.e., combine) the representation. If wouldn't say "this library is 100% conformant on Linux, and 90% conformant on Mac", but rather "this library is conformant in 95% of all tests executed".

[Loki-Astari]

That would be nice.

The problem here is that I simply took the web pages from the original author.
I have looked at the source (and fixed where I can) but this thing is old and clunky and I don't really have the skills to fix this old code (HTML and PHP and Javascript using some old google technology). But I can add some test to the page that says please also check the conformance on other OS's as they are not always the same.

Martin

[Loki-Astari]

I'll leave this issue open incase somebody who knows modern web stuff would like to contribute.

If you are just reading this and want to help please have a look at:
Data is generated by the "runTests" script which generates five csv files as output.
Two when run on Linux and Three when run on mac.

The HTML is generated by running the "success" script (which basically incorporates the csv data directly into the HTML page).

These HTML pages are generated by calling php scripts.
The php scripts are in the directory scripts: https://github.com/Loki-Astari/JsonBenchmark/tree/master/script

The rest of the "success" script. is simply generating a blog I write and deploying it to github. This does not change html in any way and simply copies the html to github pages project.

You can see the resulting source here:
https://github.com/Loki-Astari/loki-astari.github.io/tree/master/Json

And rendered here:
https://lokiastari.com/Json/Stats.html