You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A basic lua binding to [simdjson](https://simdjson.org). The simdjson library is an incredibly fast JSON parser that uses SIMD instructions and fancy algorithms to parse JSON very quickly. It's been tested with LuaJIT 2.0/2.1 and Lua 5.1, 5.2, 5.3, and 5.4 on linux/osx. It has a general parsing mode and a lazy mode that uses a JSON pointer.
4
+
A basic Lua binding to [simdjson](https://simdjson.org). The simdjson library is an incredibly fast JSON parser that uses SIMD instructions and fancy algorithms to parse JSON very quickly. It's been tested with LuaJIT 2.0/2.1 and Lua 5.1, 5.2, 5.3, and 5.4 on linux/osx/windows. It has a general parsing mode and a lazy mode that uses a JSON pointer.
5
5
6
-
Current simdjson version: 0.5.0
6
+
Current simdjson version: 3.9.5
7
7
8
8
## Installation
9
9
If all the requirements are met, lua-simdjson can be install via luarocks with:
@@ -15,28 +15,29 @@ Otherwise it can be installed manually by pulling the repo and running luarocks
15
15
16
16
## Requirements
17
17
* lua-simdjson only works on 64bit systems.
18
-
* a lua build environment with support for C++11
18
+
* a Lua build environment with support for C++11
19
19
* g++ version 7+ and clang++ version 6+ or newer should work!
20
20
21
21
## Parsing
22
22
There are two main ways to parse JSON in lua-simdjson:
23
-
1. With `parse`: this parses JSON and returns a lua table with the parsed values
23
+
1. With `parse`: this parses JSON and returns a Lua table with the parsed values
24
24
2. With `open`: this reads in the JSON and keeps it in simdjson's internal format. The values can then be accessed using a JSON pointer (examples below)
25
25
26
26
Both of these methods also have support to read files on disc with `parseFile` and `openFile` respectively. If handling JSON from disk, these methods should be used and are incredibly fast.
27
27
28
28
## Typing
29
-
* lua-simdjson uses `simdjson.null` to represent `null` values from parsed JSON.
30
-
* Any application should use that for comparison as needed.
31
-
* it uses `lua_pushnumber` and `lua_pushinteger` for JSON floats and ints respectively, so your lua version may handle that slightly differently.
32
-
* All other types map as expected.
29
+
* lua-simdjson uses `simdjson.null` to represent `null` values from parsed JSON.
30
+
* Any application should use that for comparison as needed.
31
+
* it uses `lua_pushnumber` and `lua_pushinteger` for JSON floats and ints respectively, so your Lua version may handle that slightly differently.
32
+
*`lua_pushinteger` uses signed ints. A number from JSON larger than `LUA_MAXINTEGER` will be represented as a float/number
33
+
* All other types map as expected.
33
34
34
35
### Parse some JSON
35
-
The `parse` methods will return a normal lua table that can be interacted with.
36
+
The `parse` methods will return a normal Lua table that can be interacted with.
The `open` methods currently require the use of a JSON pointer, but are very quick.
64
+
The `open` methods currently require the use of a JSON pointer, but are very quick. They are best used when you only need a part of a response. In the example below, it could be useful for just getting the `Thumnail` object with `:atPointer("/Image/Thumbnail")` which will then only create a Lua table with those specific values.
64
65
```lua
65
66
localsimdjson=require("simdjson")
66
67
localresponse=simdjson.open([[
67
-
{
68
+
{
68
69
"Image": {
69
70
"Width": 800,
70
71
"Height": 600,
@@ -82,21 +83,21 @@ local response = simdjson.open([[
print(fileResponse:atPointer("/statuses/0/id")) --using a JSON pointer
87
88
88
89
```
89
-
Starting with version 0.5.0, the the `atPointer` method is JSON pointer compliant. The previous pointer implementation is considered deprecated, but is still available with the `at` method.
90
+
Starting with version 0.5.0, the `atPointer` method is JSON pointer compliant. The previous pointer implementation is considered deprecated, but is still available with the `at` method.
90
91
91
92
The `open` and `parse` codeblocks should print out the same values. It's worth noting that the JSON pointer indexes from 0.
92
93
93
-
This lazy style of using the simdjson data structure could also be used with array access in the future, and would result in ultra-fast JSON "parsing".
94
+
This lazy style of using the simdjson data structure could also be used with array access in the future.
94
95
95
96
## Error Handling
96
97
lua-simdjson will error out with any errors from simdjson encountered while parsing. They are very good at helping identify what has gone wrong during parsing.
97
98
98
99
## Benchmarks
99
-
I ran some benchmarks against lua-cjson, rapidjson, and dkjson. For each test, I loaded the JSON into memory, and then had the parsers go through each file 100 times and took the average time it took to parse to a lua table. You can see all the results in the [benchmark](benchmark/) folder. I've included a sample output run via Lua (the LuaJIT graph looks very similar, also in the benchmark folder). The y-axis is logarithmic, so every half step down is twice as fast.
100
+
I ran some benchmarks against lua-cjson, rapidjson, and dkjson. For each test, I loaded the JSON into memory, and then had the parsers go through each file 100 times and took the average time it took to parse to a Lua table. You can see all the results in the [benchmark](benchmark/) folder. I've included a sample output run via Lua (the LuaJIT graph looks very similar, also in the benchmark folder). The y-axis is logarithmic, so every half step down is twice as fast.
@@ -109,13 +110,13 @@ All tested files are in the [jsonexamples folder](jsonexamples/).
109
110
lua-simdjson, like the simdjson library performs better on more modern hardware. These benchmarks were run on a ninth-gen i7 processor. On an older processor, rapidjson may perform better.
110
111
111
112
## Caveats & Alternatives
112
-
* there is no encoding/dumping a lua table to JSON (yet! Most other lua JSON libraries can handle this)
113
-
* it only works on 64 bit systems (untested on Windows...)
113
+
* there is no encoding/dumping a Lua table to JSON (yet! Most other lua JSON libraries can handle this)
114
+
* it only works on 64 bit systems
114
115
* it builds a large binary. On a modern linux system, it ended up being \~200k (lua-cjson comes in at 42k)
115
116
* since it's an external module, it's not quite as easy to just grab the file and go (dkjson has you covered here!)
116
117
117
118
## Philosophy
118
-
I plan to keep it fairly inline with what the original simdjson library is capable of doing, which really means not adding too many additional options. The big _thing_ that's missing so far is encoding a lua table to JSON. I may add in an encoder at some point (likely modified from an existing lua library). There are some rumours that simdjson _may_ support creating JSON structure in the future. If that happens, I would likely switch to it.
119
+
I plan to keep it fairly inline with what the original simdjson library is capable of doing, which really means not adding too many additional options. The big _thing_ that's missing so far is encoding a lua table to JSON. I may add in an encoder at some point.
119
120
120
121
## Licenses
121
122
* The jsonexamples, src/simdjson.cpp, src/simdjson.h are unmodified from the released version simdjson under the Apache License 2.0.
0 commit comments