Skip to content
This repository was archived by the owner on Jul 15, 2021. It is now read-only.

Commit 9324521

Browse files
committed
Add section on stream transform parser. Refs #27
1 parent 6880085 commit 9324521

File tree

1 file changed

+54
-0
lines changed

1 file changed

+54
-0
lines changed

README.md

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,60 @@ sqliteParser(query, function (err, ast) {
5757
});
5858
```
5959

60+
### Use parser on Node streams *experimental*
61+
62+
This library also includes *experimental* support as a
63+
[stream transform](https://nodejs.org/api/stream.html) that can accept a
64+
_readable_ stream of SQL statements and produce a JSON string, representing
65+
the AST of each statement, as it is read and transformed. Using this method,
66+
the parser can handle files containing hundreds or thousands of queries at
67+
once without running into memory limitations. The AST for each statement is
68+
pushed down the stream as soon as it is read and parsed instead of reading the
69+
entire file into memory before parsing begins.
70+
71+
``` javascript
72+
var parserTransform = require('sqlite-parser').createParser();
73+
var readStream = require('fs').createReadStream('./large-input-file.sql');
74+
75+
readStream.pipe(parserTransform);
76+
parserTransform.pipe(process.stdout);
77+
78+
parser.on('error', function (err) {
79+
console.log(err);
80+
process.exit(1);
81+
});
82+
83+
parser.on('finish', function () {
84+
process.exit(0);
85+
});
86+
```
87+
88+
_Note:_ To pipe the output into a file that contains a single valid JSON
89+
structure, the output of the parser steam transform needs to be wrapped in
90+
statement list node where every statement is separated by a comma.
91+
92+
``` javascript
93+
var fs = require('fs');
94+
var sqliteParser = require('sqlite-parser');
95+
var parserTransform = sqliteParser.createParser();
96+
var singleNodeTransform = sqliteParser.createStitcher();
97+
var readStream = fs.createReadStream('./large-input-file.sql');
98+
var writeStream = fs.createWriteStream('./large-output-file.json');
99+
100+
readStream.pipe(parserTransform);
101+
parserTransform.pipe(singleNodeTransform);
102+
singleNodeTransform.pipe(writeStream);
103+
104+
parserTransform.on('error', function (err) {
105+
console.log(err);
106+
process.exit(1);
107+
});
108+
109+
writeStream.on('finish', function () {
110+
process.exit(0);
111+
});
112+
```
113+
60114
## Syntax Errors
61115

62116
This parser will try to create _smart_ error messages when it cannot parse

0 commit comments

Comments
 (0)