Skip to content

Commit 8e4871b

Browse files
committed
Update README.md
1 parent 73defed commit 8e4871b

File tree

1 file changed

+34
-15
lines changed

1 file changed

+34
-15
lines changed

README.md

Lines changed: 34 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,8 @@
11
# gulp-etl-tap-mysql #
22

3-
*(this plugin is being developed from **gulp-etl-tap-mysql**. The original readme from [gulp-etl-tap-csv](https://github.com/gulpetl/gulp-etl-tap-csv) is below)*
3+
This plugin connects to **MySQL** databases, running SQL queries and extracting the results to **gulp-etl** **Message Stream** files. It is a **gulp-etl** wrapper for [mysql](https://www.npmjs.com/package/mysql).
44

5-
This plugin converts CSV files to **gulp-etl** **Message Stream** files; originally adapted from the [gulp-etl-handlelines](https://github.com/gulpetl/gulp-etl-handlelines) model plugin. It is a **gulp-etl** wrapper for [csv-parse](https://csv.js.org/parse/).
6-
7-
This is a **[gulp-etl](https://gulpetl.com/)** plugin, and as such it is a [gulp](https://gulpjs.com/) plugin. **gulp-etl** plugins work with [ndjson](http://ndjson.org/) data streams/files which we call **Message Streams** and which are compliant with the [Singer specification](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md#output). In the **gulp-etl** ecosystem, **taps** tap into an outside format or system (in this case, a CSV file) and convert their contents/output to a Message Stream, and **targets** convert/output Message Streams to an outside format or system. In this way, these modules can be stacked to convert from one format or system to another, either directly or with tranformations or other parsing in between. Message Streams look like this:
5+
This is a **[gulp-etl](https://gulpetl.com/)** tap, but unlike most of the other gulp-etl modules it is not a [gulp](https://gulpjs.com/) **[plugin](https://gulpjs.com/docs/en/getting-started/using-plugins)**; it is actually a **[vinyl adapter](https://gulpjs.com/docs/en/api/concepts#vinyl-adapters)**--it features a replacement for **[gulp.src()](https://gulpjs.com/docs/en/api/src)**. **gulp-etl** plugins and adapters work with [ndjson](http://ndjson.org/) data streams/files which we call **Message Streams** and which are compliant with the [Singer specification](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md#output). In the **gulp-etl** ecosystem, **taps** tap into an outside format or system (in this case, a MySQL database) and convert their contents/output to a Message Stream, and **targets** convert/output Message Streams to an outside format or system. In this way, these modules can be stacked to convert from one format or system to another, either directly or with tranformations or other parsing in between. Message Streams look like this:
86

97
```
108
{"type": "SCHEMA", "stream": "users", "key_properties": ["id"], "schema": {"required": ["id"], "type": "object", "properties": {"id": {"type": "integer"}}}}
@@ -16,25 +14,46 @@ This is a **[gulp-etl](https://gulpetl.com/)** plugin, and as such it is a [gulp
1614
```
1715

1816
### Usage
19-
**gulp-etl** plugins accept a configObj as the first parameter; the configObj
20-
will contain any info the plugin needs. For this plugin the configObj is the "Options" object for [csv-parse](https://csv.js.org/parse/), described [here](https://csv.js.org/parse/options/); the only difference is that the "columns" property cannot be falsey, since it would result in arrays being returned
21-
for each row instead of objects. A falsey value for columns will be overridden to true.
17+
**gulp** adapters take two parameters: a glob, which is used to locate file(s) in the file system, and an optional config object with settings specific to the adapter. For example: ```src('*.txt', {buffer:false})```
2218

19+
Since this adapter doesn't pull from an existing file, the "glob" parameter is a virtual filename (with optional path info) which is assigned to the data file extracted from the server, e.g. ```mysqlData.ndjson```. And the configObj should look like this:
20+
##### configObj / mysql-settings.json
21+
```
22+
{
23+
"sql": "SELECT * FROM customers LIMIT 2;",
24+
"connection": {
25+
"host" : "example.org",
26+
"user" : "bob",
27+
"password" : "secret",
28+
"database" : "schemaName"
29+
}
30+
}
31+
```
32+
You *could* embed this information in your gulpfile, but we recommend storing it outside of any repo so that you don't accidentally publish it.
33+
2334
##### Sample gulpfile.js
2435
```
25-
/* parse all .CSV files in a folder into Message Stream files in a different folder */
36+
/* Run select query on the server and save the results in a local CSV file */
2637
27-
let gulp = require('gulp')
28-
var rename = require('gulp-rename')
29-
var tapCsv = require('gulp-etl-tap-csv').tapCsv
38+
var gulp = require('gulp')
39+
var tapMysql = require('gulp-etl-tap-mysql')
40+
var targetCsv = require('gulp-etl-target-csv').targetCsv
41+
42+
// contains secure info; store in parent folder of this project, outside of repo
43+
let configObj = require('../../mysql-settings.json')
3044
3145
exports.default = function() {
32-
return gulp.src('data/*.csv')
33-
.pipe(tapCsv({ columns:true }))
34-
.pipe(rename({ extname: ".ndjson" })) // rename to *.ndjson
35-
.pipe(gulp.dest('output/'));
46+
return tapMysql.src('mysqlResults.ndjson',configObj)
47+
.pipe(targetCsv({ columns:true }))
48+
.pipe(gulp.dest('output/'));
3649
}
3750
```
51+
52+
### Under Construction - notes and warnings
53+
* This is an early-stage module, with much functionality to come. Much of the feature set will be modeled after the [Singer MySQL tap](https://www.singer.io/tap/mysql/)
54+
* Currently only works in stream mode (configObj.buffer=false)
55+
* Tests are not yet added
56+
3857
### Quick Start for Coding on This Plugin
3958
* Dependencies:
4059
* [git](https://git-scm.com/downloads)

0 commit comments

Comments
 (0)