Skip to content

Commit 8a6759c

Browse files
if0sif0s
authored
Write redesign (#102)
## 3.0.0 (July 9, 2021) * Deleted trigger: - `Read CSV attachment` * Deleted actions: - `Write CSV attachment` - `Write CSV attachment from JSON Array` - `Write CSV attachment from JSON Object` - `Read CSV file from URL` * Add New actions: - `Read CSV attachment` - `Create CSV From Message Stream` - `Create CSV From JSON Array` * Removed old dependencies
1 parent 9bbda69 commit 8a6759c

27 files changed

+5065
-3435
lines changed

‎.eslintrc.js‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
module.exports = {
2-
'extends': 'airbnb-base',
3-
'env': {
4-
'mocha': true
5-
}
6-
};
2+
extends: 'airbnb-base',
3+
env: {
4+
mocha: true,
5+
},
6+
rules: {
7+
'import/extensions': 'off',
8+
},
9+
};

‎.gitignore‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,4 +29,6 @@ build/Release
2929
# https://www.npmjs.org/doc/misc/npm-faq.html#should-i-check-my-node_modules-folder-into-git
3030
node_modules
3131

32-
.env
32+
.env
33+
.nyc_output
34+
.vscode

‎CHANGELOG.md‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,18 @@
1-
## 3.0.0 (June 25, 2021)
2-
* Deleted old action: `Read CSV file from URL`
3-
* Deleted old trigger: `Read CSV attachment`
4-
* Add New action: `Read CSV attachment`
1+
## 3.0.0 (July 9, 2021)
2+
* Deleted trigger:
3+
- `Read CSV attachment`
4+
* Deleted actions:
5+
- `Write CSV attachment`
6+
- `Write CSV attachment from JSON Array`
7+
- `Write CSV attachment from JSON Object`
8+
- `Read CSV file from URL`
9+
10+
* Add New actions:
11+
- `Read CSV attachment`
12+
- `Create CSV From Message Stream`
13+
- `Create CSV From JSON Array`
14+
15+
* Removed old dependencies
516

617
## 2.2.0 (April 23, 2021)
718
* Add pipe to list of separators in `Write CSV attachment from JSON Array` and `Write CSV attachment from JSON Object` actions

‎README.md‎

Lines changed: 55 additions & 112 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,18 @@
11
[![CircleCI](https://circleci.com/gh/elasticio/csv-component.svg?style=svg)](https://circleci.com/gh/elasticio/csv-component)
22
# CSV Component
33

4+
## Table of Contents
5+
* [Description](#description)
6+
* [How works](#how-works)
7+
* [Requirements](#requirements)
8+
* [Environment variables](#environment-variables)
9+
* [Credentials](#credentials)
10+
* [Actions](#actions)
11+
* [Read CSV attachment](#read-CSV-attachment)
12+
* [Create CSV From Message Stream](#create-CSV-from-message-stream)
13+
* [Create CSV From JSON Array](#create-CSV-from-JSON-array)
14+
* [Limitations](#limitations)
15+
416
## Description
517

618
A component to read and write Comma Separated Values (CSV) files.
@@ -45,139 +57,70 @@ To configure this action the following fields can be used:
4557
* `Delimiter` - The delimiting character. Leave blank to auto-detect from a list of most common delimiters.
4658
* `Convert Data types` - numeric, date and boolean data will be converted to their type instead of remaining strings, false by default.
4759

48-
### Write CSV attachment
49-
50-
* `Include Header` - this select configures output behavior of the component. If option is `Yes` or no value chosen than header of csv file will be written to attachment, this is default behavior. If value `No` selected than csv header will be omitted from attachment.
51-
52-
This action will combine multiple incoming events into a CSV file until there is a gap
53-
of more than 10 seconds between events. Afterwards, the CSV file will be closed
54-
and attached to the outgoing message.
55-
56-
As part of the component setup, one must specify the columns of the CSV file.
57-
These columns will be published as the header in the first row. For each incoming
58-
event, the value for each header will be `stringified` and written as the value
59-
for that cell. All other properties will be ignored. For example, headers
60-
`foo,bar` along with the following JSON events:
61-
62-
```
63-
{"foo":"myfoo", "bar":"mybar"}
64-
{"foo":"myfoo", "bar":[1,2]}
65-
{"bar":"mybar", "baz":"mybaz"}
66-
```
67-
68-
will produce the following `.csv` file:
69-
70-
```
71-
foo,bar
72-
myfoo,mybar
73-
myfoo,"[1,2]"
74-
,mybar
75-
```
76-
77-
When columns are added in the UI, you will be presented with an opportunity to
78-
provide a JSONata expression per column. If you require number formatting that
79-
is specific to a locale, the JSONata expression should handle that concern.
80-
81-
The output of the CSV Write component will be a message with an attachment. In
82-
order to access this attachment, the component following the CSV Write must be
83-
able to handle file attachments.
84-
85-
### Write CSV attachment from JSON Object
86-
87-
* `Include Header` - this select configures output behavior of the component. If option is `Yes` or no value chosen than header of csv file will be written to attachment, this is default behavior. If value `No` selected than csv header will be omitted from attachment.
88-
* `Separator` - this select configures type of CSV delimiter in an output file. There are next options: `Comma (,)`, `Semicolon (;)`, `Space ( )`, `Tab (\t)`, `Pipe (¦)`.
60+
### Create CSV From Message Stream
8961

9062
This action will combine multiple incoming events into a CSV file until there is a gap
9163
of more than 10 seconds between events. Afterwards, the CSV file will be closed
9264
and attached to the outgoing message.
9365

94-
This action will convert an incoming array into a CSV file by following approach:
95-
96-
* Header inherits names of keys from the input message;
97-
* Payload will store data from Values of relevant Keys (Columns);
98-
* Undefined values of a JSON Object won't be joined to result set (`{ key: undefined }`);
99-
* False values of a JSON Object will be represented as empty string (`{ key: false }` => `""`).
100-
101-
Requirements:
102-
103-
* The inbound message is an JSON Object, wrapped by 'inputObject' object;
104-
* This JSON object has plain structure without nested levels (structured types `objects` and `arrays` are not supported as values). Only primitive types are supported: `strings`, `numbers`, `booleans` and `null`. Otherwise, the error message will be thrown: `Inbound message should be a plain Object. At least one of entries is not a primitive type`.
105-
106-
The keys of an input JSON will be published as the header in the first row. For each incoming
107-
event, the value for each header will be `stringified` and written as the value
108-
for that cell. All other properties will be ignored. For example, headers
109-
`foo,bar` along with the following JSON events:
66+
#### Config Fields
11067

111-
```
112-
{"inputObject": {"foo":"myfoo", "bar":"mybar"}}
113-
{"inputObject": {"foo":"myfoo", "bar":[1,2]}}
114-
{"inputObject": {"bar":"mybar", "baz":"mybaz"}}
115-
```
68+
* `Upload CSV as file to attachments` - If checked store the generated CSV data as an attachment. If unchecked, place the CSV as a string in the outbound message.
69+
* `Separator` - A single char used to delimit the CSV file. Default to `,`
70+
* `Column Order` - A string delimited with the separator indicating which columns & in what order the columns should appear in the resulting file. If omitted, the column order in the resulting file will not be deterministic. Columns names will be trimmed (removed spaces in beginning and end of column name, for example: 'col 1,col 2 ,col 3, col 4' => ['col 1', 'col 2', 'col 3', 'col 4'])
11671

117-
will produce the following `.csv` file:
72+
#### Input Metadata
11873

119-
```
120-
foo,bar
121-
myfoo,mybar
122-
myfoo,"[1,2]"
123-
,mybar
124-
```
74+
* `Include Headers` - Indicates if a header row should be included in the generated file.
75+
* `Input Object` - Object to be written as a row in the CSV file. If the Column Order is specified, then individual properties can be specified.
12576

126-
The output of the CSV Write component will be a message with an attachment. In
127-
order to access this attachment, the component following the CSV Write must be
128-
able to handle file attachments.
77+
#### Output Metadata
12978

130-
### Write CSV attachment from JSON Array
79+
* If **Upload CSV as file to attachments** is checked:
80+
* `csvString` - The output CSV as a string inline in the body
13181

132-
* `Include Header` - this select configures output behavior of the component. If option is `Yes` or no value chosen than header of csv file will be written to attachment, this is default behavior. If value `No` selected than csv header will be omitted from attachment.
133-
* `Separator` - this select configures type of CSV delimiter in an output file. There are next options: `Comma (,)`, `Semicolon (;)`, `Space ( )`, `Tab (\t)`, `Pipe (¦)`.
82+
* If **Upload CSV as file to attachments** is not checked:
83+
* `attachmentUrl` - A URL to the CSV output
84+
* `type` - Always set to `.csv`
85+
* `size` - Size in bytes of the resulting CSV file
86+
* `attachmentCreationTime` - When the attachment was generated
87+
* `attachmentExpiryTime` - When the attachment is set to expire
88+
* `contentType` - Always set to `text/csv`
13489

135-
This action will convert an incoming array into a CSV file by following approach:
90+
### Create CSV From JSON Array
13691

137-
* Header inherits names of keys from the input message;
138-
* Payload will store data from Values of relevant Keys (Columns);
139-
* Undefined values of a JSON Object won't be joined to result set (`{ key: undefined }`);
140-
* False values of a JSON Object will be represented as empty string (`{ key: false }` => `""`).
92+
This action will convert an incoming array into a CSV file
14193

142-
Requirements:
94+
#### Config Fields
14395

144-
* The inbound message is an JSON Array of Objects with identical structure, wrapped by 'inputArray' object;
145-
* Each JSON object for a message has plain structure without nested levels (structured types `objects` and `arrays` are not supported as values). Only primitive types are supported: `strings`, `numbers`, `booleans` and `null`. Otherwise, the error message will be thrown: `Inbound message should be a plain Object. At least one of entries is not a primitive type`.
96+
* `Upload CSV as file to attachments` - If checked store the generated CSV data as an attachment. If unchecked, place the CSV as a string in the outbound message.
97+
* `Separator` - A single char used to delimit the CSV file. Default to `,`
98+
* `Column Order` - A string delimited with the separator indicating which columns & in what order the columns should appear in the resulting file. If omitted, the column order in the resulting file will not be deterministic.
14699

147-
The keys of an input JSON will be published as the header in the first row. For each incoming
148-
event, the value for each header will be `stringified` and written as the value
149-
for that cell. All other properties will be ignored. For example, headers
150-
`foo,bar` along with the following JSON events:
100+
#### Input Metadata
151101

152-
```
153-
{
154-
"inputArray": [
155-
{"foo":"myfoo", "bar":"mybar"}
156-
{"foo":"myfoo", "bar":[1,2]}
157-
{"bar":"mybar", "baz":"mybaz"}
158-
]
159-
}
160-
```
102+
* `Include Headers` - Indicates if a header row should be included in the generated file.
103+
* `Input Array` - Array of objects to be written as rows in the CSV file. (One row per object + headers) If the Column Order is specified, then individual properties can be specified. The component will throw an error when the array is empty.
161104

162-
will produce the following `.csv` file:
105+
#### Output Metadata
163106

164-
```
165-
foo,bar
166-
myfoo,mybar
167-
myfoo2,[1,2]"
168-
,mybar
169-
```
107+
* If **Upload CSV as file to attachments** is checked:
108+
* `csvString` - The output CSV as a string inline in the body
170109

171-
The output of the CSV Write component will be a message with an attachment. There will be one CSV file generated per incoming message. In
172-
order to access this attachment, the component following the CSV Write must be
173-
able to handle file attachments.
110+
* If **Upload CSV as file to attachments** is not checked:
111+
* `attachmentUrl` - A URL to the CSV output
112+
* `type` - Always set to `.csv`
113+
* `size` - Size in bytes of the resulting CSV file
114+
* `attachmentCreationTime` - When the attachment was generated
115+
* `attachmentExpiryTime` - When the attachment is set to expire
116+
* `contentType` - Always set to `text/csv`
174117

175-
### Limitations
118+
## Limitations
176119

177-
#### General
120+
### General
178121

179-
1. You may get `Component run out of memory and terminated.` error during run-time, that means that component needs more memory, please add
122+
* You may get `Component run out of memory and terminated.` error during run-time, that means that component needs more memory, please add
180123
`EIO_REQUIRED_RAM_MB` environment variable with an appropriate value (e.g. value `1024` means that 1024 MB will be allocated) for the component in this case.
181-
2. You may get `Error: write after end` error, as a current workaround try increase value of environment variable: `TIMEOUT_BETWEEN_EVENTS`.
182-
3. Maximal possible size for an attachment is 10 MB.
183-
4. Attachments mechanism does not work with [Local Agent Installation](https://docs.elastic.io/getting-started/local-agent.html)
124+
* Maximal possible size for an attachment is 10 MB.
125+
* Attachments mechanism does not work with [Local Agent Installation](https://docs.elastic.io/getting-started/local-agent.html)
126+
* Inbound message in `Message Stream` and each element of `JSON Array` should be a plain Object, if value not a primitive type it will be set as `[object Object]`

0 commit comments

Comments
 (0)