Skip to content

Commit 728e31a

Browse files
wiremoonswiremoons
committed
Readme Updates
Minor correction and improvements to the README.md file
1 parent f06c40b commit 728e31a

File tree

1 file changed

+70
-53
lines changed

1 file changed

+70
-53
lines changed

‎README.md‎

Lines changed: 70 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -3,61 +3,59 @@
33

44
## Application Summary
55

6-
CVS2SQL is a small simple program specifically designed to quickly convert a comma separated value (CSV) file into simple structured query language (SQL) statements, which can then be used as an import source for an SQLite database.
6+
`csv2sql` is a small simple program specifically designed to quickly convert a [comma separated value (CSV)](http://en.wikipedia.org/wiki/Comma-separated_values) file into simple [structured query language (SQL)](http://en.wikibooks.org/wiki/Structured_Query_Language) statements, which can then be used as an import source for a [SQLite](http://www.sqlite.org/) database.
77

88
## About CSV2SQL
99

10-
The program was originally created to speed up the process of checking and then importing large (often greater than 1GB) CSV files into SQLite databases. The data would vary quite a bit, often being sourced from multiple financial, audit, billing and business support corporate computer systems - so there was no consistency in the CSV file formats provided, from project to project. The data was used for ad-hoc revenue assurance investigations, and often to aide recovery processes, and reporting for the associated projects.
10+
The program was originally created to speed up the process of checking and then importing large (often greater than 1GB) CSV files into SQLite databases. The data would vary quite a bit, often being sourced from multiple financial, audit, billing and business support corporate computer systems—so there was no consistency in the CSV file formats provided, from project to project. The data was used for ad-hoc revenue assurance investigations, and often to aide recovery processes, and reporting for the associated projects.
1111

12-
The different data sources (starting as CSV files - as data was extracted from business system by a different teams) would be loaded into an ad-hoc SQLite database as tables, and then analysed with the benefit of SQL, and sometimes in later stages, scripts to produce recovery data and reports on the more complex projects. The work often required a quick turn around - so any tools that could provide increased efficiency, but still maintain integrity (or even increase the integrity checking) became key. Having simple tools to improve the work flow, and produce consistent repeatable results was very important!
12+
The different data sources (starting as CSV files—as data was extracted from business system by a different teams) would be loaded into an ad-hoc SQLite database as tables, and then analysed with the benefit of SQL, and sometimes in later stages, scripts to produce recovery data and reports on the more complex projects. The work often required a quick turn around—so any tools that could provide increased efficiency, but still maintain integrity (or even increase the integrity checking) became key. Having simple tools to improve the work flow, and produce consistent repeatable results was very important!
1313

14-
The csv2sql tool was created to quickly integrity check the source CSV file, report on it size (simple stats), and convert it into a text file that contained simple SQL statements. The simple SQL statements purpose was to both create a new database table to hold the CSV file data, and then to also insert the data directly into that new database table. These steps can be done by SQLite also, as it can directly import CSV files - but there was a wish to separately prepare and managed the CSV data files prior to involving the SQL database. This added the benefit of a simple additional integrity step, and put all the source CSV files data into a known state, and file format, prior to them being used with SQLite.
14+
The `csv2sql` tool was created to quickly integrity check the source CSV file, report on it size (simple stats), and convert it into a text file that contained simple SQL statements. The simple SQL statements purpose was to both create a new database table to hold the CSV file data, and then to also insert the data directly into that new database table. These steps can be done by SQLite also, as it can directly import CSV files—but there was a wish to separately prepare and managed the CSV data files, prior to involving the SQL database. This added the benefit of a simple additional integrity step, and put all the source CSV files data into a known state, and file format, prior to them being used with SQLite.
1515

1616
So key requirements were:
1717

18-
* be very fast - the source CSV files are often large, and therefore speed to process then quickly was important
19-
* check the CSV file contents (integrity check) - if there are any discrepancies found they are reported in a helpful way - so the CSV file can be fixed quickly
20-
* the output coveted data, in SQL format, should be consistent and basic
21-
* the data should remain in text format to allow future access, or use with other text manipulation tools if needed
22-
* the SQL statement format should be as simple as possible to reduce complexity - and as SQLite treats all data as text by default - this approach was adopted. Casting using SQL can be used then if needed at a later stage - or it was handled by high-level scripting languages
23-
* create consistently formatted SQL table column names (ie without spaces or 'strange' characters) - to allow future reference to the columns easily when constructing new queries. Different source computer systems (and their databases) had some very varied approaches to characters and formatting used!
24-
* should be cross platform if possible - so it can be used on any computer system so one toll works everywhere
25-
* should be command line based to reduce development time - and keep it simple to use. SQLite was used via the command line anyway (either directly of via scripts) - so continuing this approach was chosen.
18+
- be very fast—the source CSV files are often large, and therefore speed to process then quickly was important
19+
- check the CSV file contents (integrity check)—if there are any discrepancies found they are reported in a helpful way—so the CSV file can be fixed quickly
20+
- the output coveted data, in SQL format, should be consistent and basic
21+
- the data should remain in text format to allow future access, or use with other text manipulation tools if needed
22+
- the SQL statement format should be as simple as possible to reduce complexity—and as SQLite treats all data as text by default—this approach was adopted. Casting using SQL can be used then if needed at a later stage—or it was handled by high-level scripting languages
23+
- create consistently formatted SQL table column names (ie without spaces or 'strange' characters)—to allow future reference to the columns easily when constructing new queries. Different source computer systems (and their databases) had some very varied approaches to characters and formatting used!
24+
- should be cross platform if possible—so it can be used on any computer system so one toll works everywhere
25+
- should be command line based to reduce development time—and keep it simple to use. SQLite was used via the command line anyway (either directly of via scripts)—so continuing this approach was chosen.
2626

27+
A few different approaches were tried over time (using tcl, Python, and c) none of which were bad at the job—however the application was ported over to Go (golang)—and it immediately benefited as Go has a great built-in CSV file handling (as well as other formats), and the speed to process the file was impressive too. It might not be as fast as c, or it might not be as simple to understand as tcl or Pyhton code wise at first, but overall it suited my requirements best! Go also supports UTF8 characters without extra work, and was cross platform too!
2728

28-
A few different approaches were tried over time (using tcl, Python, and c) none of which were bad at the job - however the application was ported over to Go (golang) - and it immediately benefited as Go has a great built-in CSV file handling (as well as other formats), and the speed to process the file was impressive too. It might not be as fast as c, or it might not be as simple to understand as tcl or Pyhton code wise at first, but overall it suited my requirements best! Go also supports UTF8 characters without extra work, and was cross platform too!
29-
30-
Key features of csv2sql include:
29+
Key features of `csv2sql` include:
3130

32-
* The CSV file is integrity checked while being converted to SQL - to
33-
ensure it has a consistent number of column values. In other words the
34-
number of commas in the header (ie first line) of the CSV file, are the same
35-
throughout the rest of the file too.
36-
37-
* The first line of your CSV file will be designated as the header line -
38-
and therefore will become the column names in your subsequent SQLite
39-
database table.
40-
41-
* Any spaces or the following characters | - + @ # / \ : ( ) '
42-
found in the header line of you CSV file, will be replaced when they are
43-
used as the subsequent column names for your new SQLite table. These
44-
characters will be replaced with the underscore character (ie '_'). These
45-
changes only apply to the header line, and are carried out to avoid SQL
46-
syntax import issues, and make any future SQL statements referencing these
47-
column names easier to construct. This default feature can be disabled by
48-
using the command line parameter ' -k=true ' if you wish.
49-
50-
* You choose and specify the table name the CSV file contents will be
51-
imported into in your SQLite database when you run the program.
52-
53-
* The output file is a plain text file. It just contains the SQL commands
54-
that are used by SQLite to create and then insert all your data into your
55-
specified new table. The output file can therefore be edited (if you wish)
56-
to adapt it further - perhaps to suit you own needs.
31+
- The CSV file is integrity checked while being converted to SQL—to
32+
ensure it has a consistent number of column values. In other words the
33+
number of commas in the header (ie first line) of the CSV file, are the same
34+
throughout the rest of the file too.
35+
36+
- The first line of your CSV file will be designated as the header line—and therefore will become the column names in your subsequent SQLite
37+
database table.
38+
39+
- Any spaces or the following characters `| - + @ # / \ : ( ) '`
40+
found in the header line of you CSV file, will be replaced when they are
41+
used as the subsequent column names for your new SQLite table. These
42+
characters will be replaced with the underscore character (ie '_'). These
43+
changes only apply to the header line, and are carried out to avoid SQL
44+
syntax import issues, and make any future SQL statements referencing these
45+
column names easier to construct. This default feature can be disabled by
46+
using the command line parameter `-k=true` if you wish.
47+
48+
- You choose and specify the table name the CSV file contents will be
49+
imported into in your SQLite database when you run the program.
50+
51+
- The output file is a plain text file. It just contains the SQL commands
52+
that are used by SQLite to create and then insert all your data into your
53+
specified new table. The output file can therefore be edited (if you wish)
54+
to adapt it further—perhaps to suit you own needs.
5755

5856
## Command Line Arguments
5957

60-
When csv2sql is run without any parameters - it prints out the usage as follows:
58+
When csv2sql is run without any parameters—it prints out the usage as follows:
6159

6260
```
6361
Usage of ./csv2sql-linx64:
@@ -70,25 +68,25 @@ Usage of ./csv2sql-linx64:
7068

7169
Further details of each of these command line options is below:
7270

73-
* DEBUG: -d or -d=true
74-
This enables debug output when the program is run - so it prints additional information to the screen while it is running. This additional output might be useful to better understand what the application is doing - or to pin point where in the program a problem is occurring. For normal use it is not needed - so is turn off by default (ie -d=false)
71+
- **DEBUG:** `-d` or `-d=true`
72+
This enables debug output when the program is run—so it prints additional information to the screen while it is running. This additional output might be useful to better understand what the application is doing—or to pin point where in the program a problem is occurring. For normal use it is not needed—so is turn off by default (ie `-d=false`)
7573

76-
* CSV INPUT FILENAME: -f <filename.csv> [MANDATORY]
77-
This command line parameter is required to allow the program to run properly, so is mandatory for successful use. It specifies the name of the input CSV file, that will be used as the source data by the program to check and convert in to SQL, ready for import into an SQLite database table. The '<filename.csv>' as shown in the example should be replaced with the name of your actuals source CSV file. If you need to include the path to the CSV file, and it contains any spaces (or special characters) - you should wrap the filename and path in quotes: "/data-disk/datastore one/my_csv-data.csv" or "c:\Users\Fred Jones\My Documents\my csv-file.csv". There is no default value for this command line parameter - so the user my provide a CSV file to use to allow the program to run.
74+
- **CSV INPUT FILENAME:** `-f filename.csv` [MANDATORY]
75+
This command line parameter is required to allow the program to run properly, so is mandatory for successful use. It specifies the name of the input CSV file, that will be used as the source data by the program to check and convert in to SQL, ready for import into an SQLite database table. The `filename.csv` as shown in the example should be replaced with the name of your actuals source CSV file. If you need to include the path to the CSV file, and it contains any spaces (or special characters)—you should wrap the filename and path in quotes: `"/data-disk/datastore one/my_csv-data.csv"` or `"c:\Users\Fred Jones\My Documents\my csv-file.csv"`. There is no default value for this command line parameter - so the user my provide a CSV file to use to allow the program to run.
7876

79-
* ADDITIONAL HELP: -h or -h=true
80-
This will output additional information about the program, its purpose, and explanation of its usage. It may be useful to someone who did not originally install the program, so needs to know a bit more about it. If the program is run with this option, it will exit after displaying the help output. The default is not to show the additional help screen (ie -h=false)
77+
- **ADDITIONAL HELP:** `-h` or `-h=true`
78+
This will output additional information about the program, its purpose, and explanation of its usage. It may be useful to someone who did not originally install the program, so needs to know a bit more about it. If the program is run with this option, it will exit after displaying the help output. The default is not to show the additional help screen (ie `-h=false`)
8179

82-
* CSV HEADER CHANGES: -k=false [default]
83-
By default the program will change certain characters (ie space and | - + @ # / \ : ( ) ') to an underscore (ie _) when it uses the header of the CSV file, to create the new SQL database table column names. If you want to maintain you column names as they are in your source CSV file, then use this command line parameter to disable this behaviour. By default it will make the changes, so on the command lines specify -k=true to override. The -k stands for 'keep'.
80+
- **CSV HEADER CHANGES:** `-k=false` [default]
81+
By default the program will change certain characters (ie space and `| - + @ # / \ : ( ) '`) to an underscore (ie `_`) when it uses the header of the CSV file, to create the new SQL database table column names. If you want to maintain you column names as they are in your source CSV file, then use this command line parameter to disable this behaviour. By default it will make the changes, so on the command lines specify `-k=true` to override. The -k stands for 'keep'.
8482

85-
* TABLE NAME: -t tablename [MANDATORY]
86-
This command line parameter is required to allow the program to run properly, so is mandatory for successful use. It specifies the name of the table to be created in the SQLite database when your data is imported. Change the example <tablenname> to a name of your choice, that can be used within an SQLite database. If you need to use a tablename that contains any spaces (or special characters that SQLite allows of course) - you should wrap the tablename in quotes. Examples are: mytablename or "my table name" or my_table_name
83+
- **TABLE NAME:** `-t tablename` [MANDATORY]
84+
This command line parameter is required to allow the program to run properly, so is mandatory for successful use. It specifies the name of the table to be created in the SQLite database when your data is imported. Change the example `tablenname` to a name of your choice, that can be used within an SQLite database. If you need to use a tablename that contains any spaces (or special characters that SQLite allows of course)—you should wrap the tablename in quotes. Examples are: `mytablename` or `"my table name"` or `my_table_name`.
8785

8886

8987
## Compiling the Program
9088

91-
Assuming you already have Go install and set-up on your computer - you only need to download the single source file 'csv2sql.go'. This can then be built using the command, assuming the 'csv2sql.go' file is in you current directory:
89+
Assuming you already have Go install and set-up on your computer—you only need to download the single source file '[csv2sql.go](https://github.com/wiremoons/csv2sql/blob/master/csv2sql.go)'. This can then be built using the command, assuming the `csv2sql.go` file is in you current directory:
9290
```
9391
go build ./csv2sql.go
9492
```
@@ -99,8 +97,27 @@ make all
9997

10098
## Downloading Binary Version
10199

102-
The following binary version are available for download. Just download the file to your computer (either the .exe. or .zip version), copy the file so it is in you current path, and then run it. You may want to rename the downloaded file to just 'csv2sql' (Linux & Mac OS X) or 'csv2sql.exe' (Windows) as well.
100+
The following binary versions are available for download. Just download the file to your computer (either the .exe. or .zip version), copy the file so it is in you current path, and then run it. You may want to rename the downloaded file to just `csv2sql` (Linux & Mac OS X) or `csv2sql.exe` (Windows) as well.
101+
102+
Linux (64 bit)
103+
[Binary in zip file](https://github.com/wiremoons/csv2sql/blob/master/binaries/csv2sql-linx64.zip)
104+
[Binary file](https://github.com/wiremoons/csv2sql/blob/master/binaries/csv2sql-linx64)
105+
106+
Mac OS X (64 bit)
107+
[Binary in zip file](https://github.com/wiremoons/csv2sql/blob/master/binaries/csv2sql-macx64.zip)
108+
[Binary file](https://github.com/wiremoons/csv2sql/blob/master/binaries/csv2sql-macx64)
109+
110+
Mac OS X (64 bit)
111+
[Binary in zip file](https://github.com/wiremoons/csv2sql/blob/master/binaries/csv2sql-macx64.zip)
112+
[Binary file](https://github.com/wiremoons/csv2sql/blob/master/binaries/csv2sql-macx64)
113+
114+
Windows (64 bit)
115+
[Binary in zip file](https://github.com/wiremoons/csv2sql/blob/master/binaries/csv2sql-x64.zip)
116+
[Binary file](https://github.com/wiremoons/csv2sql/blob/master/binaries/csv2sql-x64.exe)
103117

118+
Windows (32 bit)
119+
[Binary in zip file](https://github.com/wiremoons/csv2sql/blob/master/binaries/csv2sql-x386.zip)
120+
[Binary file](https://github.com/wiremoons/csv2sql/blob/master/binaries/csv2sql-x386.exe)
104121

105122

106123
## License

0 commit comments

Comments
 (0)