You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+78-45Lines changed: 78 additions & 45 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,12 +1,31 @@
1
1
This library makes use of [pynmea2](https://github.com/Knio/pynmea2) to parse through input NMEA 0183 data, organize it, and output it to CSV files or to a PostgreSQL database.
2
2
3
+
## Terminology
4
+
**`sentence`**:
5
+
A line from your data file from a particular `talker` and of a particular `sentence_type` E.g.:
Your input file should have a format similiar to those under `test_data`. To have your data datetime stamped, it must be in a format like that of `test_data/test_data_all.nmea`, with RMC sentences containing date and time stamps proceed other sentences in the same cycle.
21
+
Input data files can contain either sentences having all the same `talker`+`sentence_type`, like that of `test_data/test_data_GLGSV.nmea`, `test_data_GNGGA.nmea`, etc., or cycles of sentences like that of `test_data/test_data_all.nmea`. Your input file should have a format similiar to those under `test_data`.
6
22
7
-
If working with a database, the database access information/credentials must be setup in `db_creds.py`.
23
+
To have your data datetime stamped, it must be in a format like that of `test_data/test_data_all.nmea`, with RMC sentences containing date and time stamps proceeding other sentences in the same cycle.
8
24
9
-
The `cycle_start``talker`+`sentence_type`, e.g. `GNRMC`, passed to `datetime_stamp_sentences()` and `assign_cycle_ids()`, must appear once and only once in each cycle, and it must be at the beginning of each cycle. For sentences to be datetime stamped, `cycle_start` sentences must contain date and time information. The `--backfill_datetimes` flag can be used to back fill datetimes for cycles where that information was not avaiable.
25
+
Useage of the `cycle_start` (`cs`), `num_sentences_per_cycle` (`spc`), and `backfill_datetimes` (`bfdt`) parameters will depend on the format of your data, and some combination of them is required. See below for examples. See the Usage section for explanations of the parameters.
26
+
27
+
28
+
If working with a database, the database access information/credentials must be setup in `db_creds.py`.
filepath file system path to file containing NMEA data
24
41
{csv,db,both} where to output data: CSV files, database, or both
25
42
26
43
optional arguments:
27
44
-h, --help show this help message and exit
28
-
--drop_previous_db_tables, --dropt
29
-
drop previous DB tables before importing new data;
30
-
only applies when output_method is 'db' or 'both'
45
+
--cycle_start CYCLE_START, --cs CYCLE_START
46
+
talker+sentence_type, e.g. 'GNRMC'; used to key off of for sentence merging, and more; must appear once and only once in each cycle, and must be at the beginning of each cycle; must contain date and time information for sentences to be datetime
If the cycle_start argument is not provided, and sentences are not all of type GSV, cycles will be inferred from this argument. Every num_sentences_per_cycle will be given the same cycle_id starting with the first sentence. Sentence merging is
50
+
based on cycle_id.
31
51
--backfill_datetimes, --bfdt
32
-
backfill datetimes where missing by extrapolating from
33
-
messages with datetime information
52
+
backfill datetimes where missing by extrapolating from messages with datetime information
53
+
--drop_previous_db_tables, --dropt
54
+
drop all previous DB tables before importing new data; only applies when output_method is 'db' or 'both'
34
55
```
35
56
## Examples
36
57
### Example 1
58
+
Output cycles of NMEA sentences to CSV files using GNRMC sentences as the cycle start:
-rw-r--r-- 1 Thomas staff 14310 Dec 30 18:19 test_data_all_GLGSV.csv
61
-
-rw-r--r-- 1 Thomas staff 9502 Dec 30 18:19 test_data_all_GNGGA.csv
62
-
-rw-r--r-- 1 Thomas staff 6852 Dec 30 18:19 test_data_all_GNGLL.csv
63
-
-rw-r--r-- 1 Thomas staff 18472 Dec 30 18:19 test_data_all_GNGSA.csv
64
-
-rw-r--r-- 1 Thomas staff 8672 Dec 30 18:19 test_data_all_GNRMC.csv
65
-
-rw-r--r-- 1 Thomas staff 5779 Dec 30 18:19 test_data_all_GNVTG.csv
66
-
-rw-r--r-- 1 Thomas staff 40263 Dec 30 18:19 test_data_all_GPGSV.csv
81
+
MacBook-Pro-4:nmea_data_convert Thomas$ ls -l *.csv
82
+
-rw-r--r-- 1 Thomas staff 16166 Jan 17 16:55 test_data_all_GLGSV.csv
83
+
-rw-r--r-- 1 Thomas staff 12067 Jan 17 16:55 test_data_all_GNGGA.csv
84
+
-rw-r--r-- 1 Thomas staff 9401 Jan 17 16:55 test_data_all_GNGLL.csv
85
+
-rw-r--r-- 1 Thomas staff 14136 Jan 17 16:55 test_data_all_GNGSA.csv
86
+
-rw-r--r-- 1 Thomas staff 12536 Jan 17 16:55 test_data_all_GNRMC.csv
87
+
-rw-r--r-- 1 Thomas staff 8344 Jan 17 16:55 test_data_all_GNVTG.csv
88
+
-rw-r--r-- 1 Thomas staff 20698 Jan 17 16:55 test_data_all_GPGSV.csv
67
89
```
68
90
69
91
### Example 2
92
+
Output cycles of NMEA sentences to both CSV files and database using GNRMC sentences as the cycle start, backfill datetimes, and drop previous tables from database:
70
93
```
71
-
$ python nmea_data_convert.py test_data/test_data_all.nmea db
72
-
73
-
Reading in data... done.
74
-
75
-
Processing data... done.
76
-
77
-
Writing data to database... data from logfile 'test_data/test_data_all.nmea' written to:
78
-
'nmea_gn_rmc' table in 'nmea_data' database
79
-
'nmea_gn_vtg' table in 'nmea_data' database
80
-
'nmea_gn_gga' table in 'nmea_data' database
81
-
'nmea_gn_gsa' table in 'nmea_data' database
82
-
'nmea_gp_gsv' table in 'nmea_data' database
83
-
'nmea_gl_gsv' table in 'nmea_data' database
84
-
'nmea_gn_gll' table in 'nmea_data' database
85
-
done.
86
-
87
-
All done. Exiting.
88
-
```
89
-
90
-
### Example 3
91
-
```
92
-
$ python nmea_data_convert.py test_data/test_data_all.nmea both --bfdt --dropt
94
+
$ python nmea_data_convert.py test_data/test_data_all.nmea both --bfdt --dropt --cs GNRMC
93
95
94
96
Reading in data... done.
95
97
@@ -126,8 +128,39 @@ done.
126
128
All done. Exiting.
127
129
```
128
130
131
+
### Example 3
132
+
Convert sentences, all of the same `talker`+`sentence_type`, to database:
133
+
```
134
+
$ python nmea_data_convert.py test_data/test_data_GNVTG.nmea db --spc 1
135
+
136
+
Reading in data... done.
137
+
138
+
Processing data... done.
139
+
140
+
Writing data to database... data from logfile 'test_data/test_data_GNVTG.nmea' written to:
141
+
'nmea_gn_vtg' table in 'nmea_data' database
142
+
done.
143
+
144
+
All done. Exiting.
145
+
146
+
```
147
+
148
+
### Example 4
149
+
Convert GSV sentences, all of the same `talker`, to database, where there may sometimes be multiple messages from the same cycle. In this case, cycles must start with the sentence having the `msg_num` field equal to `1` (see `test_data/test_data_GPGSV.nmea`:
150
+
```
151
+
$ python nmea_data_convert.py test_data/test_data_GPGSV.nmea db
152
+
[output excluded for brevity]
153
+
```
154
+
155
+
### Example 5
156
+
Convert GSA sentences, all of the same `talker`, to database, where each sentence is part of a cycle containing two GSA sentences. Cycles may contain a GSA sentence for each constellation (see `test_data/test_data_GNGSA.nmea`:
157
+
```
158
+
$ python nmea_data_convert.py test_data/test_data_GNGSA.nmea db --spc 2
If you find this tool useful, please consider supporting development of this tool and other tools like it. You can do so using the `Sponsor` button at the top of the [GitHub page](https://github.com/Petrichor-Labs/nmea_data_convert).
0 commit comments