Merge pull request #11 from midwire/develop

Release gem version 1.0.1

Author: midwire

.gitignore

@@ -1,8 +1,7 @@
-ZIP_CODES.txt
-zip_codes.zip
-zip_dump.csv
-zip_codes.db
 /.bundle
 /free_zipcode_data.sqlite3
 /data
 /build
+/stubs
+/vendor/bundle/
+/pkg

.rspec

@@ -0,0 +1 @@
+--require spec_helper

.rubocop.yml

@@ -1,4 +1,6 @@
 AllCops:
+  TargetRubyVersion: 2.3
+
   # Include gemspec and Rakefile
   Include:
     - '**/*.gemspec'
@@ -15,6 +17,7 @@ AllCops:
   Exclude:
     - 'vendor/**/*'
     - 'stubs/**/*'
+    - 'spec/support/shared_contexts/*'
 
 # Checks formatting of special comments
 CommentAnnotation:
@@ -28,6 +31,9 @@ CommentAnnotation:
 ########################################
 # Style Cops
 
+Style/ClassVars:
+  Enabled: false
+
 Style/Documentation:
   Enabled: false
 
@@ -49,32 +55,16 @@ Style/RaiseArgs:
 Style/DoubleNegation:
   Enabled: false
 
-Style/SignalException:
-  EnforcedStyle: semantic
-
-Style/ClassAndModuleChildren:
-  Enabled: false
-
-Style/MultilineMethodCallIndentation:
-  EnforcedStyle: indented
-  IndentationWidth: 4
-
-Style/CaseIndentation:
-  Enabled: false
-
-Style/TrivialAccessors:
-  Enabled: false
-
-Style/NumericLiterals:
+Style/PerlBackrefs:
   Enabled: false
 
 ########################################
 # Lint Cops
 
 Lint/Eval:
-  Enabled: true
+  Enabled: false
 
-Lint/AssignmentInCondition:
+Lint/HandleExceptions:
   Enabled: false
 
 ########################################
@@ -85,7 +75,14 @@ Metrics/LineLength:
 
 Metrics/MethodLength:
   CountComments: false # count full line comments?
-  Max: 20
+  Max: 30
 
 Metrics/ClassLength:
   Max: 120
+
+Metrics/AbcSize:
+  Enabled: false
+
+# rubocop:disable Metrics/AbcSize
+# rubocop:disable Metrics/MethodLength
+# rubocop:disable Metrics/BlockLength

CHANGELOG

@@ -0,0 +1,42 @@
+*1.0.1* (April 23, 2018)
+
+* Made it a gem with a command line executable bin/free_zipcode_data
+* Use Kiba for ETL
+* Support user switches for various options including custom table names and generating .csv files
+* Use in-memory SQLite database to create the tables, then save it as a file on disk
+* Separate concerns for each table
+* Add a progressbar with ETA
+* Fix a bug when looking up state_id
+* Add a ‘name’ index on states table
+* Add a switch to generate individual .csv files [--generate-files]
+* Add a switch to overwrite [--clobber] downloaded and generated .csv files
+
+### Previous History
+
+### 10/28/2017:
+
+* Removed old .csv files
+* Add template files (CODE_OF_CONDUCT.md, CONTRIBUTING.md, ISSUE_TEMPLATE.md, LICENSE.md, PULL_REQUEST_TEMPLATE.md)
+* Use new data provider - [GeoNames](http://www.geonames.org)
+* Test USA
+* Started to test `allCountries.zip` but it takes too long. Let me know if there are any bugs.
+* Create `country_lookup_table.yml` for country code lookups
+
+### 05/04/2011:
+
+* Removed un-assigned zipcodes, which were not valid for today
+* Added a Rakefile and some rake tasks to facilitate building a SQLite relational database for the three tables (states, counties, zipcodes)
+* Zipcodes without an associated county == 0
+* Counties without a zipcode == 1 (PISCATAGUIS, Maine)
+
+### 01/24/2011:
+
+* 670 orphaned zipcodes without an associated county
+* 1 county without any zipcodes (PISCATAGUIS, Maine)
+
+### 01/13/2011:
+
+At last check there were ...
+
+* 897 orphaned zipcodes without an associated county
+* 1 county without any zipcodes (PISCATAGUIS, Maine)

CONTRIBUTING.md

@@ -1,4 +1,4 @@
-## How to contribute to jsonapi_rspec
+## How to contribute to free_zipcode_data
 
 #### **Did you find a bug?**
 

Gemfile

@@ -1,7 +1,6 @@
-source 'http://rubygems.org'
+# frozen_string_literal: true
 
-gem 'midwire_common'
-gem 'pry-nav'
-gem 'rake'
-gem 'rubyzip', '~> 1.2.1'
-gem 'sqlite3'
+source 'https://rubygems.org'
+git_source(:github) { |repo| "https://github.com/#{repo}.git" }
+
+gemspec

Gemfile.lock

@@ -1,31 +1,82 @@
+PATH
+  remote: .
+  specs:
+    free_zipcode_data (1.0.1)
+      colored (~> 1.2)
+      kiba (~> 2.0)
+      ruby-progressbar (~> 1.9)
+      rubyzip (~> 1.2)
+      sqlite3 (~> 1.3)
+      trollop (~> 2.1)
+
 GEM
-  remote: http://rubygems.org/
+  remote: https://rubygems.org/
   specs:
+    ast (2.4.0)
     coderay (1.1.2)
+    colored (1.2)
+    diff-lcs (1.3)
+    docile (1.3.0)
+    json (2.1.0)
+    kiba (2.0.0)
     method_source (0.8.2)
-    midwire_common (0.1.16)
-      thor (~> 0.19)
+    parallel (1.12.1)
+    parser (2.5.1.0)
+      ast (~> 2.4.0)
+    powerpack (0.1.1)
     pry (0.10.4)
       coderay (~> 1.1.0)
       method_source (~> 0.8.1)
       slop (~> 3.4)
     pry-nav (0.2.4)
       pry (>= 0.9.10, < 0.11.0)
+    rainbow (3.0.0)
     rake (12.2.1)
+    rspec (3.7.0)
+      rspec-core (~> 3.7.0)
+      rspec-expectations (~> 3.7.0)
+      rspec-mocks (~> 3.7.0)
+    rspec-core (3.7.1)
+      rspec-support (~> 3.7.0)
+    rspec-expectations (3.7.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.7.0)
+    rspec-mocks (3.7.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.7.0)
+    rspec-support (3.7.1)
+    rubocop (0.55.0)
+      parallel (~> 1.10)
+      parser (>= 2.5)
+      powerpack (~> 0.1)
+      rainbow (>= 2.2.2, < 4.0)
+      ruby-progressbar (~> 1.7)
+      unicode-display_width (~> 1.0, >= 1.0.1)
+    ruby-prof (0.17.0)
+    ruby-progressbar (1.9.0)
     rubyzip (1.2.1)
+    simplecov (0.16.1)
+      docile (~> 1.1)
+      json (>= 1.8, < 3)
+      simplecov-html (~> 0.10.0)
+    simplecov-html (0.10.2)
     slop (3.6.0)
-    sqlite3 (1.3.11)
-    thor (0.19.1)
+    sqlite3 (1.3.13)
+    trollop (2.1.2)
+    unicode-display_width (1.3.2)
 
 PLATFORMS
   ruby
 
 DEPENDENCIES
-  midwire_common
-  pry-nav
-  rake
-  rubyzip (~> 1.2.1)
-  sqlite3
+  bundler (~> 1.16)
+  free_zipcode_data!
+  pry-nav (~> 0.2)
+  rake (~> 12.0)
+  rspec (~> 3.7)
+  rubocop (~> 0.55)
+  ruby-prof (~> 0.17)
+  simplecov (~> 0.16)
 
 BUNDLED WITH
-   1.15.4
+   1.16.1

README.md

@@ -2,54 +2,72 @@
 
 Zipcode data is free from the various governments around the world. Yet so many organizations, reputable or not, want to charge money for it.
 
-We originally pulled down all the US census data we could find, parsed it and exported it into .csv files.
+This project is an automated solution for retrieving and collating US and worldwide zipcode data.
 
-In 2017 we began using [GeoNames](http://www.geonames.org) data, which is licensed under Creative Commons:
+## History
 
-We thank [GeoNames](http://www.geonames.org) for sharing, and urge you to [visit their site](http://www.geonames.org) and support their work.
+In 2011, we originally pulled down all the US census data we could find, parsed it and exported it into 3 .csv files. Later, we wrote 3 rake tasks to automate this process.
+
+In 2017 we began using [GeoNames](http://www.geonames.org) data, which is licensed under Creative Commons. We are grateful to [GeoNames](http://www.geonames.org) for sharing, and urge you to [visit their site](http://www.geonames.org) and support their work.
+
+In 2018 we refactored the project and made it into a Ruby gem with a command-line executable for automating this process.
 
 ## What's Included
 
-There are three rake tasks which automatically download the latest zipcode data from the country that you specify. Not all countries are accounted for. Please check [GeoNames](http://download.geonames.org/export/zip/) to see a list of supported country zip files.
+* An executable: `free_zipcode_data` - which automates the process of downloading and process the zipcode data from GeoNames. Not all countries are accounted for. Please check [GeoNames](http://download.geonames.org/export/zip/) to see a list of supported country zip files.
 
-Each zipcode is correlated with estimated or zip-centroid, latitude and longitude coordinates. Where applicable, county/province, state and community are also correlated.
+Each zipcode is correlated with estimated or zip-centroid, latitude and longitude coordinates. Where applicable, country, county/province, state and community are also correlated.
 
 See the GeoNames [readme.txt](http://download.geonames.org/export/zip/readme.txt) file for more information.
 
 ## Usage
 
+First, you need to install Ruby and Rubygems. Though that is not a difficult task, it is beyond the scope of this README. A search engine of your choice will help discover how to do this.  Once you have done that:
+
 ```bash
-$ git clone https://github.com/midwire/free_zipcode_data
-$ cd free_zipcode_data
-$ bundle install
+$ gem install free_zipcode_data
 ```
 
-Determine the countries you want to use at [GeoNames](http://download.geonames.org/export/zip/), or just use the rake task without any `country` argument to get zipcode data for all countries...
+Determine the 2-letter country codes for the countries you want to use at [GeoNames](http://download.geonames.org/export/zip/), or don't specify a country to get all zipcodes for all available countries...
+
+### Command Line Options
+
 ```bash
-# download will pull down the zipcodes for the specified country
-$ rake data:download[country]
-# rake data:download[US] - For US-only zipcodes
-# rake data:download[GB_full.csv] - All UK data zipcodes:
-# rake data:download - ALL zipcodes
+Options:
+  -w, --work-dir=<s>             REQUIRED: Specify your work/build directory, where the SQLite and .csv files will be built
+  -f, --country=<s>              Specify the country code for processing, or all countries if not specified
+  -g, --generate-files           Generate CSV files: [counties.csv, states.csv, countries.csv, zipcodes.csv]
+  -o, --country-tablename=<s>    Specify the name for the `countries` table (default: countries)
+  -s, --state-tablename=<s>      Specify the name for the `states` table (default: states)
+  -u, --county-tablename=<s>     Specify the name for the `counties` table (default: counties)
+  -z, --zipcode-tablename=<s>    Specify the name for the `zipcodes` table (default: zipcodes)
+  -c, --clobber                  Overwrite existing files
+  -d, --dry-run                  Do not actually move or copy files
+  -v, --verbose                  Be verbose with output
+  -h, --help                     Show this message
 ```
 
+### Examples
+
+**Download and process all US zipcodes**:
+
 ```bash
-# build the .csv files
-$ rake data:build[country]
-# rake data:build - ALL countries
+$ free_zipcode_data --work-dir /tmp/work_dir --country US --generate-files
 ```
 
+**Download and process zipcodes for all available countries**:
+
 ```bash
-# populate_db will automatically download the zips and build the sqlite3 db
-$ rake data:populate_db[country]
-# rake data:populate_db - ALL countries - WARNING: takes a long time
+$ free_zipcode_data --work-dir /tmp/work_dir --generate-files
 ```
 
 The rake tasks cascade, from the bottom up. So if you run `rake data:populate_db`, it will automatically call `rake data:build` if the .csv files are missing, which will call `rake data:download` if the .zip files are missing.
 
 ## SQLite3 Database
 
-The rake task `rake data:populate_db[country]` will create an SQLite3 database with the following tables, and populate each one:
+The executable will generate an SQLite3 database in the specified directory `--work-dir` but it will not generate the `.csv` files by default. Specify `--generate-files` if you want those as well.
+
+By default the tables will be named as follows. To override the table names see the command line options above.
 
 ```sql
 create table countries (
@@ -79,7 +97,6 @@ create table zipcodes (
   id integer not null primary key,
   code varchar(10) not null,
   state_id integer,
-  county_id integer,
   city varchar(255),
   area_code varchar(3),
   lat float,
@@ -95,32 +112,4 @@ Both `lat` and `lon`, geocodes, are populated for each zipcode record.
 The zipcode data is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/3.0/">Creative Commons Attribution 3.0 Unported License</a>, carried forward from [GeoNames](http://www.geonames.org).<br />
 <a rel="license" href="http://creativecommons.org/licenses/by/3.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by/3.0/88x31.png" /></a>
 
-## Errata
-
-### 10/28/2017:
-
-* Removed old .csv files
-* Add template files (CODE_OF_CONDUCT.md, CONTRIBUTING.md, ISSUE_TEMPLATE.md, LICENSE.md, PULL_REQUEST_TEMPLATE.md)
-* Use new data provider - [GeoNames](http://www.geonames.org)
-* Test USA
-* Started to test `allCountries.zip` but it takes too long. Let me know if there are any bugs.
-* Create `country_lookup_table.yml` for country code lookups
-
-### 05/04/2011:
-
-* Removed un-assigned zipcodes, which were not valid for today
-* Added a Rakefile and some rake tasks to facilitate building a SQLite relational database for the three tables (states, counties, zipcodes)
-* Zipcodes without an associated county == 0
-* Counties without a zipcode == 1 (PISCATAGUIS, Maine)
-
-### 01/24/2011:
-
-* 670 orphaned zipcodes without an associated county
-* 1 county without any zipcodes (PISCATAGUIS, Maine)
-
-### 01/13/2011:
-
-At last check there were ...
-
-* 897 orphaned zipcodes without an associated county
-* 1 county without any zipcodes (PISCATAGUIS, Maine)
+See [CHANGELOG](CHANGELOG) for more history and errata.

Rakefile

@@ -1,5 +1,10 @@
+# frozen_string_literal: true
+
 require 'rubygems'
 require 'bundler/setup'
 
 require 'rake'
 Dir['lib/tasks/**/*.rake'].sort.each { |ext| load ext }
+
+# Install rubygem tasks
+Bundler::GemHelper.install_tasks