pwhois display bugfix, and massively improved normalization: removal of empty-like values (-, n/a, etc.), mapping of abbreviations for countries, US/CA/AU states and airport codes to full locality names.

Author: joepie91

README.md

@@ -51,6 +51,23 @@ The manual (including install instructions) can be found in the doc/ directory.
 
 If any of those apply, don't hesitate to file an issue! The goal is 100% coverage, and we need your feedback to reach that goal.
 
+## License
+
+This library may be used under the WTFPL - or, if you take issue with that, consider it to be under the CC0.
+
+## Data sources
+
+This library uses a number of third-party datasets for normalization:
+
+* `airports.dat`: [OpenFlights Airports Database](http://openflights.org/data.html) ([Open Database License 1.0](http://opendatacommons.org/licenses/odbl/1.0/), [Database Contents License 1.0](http://opendatacommons.org/licenses/dbcl/1.0/))
+* `countries.dat`: [Country List](https://github.com/umpirsky/country-list) (MIT license)
+* `countries3.dat`: [ISO countries list](https://gist.github.com/eparreno/205900) (license unspecified)
+* `states_au.dat`: Part of `pythonwhois` (WTFPL/CC0)
+* `states_us.dat`: [State Table](http://statetable.com/) (license unspecified, free reuse encouraged)
+* `states_ca.dat`: [State Table](http://statetable.com/) (license unspecified, free reuse encouraged)
+
+Be aware that the OpenFlights database in particular has potential licensing consequences; if you do not wish to be bound by these potential consequences, you may simply delete the `airports.dat` file from your distribution. `pythonwhois` will assume there is no database available, and will not perform airport code conversion (but still function correctly otherwise). This also applies to other included datasets.
+
 ## Contributing
 
 Feel free to fork and submit pull requests (to the `develop` branch)! If you change any parsing or normalization logic, ensure to run the full test suite before opening a pull request. Instructions for that are below.

pwhois

@@ -1,6 +1,6 @@
 #!/usr/bin/env python2
 
-import argparse, pythonwhois, json, datetime
+import argparse, pythonwhois, json, datetime, sys
 try:
 	from collections import OrderedDict
 except ImportError as e:
@@ -100,7 +100,10 @@ else:
 					for key, value in data_map.items():
 						if key in contact_data and contact_data[key] is not None:
 							label = "    " + value + (" " * (widest_label - len(value))) + " :"
-							actual_data = contact_data[key]
+							if sys.version_info < (3, 0):
+								actual_data = unicode(contact_data[key])
+							else:
+								actual_data = str(contact_data[key])
 							if "\n" in actual_data: # Indent multi-line values properly
 								lines = actual_data.split("\n")
 								actual_data = "\n".join([lines[0]] + [(" " * (widest_label + 7)) + line for line in lines[1:]])

pythonwhois/airports.dat

@@ -0,0 +1,265 @@
+iso,name
+AF,Afghanistan
+AL,Albania
+DZ,Algeria
+AS,"American Samoa"
+AD,Andorra
+AO,Angola
+AI,Anguilla
+AQ,Antarctica
+AG,"Antigua and Barbuda"
+AR,Argentina
+AM,Armenia
+AW,Aruba
+AU,Australia
+AT,Austria
+AZ,Azerbaijan
+BS,Bahamas
+BH,Bahrain
+BD,Bangladesh
+BB,Barbados
+BY,Belarus
+BE,Belgium
+BZ,Belize
+BJ,Benin
+BM,Bermuda
+BT,Bhutan
+BO,Bolivia
+BA,"Bosnia and Herzegovina"
+BW,Botswana
+BV,"Bouvet Island"
+BR,Brazil
+BQ,"British Antarctic Territory"
+IO,"British Indian Ocean Territory"
+VG,"British Virgin Islands"
+BN,Brunei
+BG,Bulgaria
+BF,"Burkina Faso"
+BI,Burundi
+KH,Cambodia
+CM,Cameroon
+CA,Canada
+CT,"Canton and Enderbury Islands"
+CV,"Cape Verde"
+KY,"Cayman Islands"
+CF,"Central African Republic"
+TD,Chad
+CL,Chile
+CN,China
+CX,"Christmas Island"
+CC,"Cocos [Keeling] Islands"
+CO,Colombia
+KM,Comoros
+CG,"Congo - Brazzaville"
+CD,"Congo - Kinshasa"
+CK,"Cook Islands"
+CR,"Costa Rica"
+HR,Croatia
+CU,Cuba
+CY,Cyprus
+CZ,"Czech Republic"
+CI,"Côte d’Ivoire"
+DK,Denmark
+DJ,Djibouti
+DM,Dominica
+DO,"Dominican Republic"
+NQ,"Dronning Maud Land"
+DD,"East Germany"
+EC,Ecuador
+EG,Egypt
+SV,"El Salvador"
+GQ,"Equatorial Guinea"
+ER,Eritrea
+EE,Estonia
+ET,Ethiopia
+FK,"Falkland Islands"
+FO,"Faroe Islands"
+FJ,Fiji
+FI,Finland
+FR,France
+GF,"French Guiana"
+PF,"French Polynesia"
+TF,"French Southern Territories"
+FQ,"French Southern and Antarctic Territories"
+GA,Gabon
+GM,Gambia
+GE,Georgia
+DE,Germany
+GH,Ghana
+GI,Gibraltar
+GR,Greece
+GL,Greenland
+GD,Grenada
+GP,Guadeloupe
+GU,Guam
+GT,Guatemala
+GG,Guernsey
+GN,Guinea
+GW,Guinea-Bissau
+GY,Guyana
+HT,Haiti
+HM,"Heard Island and McDonald Islands"
+HN,Honduras
+HK,"Hong Kong"
+HU,Hungary
+IS,Iceland
+IN,India
+ID,Indonesia
+IR,Iran
+IQ,Iraq
+IE,Ireland
+IM,"Isle of Man"
+IL,Israel
+IT,Italy
+JM,Jamaica
+JP,Japan
+JE,Jersey
+JT,"Johnston Island"
+JO,Jordan
+KZ,Kazakhstan
+KE,Kenya
+KI,Kiribati
+KW,Kuwait
+KG,Kyrgyzstan
+LA,Laos
+LV,Latvia
+LB,Lebanon
+LS,Lesotho
+LR,Liberia
+LY,Libya
+LI,Liechtenstein
+LT,Lithuania
+LU,Luxembourg
+MO,"Macau SAR China"
+MK,Macedonia
+MG,Madagascar
+MW,Malawi
+MY,Malaysia
+MV,Maldives
+ML,Mali
+MT,Malta
+MH,"Marshall Islands"
+MQ,Martinique
+MR,Mauritania
+MU,Mauritius
+YT,Mayotte
+FX,"Metropolitan France"
+MX,Mexico
+FM,Micronesia
+MI,"Midway Islands"
+MD,Moldova
+MC,Monaco
+MN,Mongolia
+ME,Montenegro
+MS,Montserrat
+MA,Morocco
+MZ,Mozambique
+MM,"Myanmar [Burma]"
+NA,Namibia
+NR,Nauru
+NP,Nepal
+NL,Netherlands
+AN,"Netherlands Antilles"
+NT,"Neutral Zone"
+NC,"New Caledonia"
+NZ,"New Zealand"
+NI,Nicaragua
+NE,Niger
+NG,Nigeria
+NU,Niue
+NF,"Norfolk Island"
+KP,"North Korea"
+VD,"North Vietnam"
+MP,"Northern Mariana Islands"
+NO,Norway
+OM,Oman
+PC,"Pacific Islands Trust Territory"
+PK,Pakistan
+PW,Palau
+PS,"Palestinian Territories"
+PA,Panama
+PZ,"Panama Canal Zone"
+PG,"Papua New Guinea"
+PY,Paraguay
+YD,"People's Democratic Republic of Yemen"
+PE,Peru
+PH,Philippines
+PN,"Pitcairn Islands"
+PL,Poland
+PT,Portugal
+PR,"Puerto Rico"
+QA,Qatar
+RO,Romania
+RU,Russia
+RW,Rwanda
+RE,Réunion
+BL,"Saint Barthélemy"
+SH,"Saint Helena"
+KN,"Saint Kitts and Nevis"
+LC,"Saint Lucia"
+MF,"Saint Martin"
+PM,"Saint Pierre and Miquelon"
+VC,"Saint Vincent and the Grenadines"
+WS,Samoa
+SM,"San Marino"
+SA,"Saudi Arabia"
+SN,Senegal
+RS,Serbia
+CS,"Serbia and Montenegro"
+SC,Seychelles
+SL,"Sierra Leone"
+SG,Singapore
+SK,Slovakia
+SI,Slovenia
+SB,"Solomon Islands"
+SO,Somalia
+ZA,"South Africa"
+GS,"South Georgia and the South Sandwich Islands"
+KR,"South Korea"
+ES,Spain
+LK,"Sri Lanka"
+SD,Sudan
+SR,Suriname
+SJ,"Svalbard and Jan Mayen"
+SZ,Swaziland
+SE,Sweden
+CH,Switzerland
+SY,Syria
+ST,"São Tomé and Príncipe"
+TW,Taiwan
+TJ,Tajikistan
+TZ,Tanzania
+TH,Thailand
+TL,Timor-Leste
+TG,Togo
+TK,Tokelau
+TO,Tonga
+TT,"Trinidad and Tobago"
+TN,Tunisia
+TR,Turkey
+TM,Turkmenistan
+TC,"Turks and Caicos Islands"
+TV,Tuvalu
+UM,"U.S. Minor Outlying Islands"
+PU,"U.S. Miscellaneous Pacific Islands"
+VI,"U.S. Virgin Islands"
+UG,Uganda
+UA,Ukraine
+SU,"Union of Soviet Socialist Republics"
+AE,"United Arab Emirates"
+GB,"United Kingdom"
+US,"United States"
+ZZ,"Unknown or Invalid Region"
+UY,Uruguay
+UZ,Uzbekistan
+VU,Vanuatu
+VA,"Vatican City"
+VE,Venezuela
+VN,Vietnam
+WK,"Wake Island"
+WF,"Wallis and Futuna"
+EH,"Western Sahara"
+YE,Yemen
+ZM,Zambia
+ZW,Zimbabwe
+AX,"Åland Islands"