Release 2.3.0

Author: joepie91

README.md

@@ -39,6 +39,8 @@ The manual (including install instructions) can be found in the doc/ directory.
 
 ## Important update notes
 
+*2.3.0 and up*: Python 3 support was fixed. Creation date parsing for contacts was fixed; correct timestamps will now be returned, rather than unformatted ones - if your application relies on the broken variant, you'll need to change your code. Some additional parameters were added to the `net` and `parse` methods to facilitate NIC handle lookups; the defaults are backwards-compatible, and these changes should not have any consequences for your code. Thai WHOIS parsing was implemented, but is a little spotty - data may occasionally be incorrectly split up. Please submit a bug report if you run across any issues.
+
 *2.2.0 and up*: The internal workings of `get_whois_raw` have been changed, to better facilitate parsing of WHOIS data from registries that may return multiple partial matches for a query, such as `whois.verisign-grs.com`. This change means that, by default, `get_whois_raw` will now strip out the part of such a response that does not pertain directly to the requested domain. If your application requires an unmodified raw WHOIS response and is calling `get_whois_raw` directly, you should use the new `never_cut` parameter to keep pythonwhois from doing this post-processing. As this is a potentially breaking behaviour change, the minor version has been bumped.
 
 ## It doesn't work!

doc/usage.html

@@ -160,7 +160,7 @@ To start using pythonwhois, use `import pythonwhois`.
 
 ## When you need more control...
 
-^ pythonwhois.net.get_whois_raw(**domain**[, **server=""**, **rfc3490=True**, **never_cut=False**])
+^ pythonwhois.net.get_whois_raw(**domain**[, **server=""**, **rfc3490=True**, **never_cut=False**, **with_server_list=False**])
 
 	Retrieves the raw WHOIS data for the specified domain, and returns it as a list of responses (one element for each WHOIS server queried). This method will keep following redirects, until it ends up at the right server (and all responses it picks up in the meantime, will be included). Raises `pythonwhois.shared.WhoisException` if no root server for the TLD could be found.
 
@@ -175,6 +175,9 @@ To start using pythonwhois, use `import pythonwhois`.
 
 	never_cut::
 		**Optional.** If set to `True`, pythonwhois will never strip out data from the raw WHOIS responses, **even** if that data relates to a partial match, rather than the requested domain.
+		
+	with_server_list::
+		**Optional.** If set to `True`, `pythonwhois` will return a `(response, server_list)` tuple instead of just `response`, where the `server_list` contains a list of all the WHOIS servers that were queried to collect the raw data. The order of the list is from first (root) server to last server.
 
 
 ^ pythonwhois.net.get_root_server(**domain**)
@@ -184,7 +187,7 @@ To start using pythonwhois, use `import pythonwhois`.
 	domain::
 		The domain whose TLD you want to know the root WHOIS server for.
 
-^ pythonwhois.parse.parse_raw_whois(**raw_data**[, **normalized**])
+^ pythonwhois.parse.parse_raw_whois(**raw_data**[, **normalized**, **never_query_handles=True**, **handle_server=""**])
 
 	Parses the specified raw WHOIS data. It's useful to call this method manually if you receive your WHOIS data from elsewhere, and don't need the retrieval capabilities of pythonwhois. Returns an object like `pythonwhois.get_whois` does.
 
@@ -193,4 +196,9 @@ To start using pythonwhois, use `import pythonwhois`.
 
 	normalized::
 		**Optional.** Like for `pythonwhois.get_whois`. Empty list or omitted to normalize nothing, `True` to normalize all supported fields, a list of keys if you only want certain keys to be normalized.
-
+	
+	never_handle_queries::
+		**Optional.** If this is enabled, `pythonwhois` won't try to resolve handles that require additional queries. Some WHOIS servers only provide NIC handles by default and require additional lookups to retrieve the associated contact information. If you are running an offline application (or have your own load distribution mechanism) and you don't want `pythonwhois` to make queries on your behalf, set this to `True` (the default). When using the `get_whois` method, this is set to `False` behind the scenes, and handles will be automatically resolved.
+		
+	handle_server::
+		**Optional.** When `never_handle_queries` is set to `False`, you should specify the server responsible for handle lookups here. This is usually the last server in a WHOIS chain. `get_whois` will set this for you automatically.

doc/usage.zpy

@@ -3,7 +3,7 @@
 import argparse, pythonwhois, json, datetime
 try:
 	from collections import OrderedDict
-except ImportError, e:
+except ImportError as e:
 	from ordereddict import OrderedDict
 
 parser = argparse.ArgumentParser(description="Retrieves and parses WHOIS data for a domain name.")
@@ -20,18 +20,22 @@ def json_fallback(obj):
 		return obj
 
 if args.file is None:
-	data = pythonwhois.net.get_whois_raw(args.domain[0])
+	data, server_list = pythonwhois.net.get_whois_raw(args.domain[0], with_server_list=True)
 else:
+	server_list = []
 	with open(args.file, "r") as f:
 		data = f.read().split("\n--\n")
 
 if args.raw == True:
-	print "\n--\n".join(data)
+	print("\n--\n".join([x.encode("utf-8") for x in data]))
 else:
-	parsed = pythonwhois.parse.parse_raw_whois(data, normalized=True)
+	if len(server_list) > 0:
+		parsed = pythonwhois.parse.parse_raw_whois(data, normalized=True, never_query_handles=False, handle_server=server_list[-1])
+	else:
+		parsed = pythonwhois.parse.parse_raw_whois(data, normalized=True)
 
 	if args.json == True:
-		print json.dumps(parsed, default=json_fallback)
+		print(json.dumps(parsed, default=json_fallback))
 	else:
 		data_map = OrderedDict({})
 
@@ -46,18 +50,18 @@ else:
 		data_map["emails"] = ("E-mail address", "+")
 
 		widest_label = 0
-		for key, value in data_map.iteritems():
+		for key, value in data_map.items():
 			if len(value[0]) > widest_label:
 				widest_label = len(value[0])
 
-		for key, value in data_map.iteritems():
+		for key, value in data_map.items():
 			if key in parsed and parsed[key] is not None:
 				label = value[0] + (" " * (widest_label - len(value[0]))) + " :"
 				if value[1] == 1:
-					print "%s %s" % (label, parsed[key][0])
+					print("%s %s" % (label, parsed[key][0]))
 				elif value[1] == "+":
 					for item in parsed[key]:
-						print "%s %s" % (label, item)
+						print("%s %s" % (label, item))
 
 		if parsed["contacts"] is not None:
 			# This defines the contacts shown in the output
@@ -80,23 +84,24 @@ else:
 			data_map["email"] = "E-mail address"
 			data_map["phone"] = "Phone number"
 			data_map["fax"] = "Fax number"
+			data_map["creationdate"] = "Created"
 			data_map["changedate"] = "Last changed"
 
 			for contact in contacts_map:
 				if parsed["contacts"][contact] is not None:
 					contact_data = parsed["contacts"][contact]
 
-					print "\n" + contacts_map[contact]
+					print("\n" + contacts_map[contact])
 
-					for key, value in data_map.iteritems():
+					for key, value in data_map.items():
 						if len(value) > widest_label:
 							widest_label = len(value)
 
-					for key, value in data_map.iteritems():
+					for key, value in data_map.items():
 						if key in contact_data and contact_data[key] is not None:
 							label = "    " + value + (" " * (widest_label - len(value))) + " :"
-							actual_data = str(contact_data[key])
+							actual_data = contact_data[key]
 							if "\n" in actual_data: # Indent multi-line values properly
 								lines = actual_data.split("\n")
 								actual_data = "\n".join([lines[0]] + [(" " * (widest_label + 7)) + line for line in lines[1:]])
-							print "%s %s" % (label, actual_data)
+							print("%s %s" % (label, actual_data))

pwhois

@@ -1,8 +1,11 @@
 from . import net, parse
 
 def get_whois(domain, normalized=[]):
-	raw_data = net.get_whois_raw(domain)
-	return parse.parse_raw_whois(raw_data, normalized=normalized)
+	raw_data, server_list = net.get_whois_raw(domain, with_server_list=True)
+	# Unlisted handles will be looked up on the last WHOIS server that was queried. This may be changed to also query
+	# other servers in the future, if it turns out that there are cases where the last WHOIS server in the chain doesn't
+	# actually hold the handle contact details, but another WHOIS server in the chain does.
+	return parse.parse_raw_whois(raw_data, normalized=normalized, never_query_handles=False, handle_server=server_list[-1])
 
 def whois(*args, **kwargs):
 	raise Exception("The whois() method has been replaced by a different method (with a different API), since pythonwhois 2.0. Either install the older pythonwhois 1.2.3, or change your code to use the new API.")

pythonwhois/__init__.py

@@ -1,8 +1,8 @@
-import socket, re
+import socket, re, sys
 from codecs import encode, decode
 from . import shared
 
-def get_whois_raw(domain, server="", previous=[], rfc3490=True, never_cut=False):
+def get_whois_raw(domain, server="", previous=[], rfc3490=True, never_cut=False, with_server_list=False, server_list=[]):
 	# Sometimes IANA simply won't give us the right root WHOIS server
 	exceptions = {
 		".ac.uk": "whois.ja.net",
@@ -12,12 +12,16 @@ def get_whois_raw(domain, server="", previous=[], rfc3490=True, never_cut=False)
 	}
 
 	if rfc3490:
-		domain = encode( domain if type(domain) is unicode else decode(domain, "utf8"), "idna" )
+		if sys.version_info < (3, 0):
+			domain = encode( domain if type(domain) is unicode else decode(domain, "utf8"), "idna" )
+		else:
+			domain = encode(domain, "idna").decode("ascii")
 
-	if len(previous) == 0:
+	if len(previous) == 0 and server == "":
 		# Root query
+		server_list = [] # Otherwise it retains the list on subsequent queries, for some reason.
 		is_exception = False
-		for exception, exc_serv in exceptions.iteritems():
+		for exception, exc_serv in exceptions.items():
 			if domain.endswith(exception):
 				is_exception = True
 				target_server = exc_serv
@@ -26,7 +30,7 @@ def get_whois_raw(domain, server="", previous=[], rfc3490=True, never_cut=False)
 			target_server = get_root_server(domain)
 	else:
 		target_server = server
-	if domain.endswith(".jp") and target_server == "whois.jprs.jp":
+	if target_server == "whois.jprs.jp":
 		request_domain = "%s/e" % domain # Suppress Japanese output
 	elif domain.endswith(".de") and ( target_server == "whois.denic.de" or target_server == "de.whois-servers.net" ):
 		request_domain = "-T dn,ace %s" % domain # regional specific stuff
@@ -52,14 +56,18 @@ def get_whois_raw(domain, server="", previous=[], rfc3490=True, never_cut=False)
 				break
 	if never_cut == False:
 		new_list = [response] + previous
+	server_list.append(target_server)
 	for line in [x.strip() for x in response.splitlines()]:
 		match = re.match("(refer|whois server|referral url|whois server|registrar whois):\s*([^\s]+\.[^\s]+)", line, re.IGNORECASE)
 		if match is not None:
 			referal_server = match.group(2)
-			if referal_server != server:
+			if referal_server != server and "://" not in referal_server: # We want to ignore anything non-WHOIS (eg. HTTP) for now.
 				# Referal to another WHOIS server...
-				return get_whois_raw(domain, referal_server, new_list)
-	return new_list
+				return get_whois_raw(domain, referal_server, new_list, server_list=server_list, with_server_list=with_server_list)
+	if with_server_list:
+		return (new_list, server_list)
+	else:
+		return new_list
 
 def get_root_server(domain):
 	data = whois_request(domain, "whois.iana.org")
@@ -73,11 +81,11 @@ def get_root_server(domain):
 def whois_request(domain, server, port=43):
 	sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 	sock.connect((server, port))
-	sock.send("%s\r\n" % domain)
-	buff = ""
+	sock.send(("%s\r\n" % domain).encode("utf-8"))
+	buff = b""
 	while True:
 		data = sock.recv(1024)
 		if len(data) == 0:
 			break
 		buff += data
-	return buff
+	return buff.decode("utf-8")