0.2.0: Changed from scraping to using inventory files. Closes #1.

Author: jackmaney

MANIFEST.in

@@ -1,3 +1,3 @@
-recursive-include stdlib_list/lists *.csv
+recursive-include stdlib_list/lists *.txt
 include README.rst
 include LICENSE

README.md

@@ -8,16 +8,17 @@ Listing the modules in the standard library? Wait, why on Earth would you care a
 
 Because knowing whether or not a module is part of the standard library will come in handy in [a project of mine](https://github.com/jackmaney/pypt). [And I'm not the only one](http://stackoverflow.com/questions/6463918/how-can-i-get-a-list-of-all-the-python-standard-library-modules) who would find this useful. Or, the TL;DR answer is that it's handy in situations when you're analyzing Python code and would like to find module dependencies.
 
-After googling for a way to generate a list of Python standard libraries (and looking through the answers to the previously-linked Stack Overflow question), I decided that I didn't like the existing solutions. So, I used Beautiful Soup 4 to build a scraper and scraped the list of libraries (along with a bit of metadata) from the Python Module Index.
+After googling for a way to generate a list of Python standard libraries (and looking through the answers to the previously-linked Stack Overflow question), I decided that I didn't like the existing solutions. So, I started by writing a scraper for the TOC of the Python Module Index for each of the versions of Python above.
+
+However, web scraping can be a fragile affair. Thanks to [a suggestion](https://github.com/jackmaney/python-stdlib-list/issues/1#issuecomment-86517208) by [@ncoghlan](https://github.com/ncoghlan), and some further help from [@birkenfeld](https://github.com/birkenfeld) and [@epc](https://github.com/epc), the population of the lists is now done by grabbing and parsing the Sphinx object inventory for the official Python docs of each relevant version.
 
 Usage
 -----
 
-```
->>> from stdlib_list import stdlib_list
->>> libraries = stdlib_list("2.7")
->>> libraries[:10]
-['AL', 'BaseHTTPServer', 'Bastion', 'CGIHTTPServer', 'ColorPicker', 'ConfigParser', 'Cookie', 'DEVICE', 'DocXMLRPCServer', 'EasyDialogs']
-```
+    >>> from stdlib_list import stdlib_list
+    >>> libraries = stdlib_list("2.7")
+    >>> libraries[:10]
+    ['AL', 'BaseHTTPServer', 'Bastion', 'CGIHTTPServer', 'ColorPicker', 'ConfigParser', 'Cookie', 'DEVICE', 'DocXMLRPCServer', 'EasyDialogs']
 
 For more details, check out [the docs](http://python-stdlib-list.readthedocs.org/en/latest/).
+

README.rst

@@ -8,7 +8,9 @@ Listing the modules in the standard library? Wait, why on Earth would you care a
 
 Because knowing whether or not a module is part of the standard library will come in handy in `a project of mine <https://github.com/jackmaney/pypt>`_. `And I'm not the only one <http://stackoverflow.com/questions/6463918/how-can-i-get-a-list-of-all-the-python-standard-library-modules>`_ who would find this useful. Or, the TL;DR answer is that it's handy in situations when you're analyzing Python code and would like to find module dependencies.
 
-After googling for a way to generate a list of Python standard libraries (and looking through the answers to the previously-linked Stack Overflow question), I decided that I didn't like the existing solutions. So, I used Beautiful Soup 4 to build a scraper and scraped the list of libraries (along with a bit of metadata) from the Python Module Index.
+After googling for a way to generate a list of Python standard libraries (and looking through the answers to the previously-linked Stack Overflow question), I decided that I didn't like the existing solutions. So, I started by writing a scraper for the TOC of the Python Module Index for each of the versions of Python above.
+
+However, web scraping can be a fragile affair. Thanks to `a suggestion <https://github.com/jackmaney/python-stdlib-list/issues/1#issuecomment-86517208>`_ by `@ncoghlan <https://github.com/ncoghlan>`_, and some further help from `@birkenfeld <https://github.com/birkenfeld>`_ and `@epc <https://github.com/epc>`_, the population of the lists is now done by grabbing and parsing the Sphinx object inventory for the official Python docs of each relevant version.
 
 Usage
 =====

docs/fetch.rst

@@ -0,0 +1,8 @@
+Fetching The Module Lists
+=========================
+
+The lists of modules themselves are grabbed from the Sphinx object inventory (ie the file used by :py:mod`sphinx.ext.intersphinx` in order to build links to existing Python modules/functions/etc). You probably shouldn't need to mess around with it. But if you want to, here you go.
+
+
+.. automodule:: stdlib_list.fetch
+    :members: fetch_list

docs/index.rst

@@ -13,9 +13,9 @@ Listing the modules in the standard library? Wait, why on Earth would you care a
 
 Because knowing whether or not a module is part of the standard library will come in handy in `a project of mine <https://github.com/jackmaney/pypt>`_. `And I'm not the only one <http://stackoverflow.com/questions/6463918/how-can-i-get-a-list-of-all-the-python-standard-library-modules>`_ who would find this useful. Or, the TL;DR answer is that it's handy in situations when you're analyzing Python code and would like to find module dependencies.
 
-After googling for a way to generate a list of Python standard libraries (and looking through the answers to the previously-linked Stack Overflow question), I decided that I didn't like the existing solutions.
+After googling for a way to generate a list of Python standard libraries (and looking through the answers to the previously-linked Stack Overflow question), I decided that I didn't like the existing solutions. So, I started by writing a scraper for the TOC of the Python Module Index for each of the versions of Python above.
 
-So, I decided to brute force it a bit, by putting together a scraper for the TOC of the standard library page of the official Python docs (eg `here is the page for Python 2.7 <https://docs.python.org/2/library/index.html>`_).
+However, web scraping can be a fragile affair. Thanks to `a suggestion <https://github.com/jackmaney/python-stdlib-list/issues/1#issuecomment-86517208>`_ by `@ncoghlan <https://github.com/ncoghlan>`_, and some further help from `@birkenfeld <https://github.com/birkenfeld>`_ and `@epc <https://github.com/epc>`_, the population of the lists is now done by grabbing and parsing the Sphinx object inventory for the official Python docs of each relevant version.
 
 Contents
 ========
@@ -25,8 +25,7 @@ Contents
 
    install
    usage
-   metadata
-   scraper
+   fetch
 
 
 

docs/metadata.rst

@@ -3,5 +3,17 @@ Usage (Or: How To Get The List of Libraries)
 
 The primary function that you'll care about in this package is ``stdlib_list.stdlib_list``.
 
+In particular:
+
+::
+
+    In [1]: from stdlib_list import stdlib_list
+
+    In [2]: libs = stdlib_list("3.4")
+
+    In [3]: libs[:6]
+    Out[3]: ['__future__', '__main__', '_dummy_thread', '_thread', 'abc', 'aifc']
+
+
 .. automodule:: stdlib_list
     :members: stdlib_list

docs/scraper.rst

@@ -1,3 +1,2 @@
 -r requirements.txt
-sphinx
 sphinx_rtd_theme

docs/usage.rst

@@ -1,2 +1 @@
-beautifulsoup4
-requests
+sphinx