DOC - working on data package usecases

Author: matthew-brett

doc/source/devel/data_pkg_discuss.rst

@@ -1,8 +1,8 @@
 .. _data-package-discuss:
 
-#############################
-Principles of data package NG
-#############################
+##########################
+Principles of data package
+##########################
 
 **********
 Motivation
@@ -36,7 +36,7 @@ separable.
 Package
 =======
 
-This ideas is rather difficult to define, but is a bit like a data project, that
+This idea is rather difficult to define, but is a bit like a data project, that
 is a set of information that the packager believed had something in common.  The
 package then is an abstract idea, and what is in the package could change
 completely over course of the life of the package.  The package then is a little
@@ -59,14 +59,14 @@ moment, whether this has been committed or not.
 
 It might not be enjoyable, but we'll call a package instantiation a *pinstance*.
 
-Package instantiation revision
-==============================
+Pinstance revision
+==================
 
 A revision is an instantiation of the working tree that has a unique label - the
 *revision id*.
 
-Package revision id
-===================
+Pinstance revision id
+=====================
 
 The *revision id* is a string that identifies a particular *pinstance*.  This is
 the equivalent of the revision number in subversion_, or the commit hash in
@@ -76,8 +76,8 @@ example, you might have a revision of id '200', delete a file, restore the file,
 call this revision id '201', but they might both refer to the same instantiation
 of the package.  Or they might not, that's up to you, the author of the package.
 
-Package instantiation tag
-=========================
+Pinstance tag
+=============
 
 A *tag* is a memorable string that refers to a particular pinstance.  It differs
 from a revision id only in that there is not likely to be a tag for every
@@ -91,6 +91,14 @@ equivalent of applying a git tag).  ``release-0.3`` is the tag and ``af5bd6``
 is the revision id.  Different sources of the same package might possibly
 produce different tags [#tag-sources]_
 
+Pinstance version
+=================
+
+A *pinstance* might also have a version.  A version is just a tag that can be
+compared using some algorithm.
+
+.. _prundle:
+
 Package provider bundle
 =======================
 
@@ -127,6 +135,8 @@ A release might be a package instantiation that one person has:
 #. tagged
 #. made available as one or more *provider bundles*
 
+.. _prundle-discovery:
+
 Prundle discovery
 =================
 

doc/source/devel/data_pkg_uses.rst

@@ -10,65 +10,74 @@ Usecases
 
 We are here working from :doc:`data_pkg_discuss`
 
-Distribution types
-==================
+Prundles
+========
 
-The most common type of distribution is a *file-system* distribution, that is,
-a distribution that can return content from passed filenames, as if it were a
-file-system.  The examples below should make this clearer.
+See :ref:`prundle`.
 
-An *local path* distribution is a file-system distribution with content stored
-in files in a directory accessible on the local filesystem.  We could also call
-this a *local path* distribution type.
+An *local path* format prundle is a directory on the local file system with prundle data stored in files in a
+on the local filesystem.
 
 Examples
 ========
 
-Create distribution
--------------------
+We'll call our package `dang` - data package new generation.
 
-::
+Create local-path prundle
+-------------------------
 
-    >>> import os
-    >>> import dpkg
-    >>> dst = dpkg.LocalPathDistribution.initialize(name='my-package', path=/a/path')
-    >>> dst.name
-    'my-package'
-    >>> dst.version
-    '0'
-    >>> dst.path == os.path.abspath(/a/path')
-    True
-    >>> os.listdir(/a/path')
-    ['meta.ini']
+>>> import os
+>>> import tempfile
+>>> pth = tempfile.mkdtemp() # temporary directory
 
-The local path distribution here is just the set of files in the ``a/path`` directory.
+Make a pinstance object
 
-The call to the ``initialize`` class method above creates the directory if it
-does not exist, and writes a bare ``meta.ini`` file to the directory, with the
-given ``name``, and default version of ``0``.
+>>> from dang import Pinstance
+>>> pri = Prundle(name='my-package')
+>>> pri.pkg_name
+'my-package'
+>>> pri.meta
+{}
 
-Use local path distribution
----------------------------
+Now we make a prundle.   First a directory to contain it
 
-::
+>>> import os
+>>> import tempfile
+>>> pth = tempfile.mkdtemp() # temporary directory
 
-    >>> dst = dpkg.LocalPathDistribution.from_path(path=/a/path')
-    >>> dst.name
-    'my-package'
+>>> from dang.prundle import LocalPathPrundle
+>>> prun = LocalPathPrundle(pri, pth)
+
+At the moment there's nothing in the directory.  The 'write' method will write
+the meta information - here just the package name.
+
+>>> prun.write() # writes meta.ini file
+>>> os.listdir(pth)
+['meta.ini']
+
+The local path prundle data is just the set of files in the temporary directory
+named in ``pth`` above.
+
+Now we've written the package, we can get it by a single call that reads in the
+``meta.ini`` file:
 
-Getting content
----------------
+>>> prun_back = LocalPathPrundle.from_path(pth)
+>>> prun_back.pkg_name
+'my-package'
 
-The file-system distribution types can return content by file names.
+Getting prundle data
+--------------------
 
-For example, for the local path ``dst`` distribution objects we have seen so
+The file-system prundle formats can return content by file names.
+
+For example, for the local path ``prun`` distribution objects we have seen so
 far, the following should work::
 
-    >>> fobj = dst.get_fileobj('a_file.txt')
+    >>> fobj = prun.get_fileobj('a_file.txt')
 
 In fact, local path distribution objects also have a ``path`` attribute::
 
-    >>> fname = os.path.join(dst.path, 'a_file.txt')
+    >>> fname = os.path.join(prun.path, 'a_file.txt')
 
 The ``path`` attribute might not make sense for objects with greater abstraction
 over the file-system - for example objects encapsulating web content.
@@ -77,38 +86,39 @@ over the file-system - for example objects encapsulating web content.
 Discovery
 *********
 
-So far we only have distribution objects.  In order for a program to use a
-distribution object it has to know where the distribution is.
+So far, in order to create a prundle object, we have to know where the prundle
+is (the path).
+
+We want to be able to tell the system where prundles are - and the system will
+then be able to return a prundle on request - perhaps by package name.  The
+system here is answering a :ref:`prundle-discovery` query.
 
-We will then want to ask our packaging system whether it knows about the
-distribution we are interested in.  This is a *discovery query*.
+We will then want to ask our packaging system whether it knows about the prundle
+we are interested in.
 
 Discovery sources
 =================
 
 A discovery source is an object that can answer a discovery query.
 Specifically, it is an object with a ``discover`` method, like this::
 
-    >>> dsrc = dpkg.get_source('local-system')
+    >>> import dang
+    >>> dsrc = dang.get_source('local-system')
     >>> dquery_result = dsrc.discover('my-package', version='0')
-    >>> dquery_result.distribution.name
+    >>> dquery_result[0].pkg_name
     'my-package'
     >>> dquery_result = dsrc.discover('implausible-pkg', version='0')
-    >>> dquery_result.distribution is None
-    True
-
-The ``discover`` method returns a discovery query result.  This result contains
-a distribution object if it knows about the distribution with the given name and
-version; the distribution in the query is None otherwise.
+    >>> len(dquery_result)
+    0
 
 The discovery version number spec may allow comparison operators, as for
 ``distutils.version.LooseVersion``::
 
     >>> res = dsrc.discover(name='my-package', version='>=0')
-    >>> dst = rst.distribution
-    >>> dst.name
+    >>> prun = rst[0]
+    >>> prun.pkg_name
     'my-package'
-    >>> dst.version
+    >>> prun.meta['version']
     '0'
 
 Default discovery sources
@@ -137,51 +147,50 @@ from a list of sources.  Something like this::
     >>> local_usr = dpkg.get_source('local-user')
     >>> src_pool = dpkg.SourcePool((local_usr, local_sys))
     >>> dq_res = src_pool.discover('my-package', version='0')
-    >>> dq_res.distribution.name
+    >>> dq_res[0].pkg_name
     'my-package'
 
 We'll often want to do exactly this, so we'll add this source pool to those that
 can be returned from our ``get_source`` convenience function::
 
     >>> src_pool = dpkg.get_source('local-pool')
 
-Register a distribution
-=======================
+Register a prundle
+==================
 
-In order to register a distribution, we need a distribution object and a
+In order to register a prundle, we need a prundle object and a
 discovery source::
 
-    >>> dst = dpkg.LocalPathDistribution.from_path(path=/a/path')
-    >>> local_usr = dpkg.get_source('local-user')
-    >>> local_usr.register(dst)
+    >>> from dang.prundle import LocalPathPrundle
+    >>> prun = LocalPathDistribution.from_path(path=/a/path')
+    >>> local_usr = dang.get_source('local-user')
+    >>> local_usr.register(prun)
 
 Let us then write the source to disk::
 
-    >>> local_usr.save()
+    >>> local_usr.write()
 
 Now, when we start another process as the same user, we can do this::
 
-    >>> import dpkg
-    >>> local_usr = dpkg.get_source('local-user')
-    >>> dst = local_usr.discover('my-package', '0')
+    >>> import dang
+    >>> local_usr = dang.get_source('local-user')
+    >>> prun = local_usr.discover('my-package', '0')[0]
 
 **************
 Implementation
 **************
 
 Here are some notes.  We had the hope that we could implement something that
 would be simple enough that someone using the system would not need our code,
-but could work from the specification.  In practice we hope to be able get away
-with something that uses ``ini`` format files as base storage, because these are
-fairly standard and have support in the python standard library since way back.
+but could work from the specification.
 
-Local path distributions
-========================
+Local path prundles
+===================
 
-As implied above, these are directories accessible on the local filesystem.
-The directory needs to give information about the distribution name and version.
-An ``ini`` file is probably enough for this - something like a ``meta.ini`` file
-in the directory with::
+These are directories accessible on the local filesystem.  The directory needs
+to give information about the prundle name and optionally, version, tag,
+revision id and maybe other metadata.  An ``ini`` file is probably enough for
+this - something like a ``meta.ini`` file in the directory with::
 
     [DEFAULT]
     name = my-package
@@ -192,11 +201,8 @@ might be enough to get started.
 Discovery sources
 =================
 
-The discovery source has to be able to return distribution objects for the
-distributions it knows about.  A discovery source might only be able to handle
-local path distributions, in which case all it needs to know about a
-distribution is the (name, version, path).  So, a local path discovery source
-could be stored on disk as an ``ini`` file as well::
+The discovery source has to be able to return prundle objects for the
+prundles it knows about.
 
     [my-package]
     0 = /some/path