Add a chapter on compliance to the docs

Details how to filter warnings or convert them into exceptions and
details the class hierarchy of warnings and errors defined by
compoundfiles.

Author: waveform80

README.rst

@@ -4,16 +4,18 @@ compoundfiles
 
 |pypi| |rtd| |travis|
 
-This package provides a library for reading Microsoft's `OLE Compound
-Document`_ format, which also forms the basis of the `Advanced Authoring
-Format`_ (AAF) published by Microsoft Corporation. It is compatible with
-Python 2.7 (or above) and Python 3.2 (or above).
-
-The code is pure Python and should run on any platform. The library has an
-emphasis on rigour and performs numerous validity checks on opened files.  By
-default, the library merely warnings when it comes across non-fatal errors in
-source files but this behaviour is configurable by developers through Python's
-``warnings`` mechanisms.
+This package provides a library for reading Microsoft's `Compound File Binary`_
+format (CFB), formerly known as `OLE Compound Documents`_, the `Advanced
+Authoring Format`_ (AAF), or just plain old Microsoft Office files (the non-XML
+sort). This format is also widely used with certain media systems and a number
+of scientific applications (tomography and microscopy).
+
+The code is pure Python and should run on any platform; it is compatible with
+Python 2.7 (or above) and Python 3.2 (or above). The library has an emphasis
+on rigour and performs numerous validity checks on opened files.  By default,
+the library merely warns when it comes across non-fatal errors in source files
+but this behaviour is configurable by developers through Python's ``warnings``
+mechanisms.
 
 Links
 =====
@@ -28,7 +30,8 @@ Links
 .. _documentation: http://compound-files.readthedocs.org/
 .. _source code: https://github.com/waveform80/compoundfiles
 .. _bug tracker: https://github.com/waveform80/compoundfiles/issues
-.. _OLE Compound Document: http://www.openoffice.org/sc/compdocfileformat.pdf
+.. _Compound File Binary: http://msdn.microsoft.com/en-gb/library/dd942138.aspx
+.. _OLE Compound Documents: http://www.openoffice.org/sc/compdocfileformat.pdf
 .. _Advanced Authoring Format: http://www.amwa.tv/downloads/specifications/aafcontainerspec-v1.0.1.pdf
 .. _MIT license: http://opensource.org/licenses/MIT
 .. _build status: https://travis-ci.org/waveform80/compoundfiles

compoundfiles/reader.py

@@ -77,25 +77,24 @@
     )
 
 
-# A quick personal rant: the AAF or OLE Compound Document format is yet another
-# example of bad implementations of a bad specification (thanks Microsoft! See
-# the W3C log file format for previous examples of MS' incompetence in this
-# area)...
+# Good grief! Since my last in-source rant it appears someone in MS actually
+# figured out how to write a decent spec! Unfortunately it appears someone in
+# the marketing department also thought that yet another name change was in
+# order so the Advanced Authoring Format (formerly known as OLE Compound
+# Documents) is now known as the Compound File Binary File Format.
 #
-# The specification doesn't try and keep the design simple (the DIFAT could be
-# fully in the header or partially in the header, and the header itself doesn't
-# necessarily match the sector size), whoever wrote the spec didn't quite
-# understand what version numbers are used for (several versions exist, but the
-# spec doesn't specify exactly which bits of the header became relevant in
-# which versions), and the spec has huge amounts of redundancy (always fun as
-# it inevitably leads to implementations getting one bit right and another bit
-# wrong, leaving readers to guess which is correct).
+# Anyway, silly name changes aside, the point is that someone's actually
+# written a decent spec this time rather than the half-assed AAF spec which
+# read like adhoc notes on a reference implementation. The URL is (currently)
 #
-# TL;DR: if you're looking for a nice fast binary format with good random
-# access characteristics this may look attractive, but please don't use it.
-# Ideally, loop-mounting a proper file-system would be the way to go, although
-# it generally involves jumping through several hoops due to mount being a
-# privileged operation.</rant>
+#   http://msdn.microsoft.com/en-gb/library/dd942138.aspx
+#
+# But given how MSDN changes its URLs you might just be better off Googling for
+# "MS CFB" which'll find it (assuming they haven't changed the name again for
+# kicks). The file format is still a pile of steaming underwear in places
+# (unicode names with byte-length fields...) but as long as the spec is clear
+# and well written I can forgive that (after all, it's hard to change something
+# as established as this).
 #
 # In the interests of trying to keep naming vaguely consistent and sensible
 # here's a translation list with the names we'll be using first and the names

docs/Makefile

@@ -6,6 +6,10 @@ SPHINXOPTS    =
 SPHINXBUILD   = sphinx-build
 PAPER         =
 BUILDDIR      = _build
+DOT_DIAGRAMS  = $(wildcard *.dot)
+MSC_DIAGRAMS  = $(wildcard *.mscgen)
+SVG_IMAGES    = $(wildcard *.svg) $(DOT_DIAGRAMS:%.dot=%.svg) $(MSC_DIAGRAMS:%.mscgen=%.svg)
+PDF_IMAGES    = $(SVG_IMAGES:%.svg=%.pdf)
 
 # Internal variables.
 PAPEROPT_a4     = -D latex_paper_size=a4
@@ -41,17 +45,17 @@ help:
 clean:
 	-rm -rf $(BUILDDIR)/*
 
-html:
+html: $(SVG_IMAGES)
 	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
 	@echo
 	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
 
-dirhtml:
+dirhtml: $(SVG_IMAGES)
 	$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
 	@echo
 	@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
 
-singlehtml:
+singlehtml: $(SVG_IMAGES)
 	$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
 	@echo
 	@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
@@ -95,14 +99,14 @@ epub:
 	@echo
 	@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
 
-latex:
+latex: $(PDF_IMAGES)
 	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
 	@echo
 	@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
 	@echo "Run \`make' in that directory to run these through (pdf)latex" \
 	      "(use \`make latexpdf' here to do that automatically)."
 
-latexpdf:
+latexpdf: $(PDF_IMAGES)
 	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
 	@echo "Running LaTeX files through pdflatex..."
 	$(MAKE) -C $(BUILDDIR)/latex all-pdf
@@ -151,3 +155,13 @@ doctest:
 	$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
 	@echo "Testing of doctests in the sources finished, look at the " \
 	      "results in $(BUILDDIR)/doctest/output.txt."
+
+%.svg: %.msc
+	mscgen -T svg -o $@ $<
+
+%.svg: %.dot
+	dot -T svg -o $@ $<
+
+%.pdf: %.svg
+	inkscape -A $@ $<
+

docs/compliance.rst

@@ -0,0 +1,75 @@
+.. _compliance:
+
+=====================
+Compliance mechanisms
+=====================
+
+As noted in the `CFB`_ specification, the compound document format presents a
+number of validation challenges. For example, maliciously constructed files
+might include circular references in their FAT table, leading a naive reader
+into an infinite loop, or they may allocate a large number of DIFAT sectors
+hoping to cause resource exhaustion when the reader goes to allocate memory for
+reading the FAT.
+
+The compoundfiles library goes to some lengths to detect erroneous structures
+(whether malicious in intent or otherwise) and work around them where possible.
+Some issues are considered fatal and will always raise an exception (circular
+chains in the FAT are an example of this). Other issues are considered
+non-fatal and will raise a warning (unusual sector sizes are an example of
+this). Python :mod:`warnings` are a special sort of exception with particularly
+flexible handling.
+
+With Python's defaults, a specific warning will print a message to the console
+the first time it is encountered and will then do nothing if it's encountered
+again (this avoids spamming the console in case a warning is raised in a tight
+loop). With some simple code, you can specify alternative behaviours: warnings
+can be raised as full-blown exceptions, or suppressed entirely. The
+compoundfiles library defines a large hierarchy of errors and warnings to
+enable developers to finetune their handling.
+
+For example, consider a developer writing an application for working with
+computed tomography (CT) scans. The files produced by the scanner's software
+are compound documents, but they use an unusual sector size. Whenever the
+developer's Python script opens a file the following warning is emitted::
+
+    /usr/lib/pyshared/python2.7/compoundfiles/compoundfiles/reader.py:275: CompoundFileSectorSizeWarning: unexpected sector size in v3 file (1024)
+
+Other than this, the script runs successfully. The developer decides the
+warning is unimportant (after all there's nothing he can do about it given he
+can't change the scanner's software) and wishes to suppress it entirely, so he
+adds the following line to the top of his script::
+
+    import warnings
+    import compoundfiles as cf
+
+    warnings.filterwarnings('ignore', category=cf.CompoundFileSectorSizeWarning)
+
+Another developer is working on a file validation service. She wishes to use
+the compoundfiles library to extract and examine the contents of such files.
+For safety, she decides to treat any violation of the specification as an
+error, so she adds the following line to the top of her script to tell Python
+to convert all compound file warnings into exceptions::
+
+    import warnings
+    import compoundfiles as cf
+
+    warnings.filterwarnings('error', category=cf.CompoundFileWarning)
+
+The class hierarchies for compoundfiles warnings and errors is illustrated
+below:
+
+.. image:: warnings.*
+    :align: center
+
+.. image:: errors.*
+    :align: center
+
+To set filters on all warnings in the hierarchy, simply use the category
+:exc:`~compoundfiles.CompoundFileWarning`. Otherwise, you can use intermediate
+or leaf classes in the hierarchy for more specific filters. Likewise, when
+catching exceptions you can target the root of the hierarchy
+(:exc:`~compoundfiles.CompoundFileError`) to catch any error that the
+compoundfiles library might raise, or a more specific class to deal with a
+particular error.
+
+.. _CFB: http://msdn.microsoft.com/en-gb/library/dd942138.aspx

docs/errors.dot

@@ -0,0 +1,22 @@
+digraph G {
+    graph [rankdir="LR"];
+
+    node [shape=rect,style=filled,color="#000000",fillcolor="#99aadd",fontname=Arial,fontsize=12.0];
+    CompoundFileError->IOError;
+    CompoundFileHeaderError->CompoundFileError;
+    CompoundFileMasterFatError->CompoundFileError;
+    CompoundFileNormalFatError->CompoundFileError;
+    CompoundFileMiniFatError->CompoundFileError;
+    CompoundFileDirEntryError->CompoundFileError;
+    CompoundFileInvalidMagicError->CompoundFileHeaderError;
+    CompoundFileInvalidBomError->CompoundFileHeaderError;
+    CompoundFileLargeNormalFatError->CompoundFileNormalFatError;
+    CompoundFileNormalLoopError->CompoundFileNormalFatError;
+    CompoundFileLargeMiniFatError->CompoundFileMiniFatError;
+    CompoundFileNoMiniFatError->CompoundFileMiniFatError;
+    CompoundFileMasterLoopError->CompoundFileMasterFatError;
+    CompoundFileDirLoopError->CompoundFileDirEntryError;
+    CompoundFileNotFoundError->CompoundFileError;
+    CompoundFileNotStreamError->CompoundFileError;
+}
+