2.6 readme.txt initial checking

Author: jar398

doc/Interim-taxonomy-file-format.md

@@ -0,0 +1,71 @@
+This page describes the format used to represent the taxonomies that are the inputs and outputs of the Open Tree of Life taxonomy build system.
+
+The format derives from NCBI and is intentionally rudimentary because our needs are minimal. A better format to use in the long run might be [Darwin Core Archive](https://code.google.com/p/gbif-ecat/wiki/DwCArchive), which is what is used by GBIF, EOL, and the Global Names Architecture (GNA).
+
+***
+
+Each source taxonomy (NCBI, GBIF, Index Fungorum, ...) has its own script that converts its
+native format into this format.
+
+A taxonomy consists of a directory of files with fixed names.  Example: `mycobank/taxonomy.tsv`, `mycobank/synonyms.tsv`, `mycobank/about.md`.
+
+## Character encoding
+
+All files  use the UTF-8 character encoding.  Native taxonomy files often use some other encoding, so conversion might be necessary.  Some aggregated taxonomies on the web have gotten this wrong and are a mess of mixed encodings and spurious re-encodings.
+
+## Taxonomy
+
+### File `taxonomy.tsv`
+
+Four required columns, each column followed by tab - vertical bar - tab (even for the last column, which is unlike NCBI).  The taxonomy build tool 'smasher' doesn't require the vertical bars; they are optional although they should be either all present or all absent.  But some other consumers of these files may still require the vertical bars.
+
+A header row of column names is recommended, but not required (for `Smasher`). If provided, it looks like:
+
+    uid	|	parent_uid	|	name	|	rank	|	
+
+All following rows are one row per taxon
+
+**Columns:**
+
+1. _identifier_ - an identifier for the taxon, unique within this file.  Should be native accession number whenever possible.  Usually this is an integer, but it need not be.
+2. _parent taxon identifier_ or the empty string if there is no parent (i.e., it's a root).
+3. _name_ - arbitrary text for the taxon name; not necessarily unique within the file.
+4. _rank_, e.g. species, family, class.  Should be all lower case.  If no rank is assigned, or the rank is unknown, put "no rank".
+
+Example (from NCBI):
+
+        5157	|	1028423	|	Ceratocystis	|	genus	|	
+        5156	|	91171	|	Gondwanamyces proteae	|	species	|	
+
+**Optional additional columns:**
+
+* _sourceinfo_: a comma-separated list of specifiers, each one either a URL or a CURIE.  If a URL, it should be either a DOI in the form of a URL, or a link to some other source such as a database.  URLs begin 'http://' or 'https://' and DOI URLs begin 'http://dx.doi.org/10.'.  A CURIE is an abbreviated URI using a prefix drawn from a known set, e.g. ncbi:1234 is taxon 1234 in the NCBI taxonomy.  Other prefixes include gbif:, if: (Index Fungorum), mb: (Mycobank). New prefixes can be added but this is a manual process, please request explicitly.
+* _uniqueName_: a human-readable string that is unique to this taxon, typically the taxon name if it is unique, or taxon name followed by "([rank] in [ancestor])" where rank is the taxon's rank and ancestor is an ancestor that is unique to this taxon (among the taxa that have the same name).
+* _flags_: a comma-separated list of flags or markers.  Usually these are generated by taxonomy synthesis and are used to decide whether a taxon is 'hidden' or not.  For example, if there's an 'extinct' flag then it may be desirable to suppress the taxon in an application.  See [here](https://github.com/OpenTreeOfLife/taxomachine/blob/master/src/main/java/org/opentree/taxonomy/OTTFlag.java).
+
+### Synonyms
+
+Usually there are synonyms.  These go into a second file, `synonyms.tsv`.  This file must have a header row
+
+        uid	|	name	|	type	|	rank	|	
+
+The header is necessary because it designates the order of the columns, which can sometimes change. These are the four columns:
+
+* _uid_ - the id for the taxon (from the taxonomy file) that this synonym resolves to
+* _name_ - the synonymic taxon name
+* _type_ - typically will be 'synonym' but could be any of the NCBI synonym types (authority, common name, etc.)
+* _rank_ - currently ignored for taxonomy synthesis.
+
+Example from NCBI:
+
+        89373	|	Flexibacteraceae	|	synonym	|	|	
+
+### Metadata
+
+Overall metadata for the taxonomy is placed in a separate file.  The metadata format is currently under development. `Smasher` generates this in JSON format as `about.json`, but this file is currently not used programmatically, and is in the process of being overhauled. When generating a taxonomy according to this format in external tools, for now it is best to simply write a markdown or plain text file called `about.md` (in the same directory as `taxonomy.tsv` and `synonyms.tsv`).
+
+The metadata provided in the file should include the source of the taxonomy (article or database) as a URL and any other descriptive information that's available.  The purpose of the metadata is not just explanatory but also to explain how to check the correctness of the taxonomy against its source and make corrections and other improvements should the source be updated. When using information from changing sources (databases) the date or dates of retrieval should be recorded.
+
+***
+
+_This page was originally part of the [open tree wiki](https://github.com/OpenTreeOfLife/opentree/wiki/Interim-taxonomy-file-format), and was transferred, since then maintained here on 2014-02-06._

doc/readme.txt

@@ -0,0 +1,118 @@
+This is the Open Tree Taxonomy (OTT) version 2.6, created on 2014-04-11.
+
+The taxonomy was generated using the 'smasher' utility, commit
+f624223f31, which resides on github, here:
+https://github.com/OpenTreeOfLife/reference-taxonomy/commit/f624223f31767fa1787f3ba2ddad5daa56fd939b
+
+File in this package
+====================
+
+ott/taxonomy.tsv
+ott/synonyms.tsv
+
+  The format of these files is described in doc/Interim-taxonomy-file-format.md
+    https://github.com/OpenTreeOfLife/reference-taxonomy/blob/master/doc/Interim-taxonomy-file-format.md
+
+ott/hidden.tsv
+
+  Report on 'hidden' taxa (incertae sedis and other suppressed taxa).
+  Columns are OTT id, name, source taxonomy and id, containing major
+  group, and flags (reasons for hiding).
+
+ott/used-but-hidden.tsv
+
+  Subset of hidden.tsv for taxa that are referenced from source trees.
+  Columns are as for hidden.tsv, with the addition of a column for
+  Phylografter study number.
+
+ott/conflicts.tsv
+
+  Report on taxa that are hidden because they are paraphyletic with
+  respect to a higher priority taxon.  Number at beginning is height
+  in taxonomic tree of nearest common ancestor with priority taxon
+  that 'steals' one or more children.
+
+ott/deprecated.tsv
+
+  List all of taxa that have been deprecated since version 2.5.
+
+ott/log.tsv
+
+  Additional debugging information related to deprecated taxa.
+
+Inputs required to create this version
+======================================
+
+The following information is current as of 2014-05-28.
+
+SILVA
+  Retrieved from
+    https://www.arb-silva.de/fileadmin/silva_databases/release_115/Exports/SSURef_NR99_115_tax_silva.fasta.tgz
+  Last-modified date: 2013-09-07
+  See: Quast C, Pruesse E, Yilmaz P, Gerken J, Schweer T, Yarza P,
+    Peplies J, Glockner FO (2013) The SILVA ribosomal RNA gene
+    database project: improved data processing and web-based tools.
+    Nucleic Acids Research 41 (D1): D590-D596.
+    http://dx.doi.org/10.1093/nar/gks1219
+  Web site: https://www.arb-silva.de/
+
+Taxonomy from Hibbett et al 2007, with updates through 2014
+  Retrieved from
+    http://dx.doi.org/10.6084/m9.figshare.915439
+  Last-modified date: 2014-03-10
+  There is a copy in the git repository.
+  See: A higher-level phylogenetic classification of the Fungi.
+    DS Hibbett, M Binder, JF Bischoff, M Blackwell, et al.
+    Mycological Research 111(5):509-547, 2007.
+    http://dx.doi.org/10.1016/j.mycres.2007.03.004
+
+Index Fungorum
+  We received database table dumps from Paul Kirk in email in
+  November-December 2013, and converted them to the interim taxonomy
+  format using ad hoc scripts.  The converted form is here:
+    http://purl.org/opentree/ott2.6-inputs/tax/if/taxonomy.tsv
+    http://purl.org/opentree/ott2.6-inputs/tax/if/synonyms.tsv
+  Web site: http://www.indexfungorum.org/
+
+Lamiales taxonomy from Schaferhof et al 2010
+  File prepared from figure by Open Tree of Life staff.
+  There is a copy in the git repository.
+  See:
+    Schaferhoff, B., Fleischmann, A., Fischer, E., Albach, D. C.,
+    Borsch, T., Heubl, G., and Muller, K. F. (2010). Towards resolving
+    Lamiales relationships: insights from rapidly evolving chloroplast
+    sequences. BMC evolutionary biology 10(1), 352.
+    http://dx.doi.org/10.1186/1471-2148-10-352
+
+NCBI Taxonomy
+  Retrieved from ftp://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz
+  Last-modified date: 2014-01-06
+  As far as we can tell, NCBI does not archive past versions of its
+  taxonomy.  We have captured the applicable version here:
+    http://purl.org/opentree/ott2.6-inputs/feed/ncbi/taxdump.tar.gz
+  Web site: https://www.ncbi.nlm.nih.gov/taxonomy
+
+GBIF backbone taxonomy
+  Retrieved from http://ecat-dev.gbif.org/repository/export/checklist1.zip
+  Last-modified date: 2013-07-02
+  GBIF intends to reorganize their data archives and this file may
+  move.  We will make a best effort to maintain the following PURL:
+    http://purl.org/opentree/gbif-backbone-2013-07-02.zip
+  In case these links do not work, search gbif.org data sets and use the
+  following file information for confirmation:
+    Size: 323093992 bytes
+    sha1sum: b7c7c19f1835af3f424ce4f2c086c692c1818b90
+    Format: Zip file containing a Darwin Core Archive
+    Contains file taxon.txt which has 4416348 lines
+  Web site: http://www.gbif.org/dataset/d7dddbf4-2cf0-4f39-9b2a-bb099caae36c
+
+IRMNG (Interim Register of Marine and Nonmarine Genera)
+  Retrieved from http://www.cmar.csiro.au/datacentre/downloads/IRMNG_DWC.zip
+  Last-modified date: 2014-01-12
+    http://purl.org/opentree/ott2.6-inputs/feed/irmng/in/IRMNG_DWC.zip
+  Web site: http://www.obis.org.au/irmng/
+
+OTT version 2.5
+  The previous version of OTT is used only for the purpose of ensuring
+  identifier choice consistency from one version of OTT to the next.
+    http://files.opentreeoflife.org/ott/ott2.5.tgz