Merge pull request #1656 from benjeffery/64bit-wrapup

Changelogs and versions for 64bit offset changes

Author: mergify[bot]

c/CHANGELOG.rst

@@ -2,6 +2,18 @@
 [0.99.14] - 2021-0X-XX
 ----------------------
 
+**Breaking changes**
+
+- 64 bits are now used to store the sizes of ragged table columns such as metadata,
+  allowing them to hold more data. As such ``tsk_size_t`` is now 64 bits wide.
+  This change is fully backwards and forwards compatible for all tree-sequences whose
+  ragged column sizes fit into 32 bits. New tree-sequences with
+  large offset arrays that require 64 bits will fail to load in previous versions with
+  error ``TSK_ERR_BAD_COLUMN_TYPE``.
+  (:user:`jeromekelleher`, :issue:`343`, :issue:`1527`, :issue:`1528`, :issue:`1530`,
+  :issue:`1554`, :issue:`1573`, :issue:`1589`,:issue:`1598`,:issue:`1628`, :pr:`1571`,
+  :pr:`1579`, :pr:`1585`, :pr:`1590`, :pr:`1602`, :pr:`1618`, :pr:`1620`, :pr:`1652`).
+
 **Features**
 
 - Add `tsk_X_table_update_row` methods which allow modifying single rows of tables

c/tskit/core.h

@@ -197,7 +197,7 @@ to the API or ABI are introduced, i.e., internal refactors of bugfixes.
 #define TSK_FILE_FORMAT_NAME          "tskit.trees"
 #define TSK_FILE_FORMAT_NAME_LENGTH   11
 #define TSK_FILE_FORMAT_VERSION_MAJOR 12
-#define TSK_FILE_FORMAT_VERSION_MINOR 4
+#define TSK_FILE_FORMAT_VERSION_MINOR 5
 
 /**
 @defgroup GENERAL_ERROR_GROUP General errors.

python/CHANGELOG.rst

@@ -7,6 +7,16 @@
 - The CLI `info` command now gives more detailed information on the tree sequence
   (:user:`benjeffery`, :pr:`1611`)
 
+- 64 bits are now used to store the sizes of ragged table columns such as metadata,
+  allowing them to hold more data. This change is fully backwards and forwards compatible
+  for all tree-sequences whose ragged column sizes fit into 32 bits. New tree-sequences with
+  large offset arrays that require 64 bits will fail to load in previous versions with
+  error ``_tskit.FileFormatError: An incompatible type for a column was found in the
+  file``.
+  (:user:`jeromekelleher`, :issue:`343`, :issue:`1527`, :issue:`1528`, :issue:`1530`,
+  :issue:`1554`, :issue:`1573`, :issue:`1589`,:issue:`1598`,:issue:`1628`, :pr:`1571`,
+  :pr:`1579`, :pr:`1585`, :pr:`1590`, :pr:`1602`, :pr:`1618`, :pr:`1620`, :pr:`1652`).
+
 **Features**
 
 - Add `__setitem__` to all tables allowing single rows to be updated. For example

python/lwt_interface/CHANGELOG.rst

@@ -1,3 +1,15 @@
+--------------------
+[0.1.4] - 2021-09-02
+--------------------
+
+- Offset columns are now 64 bit in tskit. For compatibility, offset arrays that fit into
+  32bits will be a 32bit array in the dict encoding. Large arrays that require 64 bit
+  will fail to ``fromdict`` in previous versions with the error:
+  ``TypeError: Cannot cast array data from dtype('uint64') to dtype('uint32') according
+  to the rule 'safe'``
+  A ``force_offset_64`` option on ``asdict`` allows the easy creation of 64bit arrays for
+  testing.
+
 --------------------
 [0.1.3] - 2021-02-01
 --------------------

python/tests/test_file_format.py

@@ -43,7 +43,7 @@
 
 
 CURRENT_FILE_MAJOR = 12
-CURRENT_FILE_MINOR = 4
+CURRENT_FILE_MINOR = 5
 
 test_data_dir = os.path.join(os.path.dirname(__file__), "data")