Merge pull request #992 from htm-community/Python311

python 3.11 compatability

Author: dkeeney

README.md

@@ -40,44 +40,39 @@ in C++ library.
 
 ## Installation
 
-### Binary releases
 
-NOTE: The Binary releases are not being maintained. Suggest building from Sources instead.
-
-If you want to use `htm.core` from Python, the easiest method is to install from [PyPI](https://test.pypi.org/project/htm.core/)
-  - Note: to install from `pip` you'll need Python 3.7 only(does not work with older or newer versions)
-
-```
-python -m pip install -i https://test.pypi.org/simple/ htm.core
-```
-Note: to run all examples with visualizations, install including extra requirements:
-`pip install -i https://test.pypi.org/simple/ htm.core[examples]`
-
-If you intend to use `htm.core` as a library that provides you Python \& C++ HTM, 
-you can use our [binary releases](https://github.com/htm-community/htm.core/releases).
-
-#### Prerequisites
+### Prerequisites
 
 For running C++ apps/examples/tests from binary release: none. 
 If you want to use python, then obviously:
 
 - [Python](https://python.org/downloads/)
-    - Standard Python 3.7+ (Recommended)
+    - Standard Python 3.7+ (Recommend using the latest)  [Tested with 3.8, 3.11.1]
     - Standard Python 2.7
       + We recommend the latest version of 2.7 where possible, but the system version should be fine.
       + Python 2 is Not Supported on Windows, use Python 3 instead.
       + Python 2 is not tested by our CI anomore. It may still work but we don't test it. We expect to drop support for Python2 around 2020.
     - [Anaconda Python](https://www.anaconda.com/products/individual#Downloads) 3.7+
       + On windows you must run from within 'Anaconda Prompt' not 'Command Prompt'.
-      + The pre-built binary releases only work with Standard Python so you must build from sources.
       + Anaconda Python is not tested in our CI.
 
   Be sure that your Python executable is in the Path environment variable.
   The Python that is in your default path is the one that will determine which
   version of Python the extension library will be built for.
   - Other implementations of Python may not work.
   - Only the standard python from python.org have been tested.
+  - On Linux you will need both the Python install and the Python-dev install
+        '$ sudo apt install python3.11'
+        '$ sudo apt install python3.11-dev'
+  - You will probably also want to setup a [https://docs.python.org/3/library/venv.html(venv environment).
+
+- **C\+\+ compiler**: c\+\+11/17 compatible (ie. g++, clang\+\+).
+  - boost library (if not a C\+\+17 or greater compiler that supports filesystem.) 
+    If the build needs boost, it will automatically download and install boost with the options it needs.
+  - CMake 3.7+  (MSVC 2019 needs CMake 3.14+, MSVC 2022 needs CMake 3.21+).  
+    Install the latest using [https://cmake.org/download/](https://cmake.org/download/)
 
+Note: Windows MSVC 2019 runs as C\+\+17 by default so boost is not needed.  On linux use -std=c++17 compile option to avoid needing boost.
 
 ### Building from Source
 
@@ -91,29 +86,21 @@ To fork the repo with `git`:
 git clone https://github.com/htm-community/htm.core
 ```
 
-#### Prerequisites
-
-- same as for Binary releases, plus:
-- **C\+\+ compiler**: c\+\+11/17 compatible (ie. g++, clang\+\+).
-- boost library (if not a C\+\+17 or greater compiler that supports filesystem.) 
-  If the build needs boost, it will automatically download and install boost with the options it needs.
-- CMake 3.7+  (MSVC 2019 needs CMake 3.14+, MSVC 2022 needs CMake 3.21+).  
-  Install the latest using [https://cmake.org/download/](https://cmake.org/download/)
-
-Note: Windows MSVC 2019 runs as C\+\+17 by default so boost is not needed.  On linux use -std=c++17 compile option to avoid needing boost.
-
 
 #### Simple Python build (any platform)
 
-1) Prerequisites: install the following python packages: `pip install setuptools packaging`
+1) Prerequisites: install the following python packages: 
+	  `python -m ensurepip --upgrade`
+      `python -m pip install setuptools packaging`
+	  `pip -r install requirements.txt`
 
 2) At a command prompt, `cd` to the root directory of this repository.
 
 3) Run: `python setup.py install --user --force`
 
    This will build and install a release version of htm.core.  The `--user` option prevents the system installed site-packages folder from being changed and avoids the need for admin privileges.  The `--force` option forces the package to be replaced if it already exists from a previous build. Alternatively you can type `pip uninstall htm.core` to remove a previous package before performing a build.
 
-   * If you are using `virtualenv` you do not need the --user or --force options.
+   * If you are using `virtualenv` you may not need the --user or --force options.
    * If you are using Anaconda Python you must run within the `Anaconda Prompt` on Windows. Do not use --user or --force options.
 
    * If you run into problems due to caching of arguments in CMake, delete the
@@ -305,10 +292,11 @@ Generate IDE solution & build.
  * Specify the build system folder (`$HTM_CORE/build/scripts`), i.e. where IDE solution will be created.
  * Click `Generate`.
 
-#### For MS Visual Studio 2017 or 2019 as the IDE
+#### For MS Visual Studio 2017, 2019 or 2022 as the IDE
 
 After downloading the repository, do the following:
  * NOTE: Visual Studio 2019 requires CMake version 3.14 or higher.
+ *       Visual Studio 2022 requires CMake version 3.21 or higher.
  * CD to the top of repository.
  * Double click on `startupMSVC.bat`
     - This will setup the build, create the solution file (build/scripts/htm.cpp.sln), and start MS Visual Studio.
@@ -404,7 +392,7 @@ Note2: It is obvious, but anyway - do not use `--user` option while using python
 The installation scripts will automatically download and build the dependencies it needs.
 
  * [Boost](https://www.boost.org/)   (Not needed by C++17 compilers that support the filesystem module)
- * [LibYaml](https://pyyaml.org/wiki/LibYAML) or [Yaml-cpp](https://github.com/jbeder/yaml-cpp)
+ * [LibYaml](https://pyyaml.org/wiki/LibYAML)
  * [Eigen](https://eigen.tuxfamily.org/index.php?title=Main_Page)
  * [PyBind11](https://github.com/pybind/pybind11)
  * [gtest](https://github.com/google/googletest)
@@ -425,18 +413,17 @@ distribution packages as listed and rename them as indicated. Copy these to
 
 | Name to give it        | Where to obtain it |
 | :--------------------- | :----------------- |
-| libyaml.zip   (*note1) | https://github.com/yaml/libyaml/archive/refs/tags/0.2.5.tar.gz |
+| libyaml.zip            | https://github.com/yaml/libyaml/archive/refs/tags/0.2.5.tar.gz |
 | boost.tar.gz  (*note3) | https://dl.bintray.com/boostorg/release/1.72.0/source/boost_1_72_0.tar.gz | 
-| googletest.tar.gz      | https://github.com/google/googletest/archive/refs/tags/release-1.11.0.tar.gz |
+| googletest.tar.gz      | https://github.com/google/googletest/archive/refs/tags/release-1.12.1.tar.gz |
 | eigen.tar.bz2          | https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.tar.gz |
 | mnist.zip     (*note4) | https://github.com/wichtounet/mnist/archive/3b65c35ede53b687376c4302eeb44fdf76e0129b.zip |
-| pybind11.tar.gz        | https://github.com/pybind/pybind11/archive/refs/tags/v2.6.2.tar.gz |
+| pybind11.tar.gz        | https://github.com/pybind/pybind11/archive/refs/tags/v2.10.1.tar.gz |
 | cereal.tar.gz          | https://github.com/USCiLab/cereal/archive/refs/tags/v1.3.2.tar.gz |
 | sqlite3.tar.gz         | https://www.sqlite.org/2022/sqlite-autoconf-3380200.tar.gz |
 | digestpp.zip           | https://github.com/kerukuro/digestpp/archive/34ff2eeae397ed744d972d86b5a20f603b029fbd.zip |
-| cpp-httplib.zip(*note4)| https://github.com/yhirose/cpp-httplib/archive/refs/tags/v0.10.4.zip |
+| cpp-httplib.zip(*note4)| https://github.com/yhirose/cpp-httplib/archive/refs/tags/v0.11.3.zip |
 
- * note1: Version 0.2.2 of libyaml is broken so use the master for the repository.
  * note3: Boost is not required for any compiler that supports C++17 with `std::filesystem` (MSVC2017, gcc-8, clang-9).
  * note4: Used for examples. Not required to run but the build expects it.
 

bindings/py/cpp_src/plugin/PyBindRegion.cpp

@@ -308,43 +308,46 @@ namespace py = pybind11;
         // 2. call class method to serialize external state
 
         //    Serialize main state of the Python module
-				//    We want this to end up in a string that we pass back to Cereal.
-				//    a. We first pickle the python into an in-memory byte stream.
-				//    b. We then convert that to a Base64 std::string that is returned.
+        //    We want this to end up in a string that we pass back to Cereal.
+        //    a. We first pickle the python into an in-memory byte stream.
+        //    b. We then convert that to a Base64 std::string that is returned.
         //
         //  Basicly, we are executing the following Python code:
         //    import io
         //    import base64
         //    import pickle
         //    f = io.BytesIO()
-        //    pickle.dump(node, f, 3)
+        //    pickle.dump(node, f, HIGHEST_PROTOCOL)
         //    b = f.getvalue()
         //    content = str(base64.b64encode(b))
         //    f.close()
 
-		    py::tuple args;
-		    auto f = py::module::import("io").attr("BytesIO")();
+        py::tuple args;
+        auto f = py::module::import("io").attr("BytesIO")();
 
 #if PY_MAJOR_VERSION >= 3
-		    auto pickle = py::module::import("pickle");
-		    args = py::make_tuple(node_, f, 3);   // use type 3 protocol
+        auto pickle = py::module::import("pickle");
+        auto highest = pickle.attr("HIGHEST_PROTOCOL");
+        args = py::make_tuple(node_, f, highest);   // use type 3,4, or 5 protocol
 #else
-		    auto pickle = py::module::import("cPickle");
-		    args = py::make_tuple(node_, f, 2);   // use type 2 protocol
+        auto pickle = py::module::import("cPickle");
+        args = py::make_tuple(node_, f, 2);   // use type 2 protocol
 #endif
-		    pickle.attr("dump")(*args);
+        pickle.attr("dump")(*args);
 
-				// copy the pickle stream into the content as a base64 encoded utf8 string
+        // copy the pickle stream into the content as a base64 encoded utf8 string
         py::bytes b = f.attr("getvalue")();
         args = py::make_tuple(b);
-		    std::string content = py::str(py::module::import("base64").attr("b64encode")(*args));
-       
-		    f.attr("close")();
-		    return content;
+        std::string content = py::str(py::module::import("base64").attr("b64encode")(*args));
+        if (content[1] == '\'') { // strip off leading "b'" and trailing "'"
+           content = content.substr(2, content.length() - 3);
+        }
+        f.attr("close")();
+        return content;
     }
     std::string PyBindRegion::extraSerialize() const
     {
-		    std::string tmp_extra = "extra.tmp";
+        std::string tmp_extra = "extra.tmp";
 
         // 2. External state
         // Call the Python serializeExtraData() method to write additional data.
@@ -353,21 +356,21 @@ namespace py = pybind11;
         // Need to put the None result in py::Ptr to decrement the ref count
         node_.attr("serializeExtraData")(*args);
 
-				// copy the extra data into the extra string
-				std::ifstream efile(tmp_extra.c_str(), std::ios::binary);
-				std::string extra((std::istreambuf_iterator<char>(efile)),
-				                   std::istreambuf_iterator<char>());
-				efile.close();
-				Path::remove(tmp_extra);
-		    return extra;
+        // copy the extra data into the extra string
+        std::ifstream efile(tmp_extra.c_str(), std::ios::binary);
+        std::string extra((std::istreambuf_iterator<char>(efile)),
+                           std::istreambuf_iterator<char>());
+        efile.close();
+        Path::remove(tmp_extra);
+        return extra;
 
     }
 
 		void PyBindRegion::pickleDeserialize(std::string p) {
         // 1. deserialize main state using pickle
         // 2. call class method to deserialize external state
         //
-		    // Tell Python to un-pickle using what is in the string p.
+        // Tell Python to un-pickle using what is in the string p.
         // but first we need to convert the base64 string into bytes.
         //
         // Basically we are executing the following Python code:
@@ -383,7 +386,7 @@ namespace py = pybind11;
         args = py::make_tuple(py::bytes(p));
         py::bytes b = py::module::import("base64").attr("b64decode")(*args);
         args = py::make_tuple(b);
-		    auto f = py::module::import("io").attr("BytesIO")(*args);
+        auto f = py::module::import("io").attr("BytesIO")(*args);
 
 #if PY_MAJOR_VERSION >= 3
         auto pickle = py::module::import("pickle");
@@ -394,19 +397,19 @@ namespace py = pybind11;
         node_ = pickle.attr("load")(*args);
 
         f.attr("close")();
-		}
+    }
 
-		void PyBindRegion::extraDeserialize(std::string e) {
+    void PyBindRegion::extraDeserialize(std::string e) {
         // 2. External state
-		    std::string tmp_extra = "extra.tmp";
-			  std::ofstream efile(tmp_extra.c_str(), std::ios::binary);
-				efile.write(e.c_str(), e.size());
-				efile.close();
+        std::string tmp_extra = "extra.tmp";
+        std::ofstream efile(tmp_extra.c_str(), std::ios::binary);
+        efile.write(e.c_str(), e.size());
+        efile.close();
 
         // Call the Python deSerializeExtraData() method
         py::tuple args = py::make_tuple(tmp_extra);
         node_.attr("deSerializeExtraData")(*args);
-				Path::remove(tmp_extra);
+        Path::remove(tmp_extra);
     }
 
 

bindings/py/tests/regions/network_test.py

@@ -369,7 +369,7 @@ def testNetworkPickle(self):
     network.initialize()
 
     if sys.version_info[0] >= 3:
-      proto = 3
+      proto = pickle.HIGHEST_PROTOCOL
     else:
       proto = 2
 

bindings/py/tests/sparse_link_test.py

@@ -25,7 +25,7 @@
 TEST_DATA_SPARSE = np.array([4, 7])
 MAX_ACTIVE = TEST_DATA_SPARSE.size
 OUTPUT_WIDTH = 10
-TEST_DATA_DENSE = np.zeros(OUTPUT_WIDTH, dtype=np.bool)
+TEST_DATA_DENSE = np.zeros(OUTPUT_WIDTH, dtype=np.bool_)
 TEST_DATA_DENSE[TEST_DATA_SPARSE] = True
 
 

external/README.md

@@ -8,9 +8,9 @@ integrated into the cmake-based build of htm.core.  The code that does this are
 - Boost.cmake      - If needed, finds the boost installation 1.69.0. Boost needs to be built with -fPIC so cannot use externally installed.
 - digestpp.cmake   - Download/install digestpp @ 36fa6ca : Hash digest lib (header only)
 - eigen.cmake      - Downloads eigen 3.4.0  (header only)
-- gtest.cmake      - Downloads and installs googletest 1.11.0
+- gtest.cmake      - Downloads and installs googletest 1.12.1
 - mnist_data.cmake - Downloads the mnist data set from repository master.
-- pybind11.cmake   - Downloads and installs pybind11 2.6.2  (header only)
+- pybind11.cmake   - Downloads and installs pybind11 2.10.1  (header only)
 - libayml.cmake    - Downloads and installs libyaml 0.2.5 which is an alternative to yaml-cpp (default) 
 - cpp-httplib.cmake- Downloads and installs cpp-httplib 0.10.4, a REST server (header only)
 - cerial.cmake     - Downloads and installs cerial 1.3.2, a serialization package (header only)

external/cpp-httplib.cmake

@@ -25,7 +25,7 @@
 if(EXISTS "${REPOSITORY_DIR}/build/ThirdParty/share/cpp-httplib.zip")
     set(URL "${REPOSITORY_DIR}/build/ThirdParty/share/cpp-httplib.zip")
 else()
-    set(URL https://github.com/yhirose/cpp-httplib/archive/refs/tags/v0.10.4.zip)
+    set(URL https://github.com/yhirose/cpp-httplib/archive/refs/tags/v0.11.3.zip)
 endif()
 
 message(STATUS "obtaining cpp-httplib")

external/gtest.cmake

@@ -35,7 +35,7 @@
 if(EXISTS "${REPOSITORY_DIR}/build/ThirdParty/share/googletest.tar.gz")
     set(URL "${REPOSITORY_DIR}/build/ThirdParty/share/googletest.tar.gz")
 else()
-    set(URL https://github.com/google/googletest/archive/refs/tags/release-1.11.0.tar.gz)
+    set(URL https://github.com/google/googletest/archive/refs/tags/release-1.12.1.tar.gz)
 endif()
 
 #

external/pybind11.cmake

@@ -21,9 +21,7 @@
 if(EXISTS "${REPOSITORY_DIR}/build/ThirdParty/share/pybind11.tar.gz")
     set(URL "${REPOSITORY_DIR}/build/ThirdParty/share/pybind11.tar.gz")
 else()
-    set(URL https://github.com/pybind/pybind11/archive/v2.6.2.tar.gz)
-#    set(URL "https://github.com/pybind/pybind11/archive/refs/tags/v2.9.2.tar.gz")
-#  This caused an error regarding Base64 someplace inside pickle load. Reverting to 2.6.2
+    set(URL https://github.com/pybind/pybind11/archive/v2.10.1.tar.gz)
 endif()
 
 message(STATUS "obtaining PyBind11")

py/tests/advanced/algorithms/apical_tiebreak_temporal_memory/shared_tests/sequence_memory_test_base.py

@@ -892,9 +892,16 @@ def noisy(pattern, wFlip, n):
     A noisy set of active indices
     """
 
+    """
+       keeney, 12/2022
+       The random.sample() method requires the first argument to be sorted.
+       The apical_tiebreak_temporal_memory_test_base:897 was failing with an error saying
+       it was not a sequence. So I added a sorted(). I would have thought that a set
+       would already be sorted. I do not know if this affects the algorithm logic.
+    """
     noised = set(pattern)
 
-    noised.difference_update(random.sample(noised, wFlip))
+    noised.difference_update(random.sample(sorted(noised), wFlip))
 
     for _ in range(wFlip):
         while True:

requirements.txt

@@ -1,15 +1,15 @@
 #See http://www.pip-installer.org/en/latest/requirements.html for details
 setuptools>=34.4.0 # needed for Windows with MSVC
-pip>=19.3.1
-wheel>=0.33.6
-cmake>=3.14 #>=3.7, >=3.14 needed for MSVC 2019
+pip>=22.3.1
+wheel>=0.38.4
+cmake>=3.14 #>=3.7, >=3.14 needed for MSVC 2019, >=3.21 needed for MSVC 2022
 ## for python bindings (in /bindings/py/)
-numpy>=1.15
+numpy>=1.23
 pytest>=4.6.5 #4.6.x series is last to support python2, once py2 dropped, we can switch to 5.x 
 ## for python code (in /py/)
-hexy>=1.4.3 # for grid cell encoder
-mock>=1.0.1 # for anomaly likelihood test
-prettytable>=0.7.2 # for monitor-mixin in htm.advanced (+its tests)
+hexy>=1.4.4 # for grid cell encoder
+mock>=3.3 # for anomaly likelihood test
+prettytable>=3.5.0 # for monitor-mixin in htm.advanced (+its tests)
 ## optional dependencies, such as for visualizations, running examples
 # should be placed in setup.py section extras_require. Install those by
 # pip install htm.core[examples]