OCIOZ archive feature (#1627) (#1696)

* OCIOZ archive feature (#1627)

Signed-off-by: Cedrik Fuoco <[email protected]>

* Add OCIOZ extension if not present.

Signed-off-by: Cedrik Fuoco <[email protected]>

* - Changing getLutData (return vector instead of passing by reference) in order to respect C++ idiom and letting compiler optimize with NRVO and Copy elision
- Improving how cmake find zlib (renamed getZLIB to Findzlib)
- Make sure that ZLIB is statically linked on all platform (it wasn't on windows)
- Fix issue with Python Unit test when using Python 2 (problem discovered on OSX)
- Change how OpenEXR finds ZLIB by using our custom Findzlib
- Patching Minizip-ng makefile to change cmake minimum version

Signed-off-by: Cedrik Fuoco <[email protected]>

* - Remove the minizip-ng patch as it doesn't work and further discussion needs to happend.
- Responding to Remi's comments.
  Notable changes:
  - Comments typo and styles fix.
  - Unit tests fixes.
  - Removing the usage of std::ifstream in getLutData and getConfig since it is not needed at all - It was slowing down getLutData.
  - Implicitly converting python list to std::vector<uint8_t> (for getLutData)
  - When archiving a config, the file won't added (overwridden) multiple times if the file matches multiple supported format.

Signed-off-by: Cedrik Fuoco <[email protected]>

* - Added implicit conversion from bytearray to std::vector<uint8_t>
- In ConfigIOProxy Python Unit test, changed getLutData to return a bytearray instead of a list. I think this makes more sense and it works with Python 2 and 3.

Signed-off-by: Cedrik Fuoco <[email protected]>

* Update comments to be in line with the current implementations

Signed-off-by: Cedrik Fuoco <[email protected]>

* - Changing ci workflow in order to install cmake version 3.13.3 for Linux (matrix.vfx-cy == '2019').

Signed-off-by: Cedrik Fuoco <[email protected]>

* - Fixing issue with the install location of zlib library
- Removing unused CMAKE variables from ZLIB_CMAKE_ARGS

Signed-off-by: Cedrik Fuoco <[email protected]>

Signed-off-by: Cedrik Fuoco <[email protected]>
Co-authored-by: Doug Walker <[email protected]>

Author: cedrik-fuoco-adsk

.github/workflows/ci_workflow.yml

@@ -252,6 +252,10 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v2
+      # minizip-ng requires CMake 3.13+ but VFX2019 image ships with 3.12
+      - name: Upgrade VFX2019 CMake
+        run: pip install cmake==3.13.3
+        if: matrix.vfx-cy == '2019'
       - name: Install docs env
         run: share/ci/scripts/linux/yum/install_docs_env.sh
         if: matrix.build-docs == 'ON'

include/OpenColorIO/OpenColorIO.h

@@ -10,13 +10,14 @@
 #include <limits>
 #include <stdexcept>
 #include <string>
+#include <fstream>
+#include <vector>
 
 #include "OpenColorABI.h"
 #include "OpenColorTypes.h"
 #include "OpenColorTransforms.h"
 #include "OpenColorAppHelpers.h"
 
-
 /*
 
 C++ API
@@ -174,6 +175,8 @@ extern OCIOEXPORT void LogMessage(LoggingLevel level, const char * message);
 /**
  * \brief Set the Compute Hash Function to use; otherwise, use the default.
  * 
+ * This is not used when using CreateFromFile with an OCIOZ archive or CreateFromConfigIOProxy.
+ * 
  * \param ComputeHashFunction
  */
 extern OCIOEXPORT void SetComputeHashFunction(ComputeHashFunction hashFunction);
@@ -201,6 +204,26 @@ extern OCIOEXPORT ConstConfigRcPtr GetCurrentConfig();
 /// Set the current configuration. This will then store a copy of the specified config.
 extern OCIOEXPORT void SetCurrentConfig(const ConstConfigRcPtr & config);
 
+/**
+ * \brief Extract an OCIO Config archive.
+ * 
+ * Converts an archived config file (.ocioz file) back to its original form as a config file
+ * and associated LUT files.  This creates destinationDir and then creates a config.ocio file
+ * at the root of that working directory and then unpacks the LUT files into their relative
+ * locations relative to that working directory, creating any necessary sub-directories in the
+ * process.  Note that configs which contain LUT files outside the working directory are not
+ * archivable, and so this function will not create directories outside the working directory.
+ * 
+ * \param archivePath Absolute path to the .ocioz file.
+ * \param destinationDir Absolute path of the directory you want to be created to contain the
+ * contents of the unarchived config.
+ * \throw Exception If the archive is not found or there is a problem extracting it.
+ */
+extern OCIOEXPORT void ExtractOCIOZArchive(
+    const char * archivePath, 
+    const char * destinationDir
+);
+
 /**
  * \brief
  * A config defines all the color spaces to be available at runtime.
@@ -245,49 +268,89 @@ class OCIOEXPORT Config
      * \brief Create an empty config of the current version.
      *
      * Note that an empty config will not pass validation since required elements will be missing.
+     * \return The Config object.
      */
     static ConfigRcPtr Create();
+
     /**
      * \brief Create a fall-back config.
      * 
      * This may be useful to allow client apps to launch in cases when the
      * supplied config path is not loadable.
+     * \return The Config object.
      */
     static ConstConfigRcPtr CreateRaw();
+
     /**
      * \brief Create a configuration using the OCIO environment variable.
      * 
-     * Also support OCIO URI format. See CreateFromFile.
+     * Also supports the OCIO URI format for Built-in configs and supports archived configs.
+     * See \ref Config::CreateFromFile.
      * 
      * If the variable is missing or empty, returns the same result as 
-     * \ref Config::CreateRaw .
+     * \ref Config::CreateRaw.
+     * \return The Config object.
      */
     static ConstConfigRcPtr CreateFromEnv();
+
     /**
      * \brief Create a configuration using a specific config file.
      * 
-     * Also support the following OCIO URI format :
+     * Also supports the following OCIO URI format for Built-in configs:
      *  "ocio://default"    - Default Built-in config.
-     *  "ocio://configName" - Built-in config named configName
-     * 
+     *  "ocio://<CONFIG NAME>" - A specific Built-in config.  For the list of available
+     *  <CONFIG NAME> strings, see \ref Config::CreateFromBuiltinConfig.
+     *
+     * Also supports archived configs (.ocioz files).
+     *
+     * \throw Exception If the file may not be read or does not parse.
+     * \return The Config object.
      */
     static ConstConfigRcPtr CreateFromFile(const char * filename);
-    /// Create a configuration using a stream.
+
+    /**
+     * \brief Create a configuration using a stream.
+     * 
+     * Note that CreateFromStream does not set the working directory so the caller would need to
+     * set that separately in order to resolve FileTransforms.  This function is typically only
+     * used for self-contained configs (no LUTs).
+     *
+     * Configs created from CreateFromStream can not be archived unless the working directory is 
+     * set and contains any necessary LUT files.
+     * 
+     * \param istream Stream to the config.
+     * \throw Exception If the stream does not parse.
+     * \return The Config object.
+     */
     static ConstConfigRcPtr CreateFromStream(std::istream & istream);
 
+    /**
+     * \brief Create a config from the supplied ConfigIOProxy object. This allows the calling
+     * program to directly provide the config and associated LUTs rather than reading them from
+     * the standard file system.
+     * 
+     * See the \ref ConfigIOProxy class documentation for more info.
+     * 
+     * \param ciop ConfigIOProxy object providing access to the config's files.
+     * \throw Exception If the config may not be read from the proxy, or does not parse.
+     * \return The Config object.
+     */
+    static ConstConfigRcPtr CreateFromConfigIOProxy(ConfigIOProxyRcPtr ciop);
+    
     /**
      * \brief Create a configuration using an OCIO built-in config.
      * 
-     * \param configName Built-in config name
+     * \param configName Built-in config name.
      * 
      * The available configNames are:
      * "cg-config-v0.1.0_aces-v1.3_ocio-v2.1.1" -- ACES CG config, basic color spaces for computer
-     * graphics apps. More information is available at: 
+     * graphics apps. More information about these configs is available at: 
      * %https://github.com/AcademySoftwareFoundation/OpenColorIO-Config-ACES
      * 
-     * \throw Exception If the configName is not recognized.
+     * Information about the available configs is available from the \ref BuiltinConfigRegistry.
      * 
-     * \return one of the configs built into OCIO library
+     * \throw Exception If the configName is not recognized.
+     * \return One of the configs built into the OCIO library.
      */
     static ConstConfigRcPtr CreateFromBuiltinConfig(const char * configName);
 
@@ -1149,6 +1212,62 @@ class OCIOEXPORT Config
                                                        const char * dstColorSpaceName,
                                                        const char * dstInterchangeName);
 
+    /// Set the ConfigIOProxy object used to provision the config and LUTs from somewhere other
+    /// than the file system.  (This is set on the config's embedded Context object.)
+    void setConfigIOProxy(ConfigIOProxyRcPtr ciop);
+    ConfigIOProxyRcPtr getConfigIOProxy() const;
+
+    /**
+     * \brief Verify if the config is archivable.
+     *
+     * A config is not archivable if any of the following are true:
+     * -- The working directory is not set
+     * -- It contains FileTransforms with a src outside the working directory
+     * -- The search path contains paths outside the working directory
+     * -- The search path contains paths that start with a context variable
+     * 
+     * Context variables are allowed but the intent is that they may only resolve to paths that
+     * are within or below the working directory.  This is because the archiving function will
+     * only archive files that are within the working directory in order to ensure that if it is
+     * later expanded, that it will not create any files outside this directory.
+     *
+     * For example, a context variable on the search path intended to contain the name of a 
+     * sub-directory under the working directory must have the form "./$DIR_NAME" rather than just
+     * "$DIR_NAME" to be considered archivable.  This is imperfect since there is no way to
+     * prevent the context variable from creating a path outside the working dir, but it should
+     * at least draw attention to the fact that the archive would fail if used with context vars
+     * that try to abuse the intended functionality.
+     * 
+     * \return bool Archivable if true.
+     */
+    bool isArchivable() const;
+
+    /**
+     * \brief Archive the config and its LUTs into the specified output stream.
+     * 
+     * The config is archived by serializing the Config object into a file named "config.ocio" and
+     * then walking through the current working directory and any sub-directories.  Any files that
+     * have an extension matching a supported LUT file format are added to the archive.  Any files
+     * that do not have an extension (or have some unsupported LUT extension, including .ocio),
+     * will not be added to the archive. To reiterate, it is the in-memory Config object that is
+     * archived, and not any .ocio file in the current working directory.  The directory structure
+     * relative to the working directory is preserved.  No files outside the working directory are
+     * archived so that if it is later expanded, no files will be created outside the working dir.
+     *
+     * The reason the archive is created using all supported LUT file extensions rather than by
+     * trying to resolve all the FileTransforms in the Config to specific files is because of the
+     * goal to allow context variables to continue to work.
+     * 
+     * If a Config is created with CreateFromStream, CreateFromFile with an OCIOZ archive, or 
+     * CreateFromConfigIOProxy, it cannot be archived unless the working directory is manually set 
+     * to a directory that contains any necessary LUT files. 
+     * 
+     * The provided output stream must be closed by the caller, if necessary (e.g., an ofstream).
+     *
+     * \param ostream The output stream to write to.
+     */
+    void archive(std::ostream & ostream) const;
+
     Config(const Config &) = delete;
     Config& operator= (const Config &) = delete;
 
@@ -3269,6 +3388,11 @@ class OCIOEXPORT Context
     /// used to resolve the filename (empty if no context variables were used).
     const char * resolveFileLocation(const char * filename, ContextRcPtr & usedContextVars) const;
 
+    /// Set the ConfigIOProxy object used to provision the config and LUTs from somewhere other
+    /// than the file system.
+    void setConfigIOProxy(ConfigIOProxyRcPtr ciop);
+    ConfigIOProxyRcPtr getConfigIOProxy() const;
+
     Context(const Context &) = delete;
     Context& operator= (const Context &) = delete;
     /// Do not use (needed only for pybind11).
@@ -3437,6 +3561,61 @@ class OCIOEXPORT SystemMonitors
     virtual ~SystemMonitors() = default;
 };
 
+
+///////////////////////////////////////////////////////////////////////////
+// ConfigIOProxy
+
+/**
+ * ConfigIOProxy is a proxy class to allow the calling program to supply the config and any
+ * associated LUT files directly, without relying on the standard file system. 
+ * 
+ * The OCIOZ archive feature is implemented using this mechanism.
+ */
+class OCIOEXPORT ConfigIOProxy
+{
+public:
+    ConfigIOProxy() = default;
+    virtual ~ConfigIOProxy() = default;
+
+    /**
+     * \brief Provide the contents of a LUT file as a buffer of uint8_t data.
+     * 
+     * \param filepath Fully resolved path to the "file."
+     * 
+     * The file path is based on the Config's current working directory and is the same absolute
+     * path that would have been provided to the file system.
+     * 
+     * \return Vector of uint8 with the content of the LUT.
+     */
+    virtual std::vector<uint8_t> getLutData(const char * filepath) const = 0;
+
+    /**
+     * \brief Provide the config file Yaml to be parsed.
+     * 
+     * \return String with the config Yaml.
+     */
+    virtual std::string getConfigData() const = 0;
+
+    /**
+     * \brief Provide a fast unique ID for a LUT file.
+     * 
+     * This is intended to supply the string that will be used in OCIO's FileCacheMap.
+     * This should be highly performant and typically should not require extensive
+     * computation such as calculating the md5 hash of the file, unless it is pre-computed.
+     * 
+     * If the "file" does not exist, in other words, if the proxy is unable to supply the requested
+     * file contents, the function must return an empty string.
+     * 
+     * \param filepath Fully resolve the path to the "file."
+     * 
+     * The file path is based on the Config's current working directory and is the same absolute
+     * path that would have been provided to the file system.
+     *
+     * \return The file hash string.
+     */
+    virtual std::string getFastLutFileHash(const char * filepath) const = 0;
+};
+
 } // namespace OCIO_NAMESPACE
 
 #endif // INCLUDED_OCIO_OPENCOLORIO_H

include/OpenColorIO/OpenColorTypes.h

@@ -114,6 +114,10 @@ class OCIOEXPORT GradingRGBCurve;
 typedef OCIO_SHARED_PTR<const GradingRGBCurve> ConstGradingRGBCurveRcPtr;
 typedef OCIO_SHARED_PTR<GradingRGBCurve> GradingRGBCurveRcPtr;
 
+class OCIOEXPORT ConfigIOProxy;
+typedef OCIO_SHARED_PTR<const ConfigIOProxy> ConstConfigIOProxyRcPtr;
+typedef OCIO_SHARED_PTR<ConfigIOProxy> ConfigIOProxyRcPtr;
+
 typedef std::array<float, 3> Float3;
 
 
@@ -960,6 +964,13 @@ extern OCIOEXPORT const char * OCIO_DISABLE_CACHE_FALLBACK;
 
 /** @}*/
 
+
+// Archive config feature
+// Default filename (with extension) of an config.
+extern OCIOEXPORT const char * OCIO_CONFIG_DEFAULT_NAME;
+extern OCIOEXPORT const char * OCIO_CONFIG_DEFAULT_FILE_EXT;
+extern OCIOEXPORT const char * OCIO_CONFIG_ARCHIVE_FILE_EXT;
+
 } // namespace OCIO_NAMESPACE
 
 #endif

share/cmake/modules/FindExtPackages.cmake

@@ -41,6 +41,15 @@ find_package(pystring 1.1.3 REQUIRED)
 set(_Imath_ExternalProject_VERSION "3.1.5")
 find_package(Imath 3.0 REQUIRED)
 
+# ZLIB
+# https://github.com/madler/zlib
+set(_zlib_ExternalProject_VERSION "1.2.12")
+find_package(zlib REQUIRED)
+
+# minizip-ng
+# https://github.com/zlib-ng/minizip-ng
+find_package(minizip-ng 3.0.6 REQUIRED)
+
 if(OCIO_BUILD_APPS)
 
     # NOTE: Depending of the compiler version lcms2 2.2 does not compile with 

share/cmake/modules/FindOpenEXR.cmake

@@ -85,7 +85,8 @@ if(NOT OpenEXR_FOUND AND NOT OCIO_INSTALL_EXT_PACKAGES STREQUAL NONE)
     set(_EXT_BUILD_ROOT "${CMAKE_BINARY_DIR}/ext/build")
 
     # Required dependency
-    find_package(ZLIB)
+    # OCIO custom module to find ZLIB. (Findzlib)
+    find_package(zlib)
     if(NOT ZLIB_FOUND)
         message(STATUS "ZLib is required to build OpenEXR.")
         return()