# This is a combination of 55 commits.

# This is the 1st commit message:

Issue #303: framework for reference_dense added

This commit comes from a split of the original commit whose details are as follows:
SHA1 ID: a7c1db0
Author: Albert-Jan Yzelman <[email protected]>  2022-01-14 15:52:26
Committer: Albert-Jan Yzelman <[email protected]>  2022-01-30 13:48:35

The reason for splitting is to separate the changes irrelevant to the branch 303 into a separate branch.
The split produced 3 separate commits.
This commit is number 1.
This commit in particular is relevant to 303 and will remain in it.

# This is the commit message #2:

Issue #303: unify script argument similar to --no-reference

# This is the commit message #3:

Issue #303: complete implementation of Vector< reference_dense >, modulo output iteration.

# This is the commit message #4:

Issue #303: modified vector implementation according to option 3 in handling dense container construction.

# This is the commit message #5:

Issue #303: vectors now have an iterator for the user to extract data from.

# This is the commit message #6:

Issue #303, fixed silly oops: the difference between two iterators should be an integer

# This is the commit message #7:

Issue #303: containers are now uninitialized on construction. Iteration over uninitialized containers returns nothing.

# This is the commit message #8:

Issue #303: configure and cmake now properly do make install with reference_dense

# This is the commit message #9:

Issue #303: cmake infra has updated-- make dense_reference additions compatible to it

# This is the commit message #10:

Drafting structured matrix interface using reference_dense backend

# This is the commit message #11:

Issue #303: Copy test mxm.cpp to dense_mxm.cpp as a starting point

# This is the commit message #12:

Issue #303: Add partial first version of Matrix interface for denseref

# This is the commit message #13:

Issue #303: WIP: dense mxm test

# This is the commit message #14:

Issue #303: Add dense mxm test to CMakeLists

# This is the commit message #15:

Issue #303: Separate reference and reference_dense includes

# This is the commit message #16:

Issue #303: Fix grb include name mismatch

# This is the commit message #17:

Issue #303: Add src files for reference dense backend

# This is the commit message #18:

Issue #303: Add project related option for dense backend

# This is the commit message #19:

Issue #303: Remove dense_reference-related code from reference cmake

# This is the commit message #20:

Keeping a clean separate denseref backend when building with cmake.

# This is the commit message #21:

Fix cmake option name

# This is the commit message #22:

Use a proper data container for a denseref matrix

# This is the commit message #23:

Change PRIVATE to INTERFACE for denseref backend

# This is the commit message #24:

fix cmake config for building backend

# This is the commit message #25:

Add friend functions for accessing matrix dimensions and raw data

# This is the commit message #26:

Add first version of blas3 mxm for reference dense backend

# This is the commit message #27:

Add temporary code for matrix data initialization

# This is the commit message #28:

Complete first working dense mxm unit test

# This is the commit message #29:

Added structured matrix test including draft static views and references

# This is the commit message #30:

Convert spaces to tabs

# This is the commit message #31:

Make use of operators and monoid provided to mxm

# This is the commit message #32:

Add friend getter/setter for field 'initialized' of denseref matrix

# This is the commit message #33:

Revert "Add temporary code for matrix data initialization"

This reverts commit 676cb4d.

# This is the commit message #34:

Add temporary way for allocating data of a denseref matrix

# This is the commit message #35:

Add functions to initialize matrix data from provided forward iterator

# This is the commit message #36:

Make getters const

# This is the commit message #37:

Exploit new features in mxm unit test

New features are building a matrix from an iterator and handling of
uninitialized matrices.

# This is the commit message #38:

added get/remove_ref type traits for StructuredMatrix types

# This is the commit message #39:

Temporary switch to pointers to target matrices to enable empty init.

# This is the commit message #40:

First draft of index mapping functions

# This is the commit message #41:

Cleaned up StructuredMatrix-related doxygen drafts

# This is the commit message #42:

Added to the spec draft

# This is the commit message #43:

Add alloc primitives for denseref

Currently they are a copy of the same primitives for reference backend.
Think whether it makes sense to group the alloc functions for these two
backends.

# This is the commit message #44:

Handle memory allocation failure. Implement destructor.

# This is the commit message #45:

WIP: Add data size calculator depending on Storage scheme and structure.

Also add a simple test to showcase the usage of this feature.

# This is the commit message #46:

Add buildMatrix for general full StructuredMatrix

# This is the commit message #47:

Add a check for size mismatch in buildMatrix

# This is the commit message #48:

Add mxm for dense Structured Matrices

# This is the commit message #49:

Introducing grb::get_view<> draft

# This is the commit message #50:

Add UpperTriangular StructuredMatrix (container and view)

# This is the commit message #51:

Implement polymorphic comparison of IMFs

# This is the commit message #52:

Add to structures a compatibility check from source structure and IMF functions

This is to be used when creating a view over a source matrix using
provided IMF functions. The compatibility of TargetStructure depends on
the SourceStructure and provided IMF functions. This is a runtime check.

# This is the commit message #53:

Add constructor taking two IMFs to reference SM with general structure

# This is the commit message #54:

Implement gather through view with provided TargetStructure and IMFs

# This is the commit message #55:

Add test for gathers on StructuredMatrix using views with IMFs

Author: hidanielesgit

CMakeLists.txt

@@ -50,7 +50,7 @@ endif()
 ### CONFIGURATION OPTIONS
 # to choose backends and dependencies
 option( WITH_REFERENCE_BACKEND "With Reference backend" ON )
-option( WITH_DENSE_BACKEND "With Dense backend" ON )
+option( WITH_DENSEREF_BACKEND "With Reference Dense backend" ON )
 option( WITH_OMP_BACKEND "With OMP backend" ON )
 option( WITH_HYPERDAGS_BACKEND "With Hyperdags backend" ON )
 if( WITH_HYPERDAGS_BACKEND )

bootstrap.sh

@@ -369,9 +369,9 @@ the current directory before invocation or confirm the deletion of its content w
 		CMAKE_OPTS+=" -DWITH_NONBLOCKING_BACKEND=OFF"
 	fi
 	if [[ "${dense}" == "no" ]]; then
-		CMAKE_OPTS+=" -DWITH_DENSE_BACKEND=OFF"
+		CMAKE_OPTS+=" -DWITH_DENSEREF_BACKEND=OFF"
 	else
-		CMAKE_OPTS+=" -DWITH_DENSE_BACKEND=ON"
+		CMAKE_OPTS+=" -DWITH_DENSEREF_BACKEND=ON"
 	fi
 	if [[ "${lpf}" == "yes" ]]; then
 		CMAKE_OPTS+=" -DLPF_INSTALL_PATH='${ABSOLUTE_LPF_INSTALL_PATH}'"

cmake/AddGRBInstall.cmake

@@ -45,6 +45,7 @@ install( EXPORT GraphBLASTargets
 set( ALP_UTILS_INSTALL_DIR "${BINARY_LIBRARIES_INSTALL_DIR}" )
 set( SHMEM_BACKEND_INSTALL_DIR "${BINARY_LIBRARIES_INSTALL_DIR}/sequential" )
 set( HYPERDAGS_BACKEND_INSTALL_DIR "${BINARY_LIBRARIES_INSTALL_DIR}/hyperdags" )
+set( DENSEREF_BACKEND_INSTALL_DIR "${BINARY_LIBRARIES_INSTALL_DIR}/denseref" )
 set( BSP1D_BACKEND_INSTALL_DIR "${BINARY_LIBRARIES_INSTALL_DIR}/spmd" )
 set( HYBRID_BACKEND_INSTALL_DIR "${BINARY_LIBRARIES_INSTALL_DIR}/hybrid" )
 
@@ -121,10 +122,10 @@ if( WITH_REFERENCE_BACKEND )
 	)
 endif()
 
-if( WITH_DENSE_BACKEND )
+if( WITH_DENSEREF_BACKEND )
 	addBackendWrapperGenOptions( "reference_dense"
-		COMPILE_DEFINITIONS "${DENSE_SELECTION_DEFS}"
-		LINK_FLAGS "${SHMEM_BACKEND_INSTALL_DIR}/lib${BACKEND_LIBRARY_OUTPUT_NAME}.a"
+		COMPILE_DEFINITIONS "DENSEREF_INCLUDE_DEFS" "${DENSEREF_SELECTION_DEFS}"
+		LINK_FLAGS "${DENSEREF_BACKEND_INSTALL_DIR}/lib${BACKEND_LIBRARY_OUTPUT_NAME}.a"
 	)
 endif()
 

cmake/AddGRBVars.cmake

@@ -33,7 +33,7 @@ set( BSP1D_BACKEND_DEFAULT_NAME "backend_bsp1d" )
 set( HYBRID_BACKEND_DEFAULT_NAME "backend_hybrid" )
 set( HYPERDAGS_BACKEND_DEFAULT_NAME "backend_hyperdags" )
 set( NONBLOCKING_BACKEND_DEFAULT_NAME "backend_nonblocking" )
-set( DENSE_BACKEND_DEFAULT_NAME "backend_reference_dense" )
+set( DENSEREF_BACKEND_DEFAULT_NAME "backend_reference_dense" )
 
 
 ### COMPILER DEFINITIONS FOR HEADERS INCLUSION AND FOR BACKEND SELECTION
@@ -44,7 +44,7 @@ set( REFERENCE_OMP_INCLUDE_DEFS "_GRB_WITH_OMP" )
 set( HYPERDAGS_INCLUDE_DEFS "_GRB_WITH_HYPERDAGS" )
 set( NONBLOCKING_INCLUDE_DEFS "_GRB_WITH_NONBLOCKING" )
 set( LPF_INCLUDE_DEFS "_GRB_WITH_LPF" )
-set( DENSE_INCLUDE_DEFS "_GRB_WITH_DENSE" )
+set( DENSEREF_INCLUDE_DEFS "_GRB_WITH_DENSEREF" )
 
 # compiler definitions to select a backend
 set( REFERENCE_SELECTION_DEFS "_GRB_BACKEND=reference" )
@@ -54,7 +54,7 @@ set( HYPERDAGS_SELECTION_DEFS
 	"_GRB_WITH_HYPERDAGS_USING=${WITH_HYPERDAGS_USING}"
 )
 set( NONBLOCKING_SELECTION_DEFS "_GRB_BACKEND=nonblocking" )
-set( DENSE_SELECTION_DEFS "_GRB_BACKEND=reference_dense" )
+set( DENSEREF_SELECTION_DEFS "_GRB_BACKEND=reference_dense" )
 set( BSP1D_SELECTION_DEFS
 		"_GRB_BACKEND=BSP1D"
 		"_GRB_BSP1D_BACKEND=reference"
@@ -94,7 +94,7 @@ if( WITH_NONBLOCKING_BACKEND )
 	list( APPEND AVAILABLE_BACKENDS "nonblocking" )
 endif()
 
-if( WITH_DENSE_BACKEND )
+if( WITH_DENSEREF_BACKEND )
 	list( APPEND AVAILABLE_BACKENDS "reference_dense" )
 endif()
 

include/CMakeLists.txt

@@ -120,7 +120,7 @@ if( WITH_REFERENCE_BACKEND_HEADERS )
 	install( TARGETS backend_reference_headers EXPORT GraphBLASTargets )
 endif()
 
-if( WITH_DENSE_BACKEND )
+if( WITH_DENSEREF_BACKEND )
 	add_library( backend_reference_dense_headers INTERFACE )
 	target_link_libraries( backend_reference_dense_headers INTERFACE backend_headers_nodefs )
 	target_compile_definitions( backend_reference_dense_headers INTERFACE "${DENSEREF_INCLUDE_DEFS}" )

include/graphblas/base/matrix.hpp

@@ -28,10 +28,10 @@
 #define _H_GRB_MATRIX_BASE
 
 #include <iterator>
-
 #include <stddef.h>
-
+#include <type_traits>
 #include <utility>
+#include <memory>
 
 #include <graphblas/backends.hpp>
 #include <graphblas/descriptors.hpp>
@@ -41,6 +41,7 @@
 #include <graphblas/structures.hpp>
 #include <graphblas/utils.hpp>
 #include <graphblas/views.hpp>
+#include <graphblas/imf.hpp>
 
 
 namespace grb {
@@ -395,39 +396,112 @@ namespace grb {
 
 	};
 
+template< typename InputType, Backend backend >
+RC clear( Matrix< InputType, backend > & A ) noexcept {
+	// this is the generic stub implementation
+	return UNSUPPORTED;
+}
+
 /**
- * An ALP structured matrix.
+ * \brief An ALP structured matrix.
  *
- * This is an opaque data type for that implements the below functions.
+ * This is an opaque data type for structured matrices. 
+ * This container allows to maintain the interface of grb::Matrix and grb::Vector 
+ * unaltered enabling back-compatibility while building on them to create
+ * semantically reacher algebraic objects.
+ * A structured matrix is generalized over five parameters further described 
+ * below: its data type, 
+ * its structure, whether it is stored using a dense or sparse storage scheme, 
+ * a static view and the backend for which it is implemented.
+ * At a high level of abstraction a structured matrix exposes a mathematical 
+ * \em logical layout which allows to express implementation-oblivious concepts 
+ * (e.g., the transpose of a symmetric matrix).
+ * At the lowest level, the logical layout maps to its physical counterpart via 
+ * a particular choice of a storage scheme within those exposed by the chosen 
+ * backend. grb::Matrix and grb::Vector are used as interfaces to the physical
+ * layout.
+ * To visualize this, you could think of a band matrix. Using either the 
+ * \a storage::Dense:full or \a storage::Dense:band storage schemes would require
+ * the use of a \a grb::Matrix container (see include/graphblas/storage.hpp for
+ * more details about the two storage schemes). However, the interpration of its 
+ * content would differ in the two cases being a function of both the Structure 
+ * information and the storage scheme combined.
+ * 
+ * Views can be used to create logical \em perspectives on top of a container. 
+ * For example, I could decide to refer to the transpose of a matrix or to see 
+ * a for limited part of my program a square matrix as symmetric. 
+ * If a view can be expressed as concept invariant of specific runtime features,
+ * such views can be defined statically (for example I can always refer to the 
+ * transpose or the diagonal of a matrix irrespective of features such as its 
+ * size). Other may depend on features such as the size of a matrix
+ * and can be expressed as linear transformations via operations such as \a mxm 
+ * (e.g., gathering/scattering the rows/columns of a matrix or permuting them).
+ * Structured matrices defined as views on other matrices do not instantiate a
+ * new container but refer to the one used by their targets. (See the 
+ * documentation of StructuredMatrix for both scenarios within the \em denseref 
+ * backend folder).
  *
  * @tparam T The type of the matrix elements. \a T shall not be a GraphBLAS
  *            type.
- * @tparam Structure  One of the matrix structures in grb::structures.
- * @tparam view  One of the matrix views in enum grb::Views. All static views except for
- * 		   \a Views::original disable the possibility to instantiate a new container and only
- * 		   allow the creation of a reference to an existing \a StructuredMatrix
- * 		   (note that this could be view itself). A view could be instanciated into a separate container
- * 		   than its original source by explicit copy.
- * @tparam backend Allows multiple backends to implement different
- *         versions of this data type.
+ * @tparam Structure  One of the matrix structures in \a grb::structures.
+ * @tparam StorageSchemeType Either \em enum \a storage::Dense or \em enum 
+ * 	       \a storage::Sparse.
+ * 		   A StructuredMatrix will be allowed to pick among the storage schemes 
+ *         within their specified \a StorageSchemeType.
+ * @tparam View  One of the matrix views in \a grb::view.
+ * 		   All static views except for \a view::Identity (via 
+ *         \a view::identity<void> cannot instantiate a new container and only 
+ *         allow to refer to an existing \a StructuredMatrix.  
+ *         The \a View parameter should not be used directly by the user but 
+ *         can be set using specific member types appropriately 
+ *         defined by each StructuredMatrix. (See examples of StructuredMatrix 
+ *         definitions within \a include/graphblas/denseref/matrix.hpp and the
+ *         \a dense_structured_matrix.cpp unit test).
+ * @tparam backend Allows multiple backends to implement different versions 
+ *         of this data type.
  *
- * \note The presence of different combination of structures and views could produce many specialization
- *       with lots of logic replication.
+ * \note The presence of different combination of structures and views could 
+ *       produce many specialization with lots of logic replication. We might
+ *       could use some degree of inheritence to limit this.
  */
 template< typename T, typename Structure, typename StorageSchemeType, typename View, enum Backend backend >
 class StructuredMatrix {
 
-	size_t m, n;
+
+	/** 
+	 * Whether the container presently is initialized or not. 
+	 * We differentiate the concept of empty matrix (matrix of size \f$0\times 0\f$)
+	 * from the one of uninitialized (matrix of size \f$m\times n\f$ which was never set)
+	 * and that of zero matrix (matrix with all zero elements).
+	 * \note in sparse format a zero matrix result in an ampty data structure. Is this
+	 * used to refer to uninitialized matrix in ALP/GraphBLAS?
+	 **/
+	bool initialized;
+
+	/**
+	 * The two following members define the \em logical layout of a structured matrix:
+	 * Its structure and access relations. This is enabled only if the structured matrix
+	 * does not define a View on another matrix.
+	 */
+	using structure = Structure;
+	/**
+	 * A pair of pointers to index mapping functions (see imf.hpp) that express the
+	 * logical access to the structured matrix.
+	 */
+	std::shared_ptr<imf::IMF> imf_l, imf_r;
 
 	/**
-	 * The container's data.
-	 * The geometry and access scheme are specified by a combination of
-	 * \a Structure, \a storage_scheme, \a m, and \a n.
+	 * When a structured matrix instanciate a \em container it defines a new \em physical
+	 * (concrete?) layout. This is characterized by an ALP container (aka a \a Matrix) and a 
+	 * storage scheme that defines a unique interpretation of its content.
+	 * The combination of the logical and physical layout of a structured matrix enables to
+	 * identify a precise mapping between an element in the structured matrix and a position
+	 * wihtin one or more 1/2D-arrays that store it.
 	 */
-	T * __restrict__ data;
+    Matrix< T, reference_dense > * _container;
 
 	/**
-	 * The container's storage scheme. \a storage_scheme is not exposed to the user as an option
+	 * A container's storage scheme. \a storage_scheme is not exposed to the user as an option
 	 * but can defined by ALP at different points in the execution depending on the \a backend choice.
 	 * For example, if the container is associated to an I/O matrix, with a reference backend
 	 * it might be set to reflect the storage scheme of the user data as specified at buildMatrix.
@@ -439,37 +513,32 @@ class StructuredMatrix {
 	 */
 	StorageSchemeType storage_scheme;
 
-	/** Whether the container presently is initialized or not. */
-	bool initialized;
+	/**
+	 * When a structured matrix defines a View over another matrix, it contains a pointer
+	 * to the latter. Its type can be identified via the View parameter.
+	 */
+	using target_type = typename std::enable_if<! std::is_same<View, view::Identity<void> >::value, typename View::applied_to>::type;
+	target_type * ref;
 
 	public :
 
-		/**
-	     * Usable only in case of an \a original view. Otherwise the view should only reference another view.
-	     */
-		StructuredMatrix( const size_t m, const size_t n );
+	StructuredMatrix( const size_t m, const size_t n );
 
-	/**
-	 * In case of non-original views should set up the container so to share the data of other.
-	 */
 	StructuredMatrix( const StructuredMatrix< T, Structure, StorageSchemeType, View, backend > & other );
 
-	/**
-	 * Usable only in case of an \a original view. Otherwise the view should only reference another view.
-	 */
 	StructuredMatrix( StructuredMatrix< T, Structure, StorageSchemeType, View, backend > && other );
 
-	/**
-	 * When view is \a original data should be properly deallocated.
-	 */
 	~StructuredMatrix();
-};
 
-template< typename InputType, Backend backend >
-RC clear( Matrix< InputType, backend > & A ) noexcept {
-	// this is the generic stub implementation
-	return UNSUPPORTED;
-}
+}; // class StructuredMatrix
+
+/**
+ * Check if type \a T is a StructuredMatrix.
+ */
+template< typename T >
+struct is_structured_matrix : std::false_type {};
+template< typename T, typename Structure, typename StorageSchemeType, typename View, enum Backend backend >
+struct is_structured_matrix< StructuredMatrix< T, Structure, StorageSchemeType, View, backend > > : std::true_type {};
 
 } // end namespace ``grb''
 

include/graphblas/benchmark.hpp

@@ -30,9 +30,9 @@
 #ifdef _GRB_WITH_REFERENCE
  #include "graphblas/reference/benchmark.hpp"
 #endif
-#ifdef _GRB_WITH_DENSE
- #include "graphblas/denseref/benchmark.hpp"
-#endif
+// #ifdef _GRB_WITH_DENSE
+//  #include "graphblas/denseref/benchmark.hpp"
+// #endif
 #ifdef _GRB_WITH_HYPERDAGS
  #include "graphblas/hyperdags/benchmark.hpp"
 #endif