Merge branch 'main' into 957-sparse-data-transfer

Author: LuloDuarte

CONTRIBUTING.md

@@ -14,9 +14,9 @@ See the License for the specific language governing permissions and
 limitations under the License.
 -->
 
-# Contributing to the DAPHNE System
+# Contributing to DAPHNE
 
-*Thank you* for your interest in contributing to the DAPHNE system.
+*Thank you* for your interest in contributing to DAPHNE.
 Our goal is to build an **open and inclusive community of developers** around the system.
 Thus, **contributions are highly welcome**, both from *within the DAPHNE project consortium* and from *external researchers/developers*.
 
@@ -25,6 +25,7 @@ In the following, you find some rough **guidelines on contributing**, which will
 ## Ways of Contributing
 
 There are **various ways of contributing** including (but not limited to):
+
 - actual implementation
 - writing test cases
 - writing documentation
@@ -36,7 +37,7 @@ That way, discussions are made *accessible and transparent* to everyone interest
 This is important to *involve people* and to *avoid repetition* in case multiple people have the same question/comment or encounter the same problem.
 So feel free to create an issue to start a discussion on a particular topic (including these contribution guidelines) or to report a bug or other problem.
 
-## Issue tracking
+## Issue Tracking
 
 All open/ongoing/completed **work is tracked as issues** on GitHub.
 These could be anything from precisely defined small tasks to requests for complex components.
@@ -65,31 +66,41 @@ That is, please try your best to make a good-quality contribution and we will he
 **The procedure is roughly as follows:**
 
 1. **Get assigned to the issue** to let others know you are going to work on it and to avoid duplicate work. Please leave a comment on the issue stating that you are going to work on it. After that, a collaborator will formally assign you.
+
 2. **Fork the repository** on GitHub and **clone your fork** (see [GitHub docs](https://docs.github.com/en/get-started/quickstart/fork-a-repo)).
-   We recommend cloning by `git clone --recursive https://github.com/<USERNAME>/daphne.git` (note the `--recursive`), as specified in [Getting Started](/doc/GettingStarted.md).
-   
-   *You may skip this step and reuse your existing fork if you have contributed before. Simply update your fork with the recent changes from the original DAPHNE repository (see [GitHub docs](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork)).*
+    <!-- TODO with containers, we don't recommend that anymore -->
+    We recommend cloning by `git clone --recursive https://github.com/<USERNAME>/daphne.git` (note the `--recursive`), as specified in [Getting Started](/doc/GettingStarted.md).
+    
+    *You may skip this step and reuse your existing fork if you have contributed before. Simply update your fork with the recent changes from the original DAPHNE repository (see [GitHub docs](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork)).*
+
 3. **Create your own local branch**: `git checkout -b BRANCH_NAME`.
-   `BRANCH_NAME` should clearly indicate what the branch is about; the recommended pattern is `123-some-short-title` (where `123` is the issue number).
+    `BRANCH_NAME` should clearly indicate what the branch is about; the recommended pattern is `123-some-short-title` (where `123` is the issue number).
+
 4. **Add as many commits as you like** to your branch, and `git push` them to your fork.
-   Use `git push --set-upstream origin BRANCH_NAME` when you push the first time.
-5. If you work longer on your contribution, make sure to **get the most recent changes from the upstream** (original DAPHNE system repository) from time to time (see [GitHub docs](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork)).
+    Use `git push --set-upstream origin BRANCH_NAME` when you push the first time.
+
+5. If you work longer on your contribution, make sure to **get the most recent changes from the upstream** (original DAPHNE repository) from time to time (see [GitHub docs](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork)).
+
 6. Once you feel ready (for integration or for discussion/feedback), **create a pull request** on GitHub (see [GitHub docs](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request)).
-   Normally, you'll want to ask for integration into `base:main`, the repo's default branch.
-   Please choose an expressive title and provide a short description of your changes.
-   Feel free to mark your pull request "WIP: " or "Draft: " in the title.
-   Note that you can add more commits to your pull request after you created it.
-   Ideally, the changes in the PR contain only the changes you made for that PR,
-   e.g, by rebasing your branch on top of the target branch. This makes it easier for others to
-   review your PR.
+    Normally, you'll want to ask for integration into `base:main`, the repo's default branch.
+    Please choose an expressive title and provide a short description of your changes.
+    Feel free to mark your pull request "WIP: " or "Draft: " in the title.
+    Note that you can add more commits to your pull request after you created it.
+    Ideally, the changes in the PR contain only the changes you made for that PR,
+    e.g, by rebasing your branch on top of the target branch. This makes it easier for others to
+    review your PR.
+    <!-- TODO link to reviewing guidelines -->
+
 7. [Resolve any open conflicts](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/addressing-merge-conflicts/about-merge-conflicts) to the target branch of the PR.
+
 8. You **receive feedback** on your proposed contribution.
-   You may be asked to apply certain changes, or we might apply straightforward adjustments ourselves before the integration.
+    You may be asked to apply certain changes, or we might apply straightforward adjustments ourselves before the integration.
+
 9. If it looks good (potentially after some help), **your contribution becomes a part of DAPHNE**.
 
 ### Experienced DAPHNE Contributors (Collaborators)
 
-We appreciate *continued commitment* to the DAPHNE system.
+We appreciate *continued commitment* to DAPHNE.
 Thus, **frequent contributors can become collaborators** on GitHub.
 Currently, this requires **at least three non-trivial contributions** to the system.
 Collaborators have *direct write access* to all branches of the repository, including the main branch.
@@ -99,25 +110,26 @@ Collaborators do not need to create a fork, and do not need to go through pull r
 At the same time, this freedom comes with certain responsibilities, which are roughly sketched here:
 
 1. Please **follow some simple guidelines when changing the code**:
-   - Feel free to directly push to the main branch, but *be mindful of what you commit*, since it will affect everyone.
-     As a guideline, commits fundamentally changing how certain things work should be announced and discussed first, whereas small changes or changes local to "your" component are not critical.
-   - But **never force push to the main branch**, since it can lead to severe inconsistencies in the Git history.
-   - Even *collaborators may still use pull requests* (just like new contributors) to suggest larger changes.
-     This is also suitable whenever you feel unsure about a change or want to get feedback first.
+    - Feel free to directly push to the main branch, but *be mindful of what you commit*, since it will affect everyone.
+      As a guideline, commits fundamentally changing how certain things work should be announced and discussed first, whereas small changes or changes local to "your" component are not critical.
+    - But **never force push to the main branch**, since it can lead to severe inconsistencies in the Git history.
+    - Even *collaborators may still use pull requests* (just like new contributors) to suggest larger changes.
+      This is also suitable whenever you feel unsure about a change or want to get feedback first.
 2. Please **engage in the [handling of pull requests](/doc/development/HandlingPRs.md)**; especially those affecting the components you are working on.
-   This includes:
-   - reading the code others suggest for integration
-   - trying if it works
-   - providing constructive and *actionable* feedback on improving the contribution prior to the integration
-   - actually merging a pull request in
-   
-   Balancing the handling of pull requests is important to *keep the development process scalable*.
+    This includes:
+    
+    - reading the code others suggest for integration
+    - trying if it works
+    - providing constructive and *actionable* feedback on improving the contribution prior to the integration
+    - actually merging a pull request in
+    
+    Balancing the handling of pull requests is important to *keep the development process scalable*.
 
 
 ### Code Style
 
 Before contributing, please make sure to run `clang-format` on your C++ (.h and
-.cpp) files. The codebase is currently formatted with `clang-format` version
+.cpp) files. The code base is currently formatted with `clang-format` version
 `18.1.3`. This is the default `clang-format` version when installing via `apt`
 on Ubuntu 24.04, and can easily be installed via `python -mpip install clang-format==18.1.3`
 on other systems.

UserConfig.json

@@ -22,13 +22,17 @@
     "explain_property_inference": false,
     "explain_sql": false,
     "explain_select_matrix_repr": false,
+    "explain_transfer_data_props": false,
     "explain_type_adaptation": false,
     "explain_vectorized": false,
     "explain_obj_ref_mgnt": false,
     "explain_mlir_codegen": false,
     "explain_mlir_codegen_sparsity_exploiting_op_fusion": false,
     "explain_mlir_codegen_daphneir_to_mlir": false,
     "explain_mlir_codegen_mlir_specific": false,
+    "enable_property_recording": false,
+    "enable_property_insert": false,
+    "properties_file_path": "properties.json",
     "taskPartitioningScheme": "STATIC",
     "numberOfThreads": -1,
     "minimumTaskSize": 1,

doc/BinaryFormat.md

@@ -17,7 +17,7 @@ limitations under the License.
 # Binary Data Format
 
 DAPHNE defines its own binary representation for the serialization of in-memory data objects (matrices/frames).
-This representation is intended to be used by default whenever we need to transfer or persistently store these in-memory objects, e.g., for
+This representation is intended to be used by default whenever we need to transfer or persistently store these in-memory objects, e.g., for:
 
 - the data transfer in the distributed runtime
 - a custom binary file format
@@ -32,7 +32,7 @@ At the moment, we focus on the case of a single block per data object.
 
 ## Binary Representation of a Whole Data Object
 
-The binary representation of a data object (matrix/frame) starts with a header containing general and data type-specific information.
+The binary representation of a data object (matrix/frame) starts with a header containing general and data-type-specific information.
 The data object is partitioned into rectangular blocks (in the extreme case, this can mean a single block).
 All blocks are represented individually (see binary representation of a single block below) and stored along with their position in the data object.
 
@@ -76,7 +76,7 @@ We currently support the following **value types**:
 | 9 | `float` |
 | 10 | `double` |
 
-Depending on the data type, there are more information in the header:
+Depending on the data type, there is more information in the header:
 
 *For `DenseMatrix` and `CSRMatrix`*:
 
@@ -108,8 +108,8 @@ size[B]   1    1    8    8      1               1         2     len[0]
 The body consists of a sequence of:
 
 - a pair of
-  - row index `rx` (uint64)
-  - column index `cx` (uint64)
+    - row index `rx` (uint64)
+    - column index `cx` (uint64)
 - a binary block representation
 
 For the special case of a single block, this looks as follows:
@@ -128,7 +128,7 @@ size[B]
 A single data block is a rectangular partition of a data object.
 In the extreme case, a single block can span the entire data object in both dimensions (one block per data object).
 
-General block header
+General block header:
 
 - number of rows `#r` (uint32)
 - number of columns `#c` (uint32)
@@ -143,7 +143,7 @@ addr[B]  0  3 4  7 8  8 9                        *
 size[B]    4    4    1               *
 ```
 
-## Block types
+## Block Types
 
 We define different block types to allow for a space-efficient representation depending on the data.
 When serializing a data object, the block types are not required to match the in-memory representation (e.g., the blocks of a `DenseMatrix` could use the *sparse* binary representation).
@@ -161,7 +161,7 @@ Most block types store their value type as part of the block type-specific infor
 Note that the value type used for the binary representation is not required to match the value type of the in-memory object (e.g., `DenseMatrix<uint64_t>` may be represented as a *dense* block with value type `uint8_t`, if the value range permits).
 Furthermore, each block may be represented using its individual value type.
 
-### Empty block
+### Empty Block
 
 This block type is used to represent blocks that contain only zeros of the respective value type very space-efficiently.
 
@@ -175,7 +175,7 @@ addr[B]  0  3 4  7 8 8
 size[B]    4    4   1
 ```
 
-### Dense block
+### Dense Block
 
 Block type-specific information:
 
@@ -192,17 +192,17 @@ addr[B]  0  3 4  7 8 8 9  9 10                             10+#r*#c*S
 size[B]    4    4   1    1      S         S                  S
 ```
 
-### Sparse block (compressed sparse row, CSR)
+### Sparse Block (Compressed Sparse Row, CSR)
 
 Block type-specific information:
 
 - value type `vt`  (uint8)
 - number of non-zeros in the block `#nzb` (uint64)
 - for each row
-  - number of non-zeros in the row `#nzr` (uint32)
-  - for each non-zero in the row
-    - column index `cx` (uint32)
-    - value `v` (value type `vt`)
+    - number of non-zeros in the row `#nzr` (uint32)
+    - for each non-zero in the row
+        - column index `cx` (uint32)
+        - value `v` (value type `vt`)
 
 Note that both a row and the entire block might contain no non-zeros.
 
@@ -233,20 +233,20 @@ size[B]    4    4   1    1     8              4+#nzr[i]*(4+S)
                                        4             S
 ```
 
-### Ultra-sparse block (coordinate, COO)
+### Ultra-Sparse Block (Coordinate, COO)
 
 Ultra-sparse blocks contain almost no non-zeros, so we want to keep the overhead of the meta data low.
 Thus, we distinguish blocks with a single column (where we don't need to store the column index) and blocks with more than one column.
 
-### Blocks with a single column
+### Blocks with a Single Column
 
 Block type-specific information:
 
 - value type `vt` (uint8)
 - number of non-zeros in the block `#nzb` (uint32)
 - for each non-zero
-  - row index `rx` (uint32)
-  - value `v` (value type `vt`)
+    - row index `rx` (uint32)
+    - value `v` (value type `vt`)
 
 Below, `S` denotes the size (in bytes) of a single value of type `vt`.
 
@@ -266,16 +266,16 @@ size[B]    4    4   1    1     4     4+S           4+S              4+S
                                              4          S
 ```
 
-### Blocks with more than one column
+### Blocks with More than One Column
 
 Block type-specific information:
 
 - value type `vt` (uint8)
 - number of non-zeros in the block `#nzb` (uint32)
 - for each non-zero
-  - row index `rx` (uint32)
-  - column index `cx` (uint32)
-  - value `v` (value type `vt`)
+    - row index `rx` (uint32)
+    - column index `cx` (uint32)
+    - value `v` (value type `vt`)
 
 Below, `S` denotes the size (in bytes) of a single value of type `vt`.
 

doc/BuildEnvironment.md

@@ -14,22 +14,21 @@ See the License for the specific language governing permissions and
 limitations under the License.
 -->
 
-# Building in unsupported environments
+# Building in Unsupported Environments
 
 DAPHNE can also be built in environments other than the one endorsed by the DAPHNE project team. To help with that,
-there are scripts to build your own toolchain in the [``buildenv``](/buildenv) directory.
+there are scripts to build your own toolchain in the [`buildenv`](/buildenv) directory.
 These scripts can be used to build the compiler and necessary libraries to build DAPHNE in unsupported environments 
 (RHEL UBI, CentOS, ...) Besides the build scripts, there is a docker file to create a UBI image to build for Redhat 8.
 
-
 Usage:
-* Create Docker image with build-ubi8.sh
-* Run the Docker image with run-ubi8.sh
-* Run build-all.sh
-* Run source env.sh inside or outside the Docker image to set PATH and linker variables to use the created environment 
-(you need to cd into the directory containing env.sh as this uses $PWD)
 
+* Create the Docker image with `build-ubi8.sh`
+* Run the Docker image with `run-ubi8.sh`
+* Run `build-all.sh`
+* Run `source env.sh` inside or outside the Docker image to set `PATH` and linker variables to use the created environment 
+(you need to cd into the directory containing `env.sh` as this uses `$PWD`)
 
 Beware that this procedure needs ~50GB of free disk space. Also, the provided CUDA SDK expects a recent driver (550+) 
 version. That will most likely be an issue on large installations - exchange the relevant version and file names for 
-another CUDA version in env.sh to build another version.
+another CUDA version in `env.sh` to build another version.