Rework serializing #15

Alex-PLACET · 2025-09-02T13:11:33Z

No description provided.

pitrou · 2025-09-04T09:39:08Z

include/sparrow_ipc/encapsulated_message.hpp

+
+    private:
+
+        const uint8_t* m_buf_ptr;


If you don't store the buffer length, how can you make sure you're not accessing memory out of bounds?

pitrou · 2025-09-04T09:43:07Z

include/sparrow_ipc/encapsulated_message.hpp

+        const uint8_t* m_buf_ptr;
+    };
+
+    [[nodiscard]] EncapsulatedMessage create_encapsulated_message(const uint8_t* buf_ptr);


Perhaps you want an API like:

// Return the encapsulated message and the rest of the span std::pair<EncapsulatedMessage, std::span<const uint8_t>> extract_encapsulated_message(std::span<const uint8_t>);

Hind-M · 2025-09-04T11:15:49Z

include/sparrow_ipc/deserialize_primitive_array.hpp

+namespace sparrow_ipc
+{
+    template <typename T>
+    [[nodiscard]] sparrow::primitive_array<T> deserialize_primitive_array_bis(


Maybe rename deserialize_primitive_array_bis to deserialize_primitive_array_from_record_batch and deserialize_primitive_array to deserialize_primitive_array_from_buffer?

Hind-M · 2025-09-04T11:39:29Z

src/encapsulated_message.cpp

+    {
+        const size_t offset = sizeof(uint32_t) * 2  // 4 bytes continuation + 4 bytes metadata size
+                      + metadata_length();
+        const size_t padded_offset = (offset + 7) & ~7;  // Round up to 8-byte boundary


We should use align_to_8 function everywhere instead of replicating it.
(You can change & -8 to & ~7 if preferred)

Hind-M · 2025-09-04T11:39:40Z

src/encapsulated_message.cpp

+    {
+        const size_t offset = sizeof(uint32_t) * 2  // 4 bytes continuation + 4 bytes metadata size
+                      + metadata_length();
+        const size_t padded_offset = (offset + 7) & ~7;  // Round up to 8-byte boundary


Hind-M · 2025-09-04T12:49:45Z

As a side note, I think in the future, and for relatively big PRs with multiple files like this one, having multiple commits where corresponding messages describe what they are doing would make the review easier.
The linting changes are adding some noise as well, maybe keep that in specific independent PRs in the future.

I don't know if the additional code is making the project not buildable anymore, but maybe it's best if we remove all the versioning and install parts to focus on the core changes in this PR.

Hind-M · 2025-09-04T12:52:07Z

Are we planning eventually to use functions from sparrow directly (everything related to arrow_interface and comparison functions in the tests)?

wip wip wip WIP wip wip wip wip wip

Copilot

Pull Request Overview

This PR implements a comprehensive rework of the serialization system to improve code organization, add new functionality, and enhance testing infrastructure. The changes restructure the codebase with proper namespacing, introduce new deserialization capabilities, and add extensive integration testing with Arrow data files.

Reorganized headers and source files with proper sparrow_ipc namespace structure
Added new deserialization functionality for streams and various array types
Introduced comprehensive integration testing with Arrow testing data files

Reviewed Changes

Copilot reviewed 41 out of 44 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/test_utils.cpp	Reformatted test assertions for better readability
tests/test_primitive_array_with_files.cpp	New integration tests comparing stream vs JSON deserialization
tests/test_primitive_array_serialization.cpp	Updated includes and minor formatting improvements
tests/test_null_array_serialization.cpp	Updated includes to use new header structure
tests/test_arrow_schema.cpp	New comprehensive tests for Arrow schema functionality
tests/metadata_sample.hpp	New helper for metadata testing with endianness support
tests/CMakeLists.txt	Added new test files and dependencies
src/utils.cpp	Updated includes and improved code formatting
src/serialize_null_array.cpp	Updated to use new deserialization functions
src/serialize.cpp	Moved deserialization functions and improved formatting
src/metadata.cpp	New utility for metadata conversion
src/encapsulated_message.cpp	New class for handling encapsulated Arrow messages
src/deserialize_utils.cpp	New utilities for deserialization operations
src/deserialize_fixedsizebinary_array.cpp	New deserialization for fixed-size binary arrays
src/deserialize.cpp	New comprehensive deserialization implementation
Multiple header files	Reorganized with proper namespace structure and new functionality

Comments suppressed due to low confidence (3)

src/utils.cpp:1

[nitpick] These multi-line trailing comments are hard to read and maintain. Consider moving them above the variable declarations or making them single-line comments.

#include "sparrow_ipc/utils.hpp"

src/utils.cpp:1

[nitpick] These multi-line trailing comments are hard to read and maintain. Consider moving them above the variable declarations or making them single-line comments.

#include "sparrow_ipc/utils.hpp"

src/utils.cpp:1

[nitpick] These multi-line trailing comments are hard to read and maintain. Consider moving them above the variable declarations or making them single-line comments.

#include "sparrow_ipc/utils.hpp"

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

tests/test_utils.cpp

Copilot · 2025-09-05T13:14:29Z

src/utils.cpp

+                    const auto map_type = org::apache::arrow::flatbuf::CreateMap(builder, false);  // not
+                                                                                                   // sorted
+                                                                                                   // keys


[nitpick] The multi-line comment is unnecessarily fragmented. Consider using a single-line comment: // not sorted keys or moving the comment above the line.

Suggested change

const auto map_type = org::apache::arrow::flatbuf::CreateMap(builder, false); // not

// sorted

// keys

const auto map_type = org::apache::arrow::flatbuf::CreateMap(builder, false); // not sorted keys

Copilot · 2025-09-05T13:14:29Z

include/sparrow_ipc/encapsulated_message.hpp

@@ -0,0 +1,44 @@
+#include <cstdint>


Missing header guard. Add #pragma once at the beginning of the file to prevent multiple inclusions.

JohanMabille · 2025-09-05T06:56:56Z

include/sparrow_ipc/arrow_interface/arrow_schema/private_data.hpp

+    private:
+
+        std::string m_format;
+        const char* m_name;


Why not storing an optional string instead?

We keep a direct pointer to the name in the flatbuffer . The format is not stored as a string in the flatbuffer.

JohanMabille · 2025-09-05T07:03:37Z

include/sparrow_ipc/config/sparrow_ipc_version.hpp

+    constexpr int SPARROW_IPC_VERSION_MINOR = 1;
+    constexpr int SPARROW_IPC_VERSION_PATCH = 0;
+
+    constexpr int SPARROW_IPC_BINARY_CURRENT = 9;


Nitpicking: I think we can keep 1 for the binary_current version.

JohanMabille · 2025-09-05T07:04:39Z

include/sparrow_ipc/deserialize.hpp

+
+#include <sparrow/record_batch.hpp>
+
+#include "config/config.hpp"


Suggested change

#include "config/config.hpp"

#include "sparrow_ipc/config/config.hpp"

JohanMabille · 2025-09-05T14:27:47Z

src/deserialize.cpp

+                    {
+                        switch (bit_width)
+                        {
+                                // clang-format off


JohanMabille · 2025-09-05T14:31:53Z

include/sparrow_ipc/serialize_primitive_array.hpp

@@ -50,18 +53,22 @@ namespace sparrow_ipc
    }

    template <typename T>
-    sparrow::primitive_array<T> deserialize_primitive_array(const std::vector<uint8_t>& buffer) {
+    sparrow::primitive_array<T> deserialize_primitive_array(const std::vector<uint8_t>& buffer)


This should live in deserialize_primitive_array.hpp, or deserialize_primitive_array.hpp should be merged into this file.

JohanMabille · 2025-09-05T14:46:11Z

include/sparrow_ipc/encapsulated_message.hpp

+
+namespace sparrow_ipc
+{
+    class EncapsulatedMessage


Code convention: should be snake case.

JohanMabille · 2025-09-05T14:49:24Z

include/sparrow_ipc/serialize.hpp

+#include <sparrow/buffer/dynamic_bitset/dynamic_bitset_view.hpp>
+#include <sparrow/record_batch.hpp>
+
+#include "config/config.hpp"


Suggested change

#include "config/config.hpp"

#include "sparrow_ipc/config/config.hpp"

JohanMabille · 2025-09-05T14:49:53Z

include/sparrow_ipc/serialize.hpp

+            const ArrowArray& arrow_arr,
+            const std::vector<int64_t>& buffers_sizes,
+            std::vector<uint8_t>& final_buffer
+        );


Why having these methods nested in a details namespace while their deserialize "counterparts" are not?

JohanMabille · 2025-09-05T14:51:25Z

include/sparrow_ipc/serialize_primitive_array.hpp

+#include "deserialize.hpp"
 #include "serialize.hpp"
 #include "utils.hpp"


Suggested change

#include "deserialize.hpp"

#include "serialize.hpp"

#include "utils.hpp"

#include "sparrow_ipc/deserialize.hpp"

#include "sparrow_ipc/serialize.hpp"

#include "sparrow_ipc/utils.hpp"

JohanMabille · 2025-09-05T14:52:01Z

include/sparrow_ipc/utils.hpp

+#include <string_view>
+#include <utility>
+
+#include "config/config.hpp"


Suggested change

#include "config/config.hpp"

#include "sparrow_ipc/config/config.hpp"

Alex-PLACET self-assigned this Sep 2, 2025

Alex-PLACET requested a review from Copilot September 3, 2025 15:05

Alex-PLACET force-pushed the rework_serializing branch 2 times, most recently from 85ef02d to f82d723 Compare September 3, 2025 15:07

This comment was marked as outdated.

Sign in to view

Alex-PLACET marked this pull request as ready for review September 4, 2025 09:27

Alex-PLACET requested review from JohanMabille and Hind-M September 4, 2025 09:28

pitrou reviewed Sep 4, 2025

View reviewed changes

Hind-M reviewed Sep 4, 2025

View reviewed changes

Alex-PLACET added 4 commits September 4, 2025 15:38

Rework serialization

df92795

wip wip wip WIP wip wip wip wip wip

wip

190af27

wip

c6f0202

wip

620ea81

Alex-PLACET force-pushed the rework_serializing branch from 9347b29 to 620ea81 Compare September 4, 2025 13:38

Alex-PLACET added 8 commits September 4, 2025 15:46

Fix osx build

9fd39b8

fix compilation

5b779ba

wip

9c32fc0

compilation fix

1c1cd18

wip

2db5795

Use std::span

e618b0b

wip

4b8cf6c

wip

db55113

Alex-PLACET requested review from pitrou, Hind-M and Copilot September 5, 2025 13:13

Copilot AI reviewed Sep 5, 2025

View reviewed changes

fix

9294c41

JohanMabille reviewed Sep 5, 2025

View reviewed changes


		#include <sparrow/record_batch.hpp>

		#include "config/config.hpp"

	#include "config/config.hpp"
	#include "sparrow_ipc/config/config.hpp"

Rework serializing #15

Are you sure you want to change the base?

Rework serializing #15

Uh oh!

Conversation

Alex-PLACET commented Sep 2, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hind-M commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Hind-M commented Sep 4, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Hind-M commented Sep 4, 2025 •

edited

Loading