Skip to content

Commit bc10a7d

Browse files
d-kamathd-kamath
authored
Merge pull request #666 from UWB-Biocomputing/issue-581-serialization-documentation
Added Serialization Documentation
2 parents b36fd7c + b6de0d4 commit bc10a7d

File tree

2 files changed

+383
-0
lines changed

2 files changed

+383
-0
lines changed

‎docs/Developer/Serialization.md‎

Lines changed: 381 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,381 @@
1+
# Serialization and Deserialization using Cereal
2+
3+
This guide explains how to implement serialization using the Cereal library within Graphitti. If you're looking to add serialization to your class, follow this guide. For more comprehensive information on Cereal, refer to their [official documentation](https://uscilab.github.io/cereal/index.html).
4+
5+
<strong>
6+
What is Serialization and Deserialization?
7+
</strong>
8+
9+
Serialization involves converting an object or data structure into a format that can be stored or transmitted, while deserialization is the reverse process— reconstructing an object from the serialized format.
10+
<br>
11+
12+
<strong>
13+
What is Cereal?
14+
</strong>
15+
16+
Cereal is a lightweight C++11 library designed for object serialization and deserialization. It provides a straightforward interface for serializing objects and supports a wide range of data types. Cereal is efficient and well-suited for handling large data sets, making it a preferred choice for serialization tasks in Graphitti.
17+
<br>
18+
19+
20+
<strong>
21+
Why Graphitti Uses Serialization?
22+
</strong>
23+
24+
Graphitti utilizes Cereal to enable efficient serialization and deserialization of its simulation state and network structure. Serialized data can serve as a checkpoint for large simulations or as input for subsequent simulations with varying conditions. This flexibility enhances Graphitti's efficiency and adaptability in modeling scenarios.
25+
<br>
26+
27+
## Understand Cereal at a High Level
28+
29+
### C++ Features Supported by Cereal
30+
31+
- **Standard Library Support**: Cereal fully supports the C++11 [standard library](http://en.cppreference.com/w/). You can include the necessary headers from Cereal to enable support for various types (e.g., If you need to serialize a `std::vector`, add `<cereal/types/vector.hpp>`). Refer to the complete list of supported types below or in [Cereal Doxygen docs](https://uscilab.github.io/cereal/assets/doxygen/group__STLSupport.html).
32+
33+
- **Smart Pointers**: Cereal supports modern smart pointers (`std::shared_ptr` and `std::unique_ptr`) via `<cereal/types/memory.hpp>`. <strong> However, raw pointers or references are NOT supported.</strong>
34+
35+
- **Inheritance and Polymorphism**: Cereal can seamlessly handles inheritance and polymorphism- more on this in the coming section.
36+
37+
### Serialization Archive Types
38+
39+
- **Supported Archive Types**: Cereal provides three basic archive types: binary (with a portable version), XML, and JSON. Graphitti primarily utilizes binary and XML archives, managed within the `Serializer.cpp` class.
40+
41+
- **RAII Handling**: Cereal archives are designed to be used in a Resource Acquisition Is Initialization (RAII) manner and flush their contents only upon destruction.
42+
43+
- **Serialization Order**: By default, Cereal deserializes data in the order it was serialized.
44+
45+
### Serialization Functions in Cereal
46+
47+
- **Defining Serialization Functions**: Cereal requires to know which data members to serialize in a class. By implementing a serialization function `serialize` within a class, you indicate to Cereal which data members should be serialized.
48+
49+
- **Default Constructor Requirement**: Cereal requires access to a default constructor for classes it serializes to construct the object during deserialization.
50+
51+
- **Name-Value Pairs**: Cereal supports name-value pairs, allowing you to attach names to serialized objects. This feature is particularly useful for XML archives and is adopted in Graphitti's serialization process.
52+
53+
- **Class Versioning**: While still in work in progress in Graphitti, Cereal supports class versioning, enabling compatibility between different versions of serialized objects.
54+
55+
## Incorporate Cereal in your class
56+
57+
- Cereal supports two approaches for serialization functions: internal or external. Cereal also provides two types of serialization function `serialize` and `load and save`. For consistency within Graphitti, use an internal single `serialize` function.
58+
- Throughout Graphitti, we typically serialize all data members (private, protected, and public) of a class.
59+
60+
### **STEP 01: ADD CEREAL HEADERS**
61+
Before implementing serialization in your class, you need to include the appropriate Cereal headers for the types of data members you want to serialize. Cereal provides headers for various standard types, such as vectors, strings, and other containers.
62+
63+
- For example, if you're serializing a `std::vector<int>`, you'll need to include the following header:
64+
```cpp
65+
#include <cereal/types/vector.hpp>
66+
```
67+
68+
- If you are using custom types, ensure the serialization process is correctly implemented for those types as well. The same approach applies to user-defined types, so include the appropriate headers and define the serialize function in that class or struct.
69+
70+
<details>
71+
72+
<summary><strong>Commonly used C++ data types and their corresponding Cereal headers</strong></summary>
73+
74+
75+
| Type | Header to include |
76+
|:--------------------|:----------------------------------------------|
77+
| `std::array` | `#include <cereal/types/array.hpp>` |
78+
| `std::atomic` | `#include <cereal/types/atomic.hpp>` |
79+
| `std::bitset` | `#include <cereal/types/bitset.hpp>` |
80+
| `std::chrono` | `#include <cereal/types/chrono.hpp>` |
81+
| `std::complex` | `#include <cereal/types/complex.hpp>` |
82+
| `std::deque` | `#include <cereal/types/deque.hpp>` |
83+
| `std::forward_list` | `#include <cereal/types/forward_list.hpp>` |
84+
| `std::functional` | `#include <cereal/types/functional.hpp>` |
85+
| `std::list` | `#include <cereal/types/list.hpp>` |
86+
| `std::map` | `#include <cereal/types/map.hpp>` |
87+
| `std::memory` | `#include <cereal/types/memory.hpp>` |
88+
| `std::optional` | `#include <cereal/types/optional.hpp>` |
89+
| `std::queue` | `#include <cereal/types/queue.hpp>` |
90+
| `std::set` | `#include <cereal/types/set.hpp>` |
91+
| `std::stack` | `#include <cereal/types/stack.hpp>` |
92+
| `std::string` | `#include <cereal/types/string.hpp>` |
93+
| `std::tuple` | `#include <cereal/types/tuple.hpp>` |
94+
| `std::unordered_map`| `#include <cereal/types/unordered_map.hpp>` |
95+
| `std::unordered_set`| `#include <cereal/types/unordered_set.hpp>` |
96+
| `std::utility` | `#include <cereal/types/utility.hpp>` |
97+
| `std::valarray` | `#include <cereal/types/valarray.hpp>` |
98+
| `std::variant` | `#include <cereal/types/variant.hpp>` |
99+
| `std::vector` | `#include <cereal/types/vector.hpp>` |
100+
101+
</details>
102+
103+
<br>
104+
105+
### **STEP 02: ADD SERIALIZE FUNCTION**
106+
107+
Within your class header file
108+
- Firstly, declare the `serialize` function inside the class:
109+
- Add the following template function signature in the **public** section of your class to allow serialization for any archive type.
110+
```cpp
111+
template <class Archive>
112+
void serialize(Archive & archive);
113+
```
114+
- Secondly, define the `serialize` function outside the class:
115+
- After your class declaration, define the serialize function at the end of the header file.
116+
- This function specifies which member variables are serialized and deserialized, and how that process occurs.
117+
```cpp
118+
template <class Archive>
119+
void YOUR_CLASS_NAME::serialize(Archive &archive)
120+
{
121+
archive(ADD_YOUR_MEMBER_VARIABLES_HERE);
122+
// Refer to the example below
123+
}
124+
```
125+
Here’s a sample implementation to guide you:
126+
127+
``` cpp
128+
129+
// STEP 01: Add Necessary Header
130+
#include <cereal/vector.hpp>
131+
132+
class MyCoolClass
133+
{
134+
135+
public:
136+
// STEP 02 (a): Declare the serialize function in the public section
137+
template <class Archive>
138+
void serialize( Archive & archive );
139+
140+
private:
141+
std::vector<int> myVector_;
142+
int x_;
143+
};
144+
145+
//STEP 02 (b): Define the serialize function outside the class
146+
147+
template <class Archive>
148+
void MyCoolClass::serialize(Archive &archive)
149+
{
150+
archive(cereal::make_nvp("myVector", myVector_), cereal::make_nvp("myInt", x_));
151+
}
152+
153+
154+
```
155+
156+
Adjust the function names and data member names as per your specific requirements.
157+
158+
NOTE:
159+
160+
- The `template <class Archive>` declaration allows the serialize function to be flexible, enabling it to work with different types of Cereal archives, such as JSON, XML, or binary formats.
161+
162+
- When defining the `serialize` function, use `make_nvp()` and `CEREAL_NVP()` for each member variable:
163+
- `make_nvp()` is used when you want to assign custom names to your serialized member variables, which can be helpful for clarity in the serialized output.
164+
- `CEREAL_NVP()` automatically uses the variable's name for serialization without the need to explicitly name it.
165+
166+
- Why define `serialize` in the header?
167+
- Cereal relies heavily on templates, and C++ templates require full implementation details to be available during compilation for proper instantiation. Since templates must be instantiated at compile time, placing the serialize function in a `.cpp` file could result in missing template information, leading to linker errors. By defining the function in the header file, the compiler has all the necessary information to properly instantiate the serialize function for various data types.
168+
169+
- Defining the function outside the class (but still in the header) promotes a clean code style, making the class declaration less cluttered and easier to maintain. It also makes serialized code easier to find, which is especially important in larger projects like Graphitti.
170+
171+
### **STEP 03: SPECIAL CASES**
172+
173+
Cereal requires additional steps for certain special cases such as inheritance, polymorphism, and templates. In this section, we outline specific steps based on whether your class matches one of these conditions. If you answer "yes" to any of the following questions, follow the corresponding steps. Some classes may fall under multiple categories, so be sure to review all the details carefully.
174+
175+
### **1. DERIVED CLASS?**
176+
This step explains how to serialize base classes in a derived class. Cereal requires a path from the derived to the base type(s), typically done with `cereal::base_class` or `cereal::virtual_base_class`.
177+
178+
<details>
179+
<summary><strong> Virtual Inheritance ? </summary></strong>
180+
If your derived class uses virtual inheritance (`class Derived : virtual Base`), use `cereal::virtual_base_class<BaseT>(this)` to cast the derived class to its base class. Ensure this is placed at the start of the `archive` in the `serialize` function before member variables in the derived class.
181+
182+
```cpp
183+
184+
class MyDerived : virtual MyBase
185+
{
186+
int x_;
187+
template <class Archive>
188+
void serialize( Archive & ar );
189+
};
190+
191+
template <class Archive>
192+
void MyDerived::serialize( Archive & archive )
193+
{
194+
// We pass this cast to the base type for each base type we need to serialize.
195+
archive(cereal::virtual_base_class<MyBase>(this), cereal::make_nvp("myInt", x_));
196+
197+
// For multiple inheritance, link all the base classes one after the other
198+
//archive(cereal::virtual_base_class<MyBase1>(this), cereal::virtual_base_class<MyBase2>(this), cereal::make_nvp("myInt", X_));
199+
}
200+
```
201+
</details>
202+
203+
<details>
204+
<summary><strong> Normal Inheritance ?</summary></strong>
205+
For non-virtual inheritance (`class Derived : public Base`), use `cereal::base_class<BaseT>(this) to serialize the base class. Ensure this is placed at the start of the `archive` in the `serialize` function before member variables in the derived class.
206+
207+
```cpp
208+
209+
class MyDerived : public MyBase
210+
{
211+
int x_;
212+
template <class Archive>
213+
void serialize( Archive & ar );
214+
};
215+
216+
template <class Archive>
217+
void MyDerived::serialize( Archive & archive )
218+
{
219+
// We pass this cast to the base type for each base type we need to serialize.
220+
archive(cereal::base_class<MyBase>(this), cereal::make_nvp("myInt", x_));
221+
222+
// For multiple inheritance, link all the base classes one after the other
223+
//archive(cereal::base_class<MyBase1>(this), cereal::base_class<MyBase2>(this), cereal::make_nvp("myInt", X_));
224+
}
225+
```
226+
For more details, refer to the official Cereal documentation on [inheritance](https://uscilab.github.io/cereal/inheritance.html)
227+
228+
</details>
229+
230+
### **2. EXHIBIT POLYMORPHISM?**
231+
232+
If you answered "yes" to the previous question about your class being a derived class, this is likely "yes" as well.
233+
234+
<details>
235+
<summary><strong>
236+
Follow these steps if your class exhibits polymorphic behavior:
237+
</summary></strong>
238+
239+
1. Include Necessary Headers:
240+
241+
Make sure to include the polymorphic header to enable support for polymorphism in Cereal in the derived class.
242+
``` #include <cereal/types/polymorphic.hpp> ```
243+
244+
2. Register Your Derived Types:
245+
246+
Register each derived class above the definition of the `serialize` function using `CEREAL_REGISTER_TYPE(DerivedClassName)` in the respective derived class.
247+
248+
```cpp
249+
// be sure to include support for polymorphism
250+
#include <cereal/types/polymorphic.hpp>
251+
252+
class MyDerived : public MyBase
253+
{
254+
int x_;
255+
template <class Archive>
256+
void serialize( Archive & ar );
257+
};
258+
259+
//Registering the Derived class
260+
CEREAL_REGISTER_TYPE(MyDerived);
261+
262+
template <class Archive>
263+
void MyDerived::serialize( Archive & archive )
264+
{
265+
archive(cereal::base_class<MyBase>(this), cereal::make_nvp("myInt", x_));
266+
}
267+
```
268+
269+
3. Register Your Base Class (if not registered automatically):
270+
271+
Normally, registering base classes is handled automatically if you serialize a derived type with either `cereal::base_class` or `cereal::virtual_base_class`. However, in situations where neither of these is used, explicit registration is required using the `CEREAL_REGISTER_POLYMORPHIC_RELATION` macro in the derived class.
272+
273+
```cpp
274+
struct MyEmptyBase
275+
{
276+
virtual void foo() = 0;
277+
};
278+
279+
struct MyDerived: MyEmptyBase
280+
{
281+
void foo() {}
282+
double y_;
283+
template <class Archive>
284+
void serialize( Archive & archive );
285+
};
286+
287+
CEREAL_REGISTER_TYPE(MyDerived)
288+
289+
//Registering the Base Class
290+
CEREAL_REGISTER_POLYMORPHIC_RELATION(MyEmptyBase, MyDerived)
291+
292+
template <class Archive>
293+
void MyDerived::serialize( Archive & archive )
294+
{
295+
archive( cereal::make_nvp("myDouble", y_) );
296+
}
297+
```
298+
299+
For more detailed information and examples on polymorphism in Cereal, refer to the official documentation on [Polymorphism](https://uscilab.github.io/cereal/polymorphism.html).
300+
301+
</details>
302+
303+
### **3. TEMPLATE?**
304+
<details>
305+
<summary><strong>
306+
Template involves inheritance?
307+
</summary></strong>
308+
309+
Follow all the steps from STEP 01 as if your class is a regular class. However, if the template involves inheritance, you might need to register all potential instantiations of the template during polymorphism handling.
310+
311+
```cpp
312+
313+
// Include necessary Cereal headers
314+
#include <cereal/types/polymorphic.hpp>
315+
#include <cereal/types/vector.hpp>
316+
317+
// A pure virtual base class
318+
struct BaseClass
319+
{
320+
virtual void sayType() = 0;
321+
};
322+
323+
// A templated class derived from BaseClass
324+
template <typename T>
325+
struct DerivedClassTemplate : public BaseClass
326+
{
327+
T value_;
328+
void sayType();
329+
330+
template <class Archive>
331+
void serialize(Archive & archive)
332+
{
333+
archive(cereal::virtual_base_class<MyEmptyBase>(this), cereal::make_nvp("myValue", value_));
334+
}
335+
};
336+
337+
// Register template instantiations
338+
CEREAL_REGISTER_TYPE(DerivedClassTemplate<int>);
339+
CEREAL_REGISTER_TYPE(DerivedClassTemplate<float>);
340+
341+
// If using Register polymorphic relationships
342+
// CEREAL_REGISTER_POLYMORPHIC_RELATION(BaseClass, DerivedClassTemplate<int>);
343+
// CEREAL_REGISTER_POLYMORPHIC_RELATION(BaseClass, DerivedClassTemplate<float>);
344+
345+
```
346+
</details>
347+
348+
### **4. NO DEFAULT CONSTRUCTOR?**
349+
350+
Cereal provides a special overload method to handle this situation. Refer to the [Cereal documentation](https://uscilab.github.io/cereal/pointers) for detailed information on this technique.
351+
352+
## Common Cereal Errors
353+
354+
Encountering a Cereal error during compiling or running Graphitti? Here's a checklist to troubleshoot:
355+
356+
1. **Include Correct Cereal Headers**: Ensure you've included the necessary Cereal headers for the types you're serializing. If using polymorphic serialization, include `#include <cereal/types/polymorphic.hpp>`.
357+
358+
2. **Default Constructor**: Verify that your class has a default constructor. If not possible, utilize Cereal's special overload methods for handling this scenario.
359+
360+
3. **Polymorphic Type Registration**: If serialization of polymorphic types fails or results in incorrect type information, double-check your type registration. Use `CEREAL_REGISTER_TYPE` and `CEREAL_REGISTER_POLYMORPHIC_RELATION` to register polymorphic types correctly.
361+
362+
4. **Runtime Exceptions**: If encountering a runtime exception like
363+
```
364+
what(): Trying to save an unregistered polymorphic type (AllDSSynapses).
365+
Make sure your type is registered with CEREAL_REGISTER_TYPE and that the archive you are using was included (and registered with CEREAL_REGISTER_ARCHIVE) prior to calling CEREAL_REGISTER_TYPE.
366+
```
367+
include the following two archive headers for the respective class:
368+
369+
```
370+
#include <cereal/archives/portable_binary.hpp>
371+
#include <cereal/archives/xml.hpp>
372+
```
373+
Reasoning: Polymorphic type registration requires mapping your registered type to archives included prior to CEREAL_REGISTER_TYPE being called. Missing archive headers in certain classes could lead to this error.
374+
375+
With these checks, you should be able to diagnose and resolve common Cereal errors in Graphitti.
376+
377+
---------
378+
[<< Go back to the Developer Documentation page](index.md)
379+
380+
---------
381+
[<< Go back to the Graphitti home page](../index.md)

‎docs/index.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,8 @@
3030

3131
2.7 [Unit Tests](Developer/UnitTests.md)
3232

33+
2.8 [Serialization](Developer/Serialization.md)
34+
3335
4. [Testing](Testing/index.md)
3436

3537
3.1 Array Performance Testing

0 commit comments

Comments
 (0)