Skip to content

Commit 80da577

Browse files
franzpoeschelfranzpoeschel
committed
Custom hierarchies (squashed commit)
JSON backend: Fail when trying to open non-existing groups Insert CustomHierarchy class to Iteration Help older compilers deal with this Add vector variants of meshes/particlesPath Move meshes and particles over to CustomHierarchies class Move dirtyRecursive to CustomHierarchy Move Iteration reading logic to CustomHierarchy Move Iteration flushing logic to CustomHierarchy class Support for custom datasets Treat "meshes"/"particles" as normal subgroups Introduction of iteration["meshes"].asContainerOf<Mesh>() as a more explicit variant for iteration.meshes. Regex-based list of meshes/particlesPaths More extended testing Fix Python bindings without adding new functionality yet Overload resolution Add simple Python bindings and an example Replace Regexes with Globbing TODO: Since meshes/particles can no longer be directly addressed with this, maybe adapt the class hierarchy to disallow mixed groups that contain meshes, particles, groups and datasets at the same time. Only maybe though.. Move .meshes and .particles back to Iteration class The have their own meaning now and are no longer just carefully maintained for backwards compatibility. Instead, they are supposed to serve as a shortcut to all openPMD data found further down the hierarchy. Some fixes in read error handling More symmetric design for container types Don't write unitSI in custom datasets Discouraged support for custom datasets inside the particlesPath Fix after rebase: dirtyRecursive Fixes to the dirty/dirtyRecursive logic [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Some cleanup in CustomHierarchies class Use polymorphism for meshes/particlesPath in Python Remove hasMeshes / hasParticles logic Sort dirty files This is a workaround only, only one file should be dirty in this test. Formatting [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Fixes after rebase
1 parent e248cc7 commit 80da577

34 files changed

+1883
-333
lines changed

‎CMakeLists.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -394,6 +394,7 @@ include(${openPMD_SOURCE_DIR}/cmake/dependencies/pybind11.cmake)
394394
set(CORE_SOURCE
395395
src/config.cpp
396396
src/ChunkInfo.cpp
397+
src/CustomHierarchy.cpp
397398
src/Dataset.cpp
398399
src/Datatype.cpp
399400
src/Error.cpp
@@ -578,6 +579,7 @@ if(openPMD_HAVE_PYTHON)
578579
src/binding/python/Attributable.cpp
579580
src/binding/python/BaseRecordComponent.cpp
580581
src/binding/python/ChunkInfo.cpp
582+
src/binding/python/CustomHierarchy.cpp
581583
src/binding/python/Dataset.cpp
582584
src/binding/python/Datatype.cpp
583585
src/binding/python/Error.cpp
@@ -746,6 +748,7 @@ set(openPMD_PYTHON_EXAMPLE_NAMES
746748
11_particle_dataframe
747749
12_span_write
748750
13_write_dynamic_configuration
751+
14_custom_hierarchy
749752
15_compression
750753
)
751754

‎examples/14_custom_hierarchy.py‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
import numpy as np
2+
import openpmd_api as io
3+
4+
5+
def main():
6+
if "bp" in io.file_extensions:
7+
filename = "../samples/custom_hierarchy.bp"
8+
else:
9+
filename = "../samples/custom_hierarchy.json"
10+
s = io.Series(filename, io.Access.create)
11+
it = s.write_iterations()[100]
12+
13+
# write openPMD part
14+
temp = it.meshes["temperature"]
15+
temp.axis_labels = ["x", "y"]
16+
temp.unit_dimension = {io.Unit_Dimension.T: 1}
17+
temp.position = [0.5, 0.5]
18+
temp.grid_spacing = [1, 1]
19+
temp.grid_global_offset = [0, 0]
20+
temp.reset_dataset(io.Dataset(np.dtype("double"), [5, 5]))
21+
temp[()] = np.zeros((5, 5))
22+
23+
# write NeXus part
24+
nxentry = it["Scan"]
25+
nxentry.set_attribute("NX_class", "NXentry")
26+
nxentry.set_attribute("default", "data")
27+
28+
data = nxentry["data"]
29+
data.set_attribute("NX_class", "NXdata")
30+
data.set_attribute("signal", "counts")
31+
data.set_attribute("axes", ["two_theta"])
32+
data.set_attribute("two_theta_indices", [0])
33+
34+
counts = data.as_container_of_datasets()["counts"]
35+
counts.set_attribute("units", "counts")
36+
counts.set_attribute("long_name", "photodiode counts")
37+
counts.reset_dataset(io.Dataset(np.dtype("int"), [15]))
38+
counts[()] = np.zeros(15, dtype=np.dtype("int"))
39+
40+
two_theta = data.as_container_of_datasets()["two_theta"]
41+
two_theta.set_attribute("units", "degrees")
42+
two_theta.set_attribute("long_name", "two_theta (degrees)")
43+
two_theta.reset_dataset(io.Dataset(np.dtype("double"), [15]))
44+
two_theta[()] = np.zeros(15)
45+
46+
47+
if __name__ == "__main__":
48+
main()

‎include/openPMD/CustomHierarchy.hpp‎

Lines changed: 256 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,256 @@
1+
/* Copyright 2023 Franz Poeschel
2+
*
3+
* This file is part of openPMD-api.
4+
*
5+
* openPMD-api is free software: you can redistribute it and/or modify
6+
* it under the terms of of either the GNU General Public License or
7+
* the GNU Lesser General Public License as published by
8+
* the Free Software Foundation, either version 3 of the License, or
9+
* (at your option) any later version.
10+
*
11+
* openPMD-api is distributed in the hope that it will be useful,
12+
* but WITHOUT ANY WARRANTY; without even the implied warranty of
13+
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14+
* GNU General Public License and the GNU Lesser General Public License
15+
* for more details.
16+
*
17+
* You should have received a copy of the GNU General Public License
18+
* and the GNU Lesser General Public License along with openPMD-api.
19+
* If not, see <http://www.gnu.org/licenses/>.
20+
*/
21+
#pragma once
22+
23+
#include "openPMD/IO/AbstractIOHandler.hpp"
24+
#include "openPMD/Mesh.hpp"
25+
#include "openPMD/ParticleSpecies.hpp"
26+
#include "openPMD/RecordComponent.hpp"
27+
#include "openPMD/backend/Container.hpp"
28+
29+
#include <memory>
30+
#include <regex>
31+
#include <set>
32+
#include <stdexcept>
33+
#include <string>
34+
#include <type_traits>
35+
#include <utility>
36+
#include <vector>
37+
38+
namespace openPMD
39+
{
40+
class CustomHierarchy;
41+
namespace internal
42+
{
43+
enum class ContainedType
44+
{
45+
Group,
46+
Mesh,
47+
Particle
48+
};
49+
struct MeshesParticlesPath
50+
{
51+
std::regex meshRegex;
52+
std::set<std::string> collectNewMeshesPaths;
53+
std::regex particleRegex;
54+
std::set<std::string> collectNewParticlesPaths;
55+
56+
/*
57+
* These values decide which path will be returned upon use of the
58+
* shorthand notation s.iterations[0].meshes or .particles.
59+
*
60+
*/
61+
std::string m_defaultMeshesPath = "meshes";
62+
std::string m_defaultParticlesPath = "particles";
63+
64+
explicit MeshesParticlesPath() = default;
65+
MeshesParticlesPath(
66+
std::vector<std::string> const &meshes,
67+
std::vector<std::string> const &particles);
68+
MeshesParticlesPath(Series const &);
69+
70+
[[nodiscard]] ContainedType
71+
determineType(std::vector<std::string> const &path) const;
72+
[[nodiscard]] bool
73+
isParticleContainer(std::vector<std::string> const &path) const;
74+
[[nodiscard]] bool
75+
isMeshContainer(std::vector<std::string> const &path) const;
76+
};
77+
78+
struct CustomHierarchyData
79+
: ContainerData<CustomHierarchy>
80+
, ContainerData<RecordComponent>
81+
, ContainerData<Mesh>
82+
, ContainerData<ParticleSpecies>
83+
{
84+
explicit CustomHierarchyData();
85+
86+
void syncAttributables();
87+
88+
#if 0
89+
inline Container<CustomHierarchy> customHierarchiesWrapped()
90+
{
91+
Container<CustomHierarchy> res;
92+
res.setData(
93+
{static_cast<ContainerData<CustomHierarchy> *>(this),
94+
[](auto const *) {}});
95+
return res;
96+
}
97+
#endif
98+
inline Container<RecordComponent> embeddedDatasetsWrapped()
99+
{
100+
Container<RecordComponent> res;
101+
res.setData(
102+
{static_cast<ContainerData<RecordComponent> *>(this),
103+
[](auto const *) {}});
104+
return res;
105+
}
106+
inline Container<Mesh> embeddedMeshesWrapped()
107+
{
108+
Container<Mesh> res;
109+
res.setData(
110+
{static_cast<ContainerData<Mesh> *>(this),
111+
[](auto const *) {}});
112+
return res;
113+
}
114+
115+
inline Container<ParticleSpecies> embeddedParticlesWrapped()
116+
{
117+
Container<ParticleSpecies> res;
118+
res.setData(
119+
{static_cast<ContainerData<ParticleSpecies> *>(this),
120+
[](auto const *) {}});
121+
return res;
122+
}
123+
124+
#if 0
125+
inline Container<CustomHierarchy>::InternalContainer &
126+
customHierarchiesInternal()
127+
{
128+
return static_cast<ContainerData<CustomHierarchy> *>(this)
129+
->m_container;
130+
}
131+
#endif
132+
inline Container<RecordComponent>::InternalContainer &
133+
embeddedDatasetsInternal()
134+
{
135+
return static_cast<ContainerData<RecordComponent> *>(this)
136+
->m_container;
137+
}
138+
inline Container<Mesh>::InternalContainer &embeddedMeshesInternal()
139+
{
140+
return static_cast<ContainerData<Mesh> *>(this)->m_container;
141+
}
142+
143+
inline Container<ParticleSpecies>::InternalContainer &
144+
embeddedParticlesInternal()
145+
{
146+
return static_cast<ContainerData<ParticleSpecies> *>(this)
147+
->m_container;
148+
}
149+
};
150+
} // namespace internal
151+
152+
template <typename MappedType>
153+
class ConversibleContainer : public Container<MappedType>
154+
{
155+
template <typename>
156+
friend class ConversibleContainer;
157+
158+
protected:
159+
using Container_t = Container<MappedType>;
160+
using Data_t = internal::CustomHierarchyData;
161+
static_assert(
162+
std::is_base_of_v<typename Container_t::ContainerData, Data_t>);
163+
164+
ConversibleContainer(Attributable::NoInit)
165+
: Container_t(Attributable::NoInit{})
166+
{}
167+
168+
std::shared_ptr<Data_t> m_customHierarchyData;
169+
170+
[[nodiscard]] Data_t &get()
171+
{
172+
return *m_customHierarchyData;
173+
}
174+
[[nodiscard]] Data_t const &get() const
175+
{
176+
return *m_customHierarchyData;
177+
}
178+
179+
inline void setData(std::shared_ptr<Data_t> data)
180+
{
181+
m_customHierarchyData = data;
182+
Container_t::setData(std::move(data));
183+
}
184+
185+
public:
186+
template <typename TargetType>
187+
auto asContainerOf() -> ConversibleContainer<TargetType>
188+
{
189+
if constexpr (
190+
std::is_same_v<TargetType, CustomHierarchy> ||
191+
std::is_same_v<TargetType, Mesh> ||
192+
std::is_same_v<TargetType, ParticleSpecies> ||
193+
std::is_same_v<TargetType, RecordComponent>)
194+
{
195+
ConversibleContainer<TargetType> res(Attributable::NoInit{});
196+
res.setData(m_customHierarchyData);
197+
return res;
198+
}
199+
else
200+
{
201+
static_assert(
202+
auxiliary::dependent_false_v<TargetType>,
203+
"[CustomHierarchy::asContainerOf] Type parameter must be "
204+
"one of: CustomHierarchy, RecordComponent, Mesh, "
205+
"ParticleSpecies.");
206+
}
207+
}
208+
};
209+
210+
class CustomHierarchy : public ConversibleContainer<CustomHierarchy>
211+
{
212+
friend class Iteration;
213+
friend class Container<CustomHierarchy>;
214+
215+
private:
216+
using Container_t = Container<CustomHierarchy>;
217+
using Parent_t = ConversibleContainer<CustomHierarchy>;
218+
using Data_t = typename Parent_t::Data_t;
219+
220+
using EraseStaleMeshes = internal::EraseStaleEntries<Container<Mesh>>;
221+
using EraseStaleParticles =
222+
internal::EraseStaleEntries<Container<ParticleSpecies>>;
223+
void readNonscalarMesh(EraseStaleMeshes &map, std::string const &name);
224+
void readScalarMesh(EraseStaleMeshes &map, std::string const &name);
225+
void readParticleSpecies(EraseStaleParticles &map, std::string const &name);
226+
227+
protected:
228+
CustomHierarchy();
229+
CustomHierarchy(NoInit);
230+
231+
void read(internal::MeshesParticlesPath const &);
232+
void read(
233+
internal::MeshesParticlesPath const &,
234+
std::vector<std::string> &currentPath);
235+
236+
void flush_internal(
237+
internal::FlushParams const &,
238+
internal::MeshesParticlesPath &,
239+
std::vector<std::string> currentPath);
240+
void flush(std::string const &path, internal::FlushParams const &) override;
241+
242+
/**
243+
* @brief Link with parent.
244+
*
245+
* @param w The Writable representing the parent.
246+
*/
247+
void linkHierarchy(Writable &w) override;
248+
249+
public:
250+
CustomHierarchy(CustomHierarchy const &other) = default;
251+
CustomHierarchy(CustomHierarchy &&other) = default;
252+
253+
CustomHierarchy &operator=(CustomHierarchy const &) = default;
254+
CustomHierarchy &operator=(CustomHierarchy &&) = default;
255+
};
256+
} // namespace openPMD

‎include/openPMD/IO/JSON/JSONIOHandlerImpl.hpp‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -399,7 +399,8 @@ class JSONIOHandlerImpl : public AbstractIOHandlerImpl
399399

400400
// make sure that the given path exists in proper form in
401401
// the passed json value
402-
static void ensurePath(nlohmann::json *json, std::string const &path);
402+
static void
403+
ensurePath(nlohmann::json *json, std::string const &path, Access);
403404

404405
// In order not to insert the same file name into the data structures
405406
// with a new pointer (e.g. when reopening), search for a possibly

0 commit comments

Comments
 (0)