Discuss deprecation of dat file format

[sebproell]

I want to use this thread to discuss if, how and when to deprecate the dat format. Depending on the outcome, we can derive the concrete tasks in an issue.

Everybody uses input files, so tagging all @4C-multiphysics/developers and @gilrrei for QUEENS. Please voice your ideas or concerns. Here are ideas that came up in discussion with a few of you.

Why should we deprecate the dat format?

• It is a format that we made up. We would certainly not do this today but use a standard format. Interaction with dat files is a hassle, not only for 4C itself but for all external scripting.
• The transition is rather straightforward. We have yaml support as an alternative. Conversion to yaml is trivial because 4C can read in dat and dump out equivalent yaml (needs a small implementation I haven't done yet in 4C to enable this).
• We made input facilities more consistent and powerful, with a format like yaml in mind, which supports schema files to add all of the information to a way better user experience. The knowledge we have is not exportable to dat to the same extent.
• Maintaining a separate path for dat files is an overhead, especially because dat is essentially unparseable without knowing what exactly you expect.

Inlined dat-style strings

When we switch to yaml, we might want to keep the option to "inline" a yaml object into a dat-style string. What I mean is the following:

This yaml object

- ID: 0
  SOLID:
    HEX8: [1, 2, 3, 4, 5, 6, 7, 8]
    MAT: 1

might be entered equivalently as an inlined dat string

- ID 0 SOLID HEX8 1 2 3 4 5 6 7 8 MAT 1

The motivation here is mainly to save vertical space (and a bit of file size).

When to switch

From an implementation side, nothing is really missing. Technically, we could do it right now because we can read yaml, although we only support the "inlined dat" style mentioned above. Thus, I would consider supporting the object yaml style first so we can convert 4C input files to yaml files with a useful structure right away. Second, we might want to have the schema file describing these objects ready before we fully switch so it is a pleasurable experience for everybody to work with yaml :)

We might need to consider external scripts and grant a (short) transition period.

Related

Related to #113, which is the driving force and obvious benefit of yaml over dat in terms of usability.

This discussion is tangentially related to #60, although it does not really matter if we read our mesh from dat or yaml, since both are equally unsuited for this kind of information.

[gilrrei]

I've always been ready to move to yaml 😄 , but since this breaks the daily workflow of a lot of people:

• As you mentioned, we need scripts (or similar) to convert from dat to yaml before the move
• I think a tutorial highlighting the changes and how to convert would be really helpful (something like a user group meeting via zoom?). I think for a lot of users the benefits are not clear yet, and this would be a great way to communicate the changes and get some possible feedback
• It's also important to think about the 4C friends, such as meshpy, pytoda, lnmeshio, as this will break their packages

[amgebauer]

Adapting to yaml input is very easy for lnmmeshio. In the long run, I'm looking forward to retiring it with #60.

[gilrrei]

tagging @isteinbrecher for meshpy, @davidrudlstorfer for pytoda

[davidrudlstorfer]

Thanks for tagging me @gilrrei. In my opinion we can directly drop the .dat input mechanism. PyToDa only contains a .dat file formatter which would be necessary once we move to .yaml files. So this is absolutely no problem.

Regarding MeshPy a small note from my side - I've added a small proof of concept a while ago which converts the output to .yaml (beamme-py/beamme#136) so this change is also doable. But let's hear what @isteinbrecher has to say

[amgebauer]

Thus, I would consider supporting the object yaml style first so we can convert 4C input files to yaml files with a useful structure right away.

I think this is an important point. If possible with not too much effort, we should enable the full bandwidth of yaml input first and then do the deprecation step so that users don't get annoyed from adapting their input files/scripts.

[isteinbrecher]

I appreciate the recent effort to add the yaml support, and definitely agree that a standard file format is to be preferred when reading in simulation parameters. However, I think the main drawback of the old dat format is still present with the yaml format, i.e., that we store coordinates, topology, node sets and so on in ASCII strings. From my point of view, it all boils down to tackling #60 and providing a good scalable solution for reading in that kind of data. Before this is done, I am very hesitant to drop the dat file support.

[mayrmt]

For your personal scripts, you can, of course, wait until we make both changes and then adapt to both at the same time, although you still perform two conceptually different changes.

This is only true, if the dat-file format is not deprecated until completion of #60, right?

[isteinbrecher]

@sebproell in theory you are absolutely right, the parameters and the mesh are two different things. But since they are currently all mixed together, and more importantly, we don't know yet how the final format will look like, I will think there will be considerable overlap when refactoring all out input mechanisms.

However, this was not my main point. My main point was, that while yaml is nice, in the current state, the benefits of that format compared to the dat format are more or less cosmetic. For many use cases it will simply not be worth the effort to switch. Again, if we were to switch to a solution that allows for state of the art data formats, I am happy to switch ASAP.

[gilrrei]

My main point was, that while yaml is nice, in the current state, the benefits of that format compared to the dat format are more or less cosmetic.

I strongly disagree on this point. The advantage of using a standard format (it doesn't matter if JSON, YAML, XML, or whatever) is that they are easy to read and write, which makes building tools for them super easy. The dat format is the biggest bottleneck for creating Python tools to construct input files, highlighted by the fact that multiple projects implemented the dat logic manually, which they must maintain manually! In fact, based on the json schema that 4C currently produces, with datamodel-code-generator and not many lines (<200) of code, I wrote a small library that generates python classes that is able to create python dataclasses for input file for which the available valid options are validated against! Reading and writing are trivial using YAML, but would need a lot of manual input for .dat due to inconsistencies between sections. Note that this project does not require updates if sections change, as the schema file itself handles this. I am more than willing to share this project and collaborate with all the 4C&friends on this, as this not only affects updating existing tools but also facilitates the creation of new ones!

[sebproell]

This is only true, if the dat-file format is not deprecated until completion of #60, right?

@mayrmt We incrementally develop new features. How you stay in sync with these developments in your personal projects is up to you. That's what I mean by "waiting". You probably don't want to wait too long if you use the scripts in daily work.

[mayrmt]

@sebproell Sure, but this is not about developing new features (that I can stay in sync rather fast or slow), but about deprecating existing features -- and this is not only a personal problem.

[mayrmt]

Since a possible deprecation of the dat file format is a major break in backwards compatibility, I'll share my thoughts on the topic (in no particular order):

1. As far as I remember, the yaml format has been introduced as an additional option to facilitate some use cases for some QUEENS users. Along these lines, I want to stick to the idea of an "additional input format". When introducing yaml, there have not been any concrete plans to deprecate the dat file format any time soon. (Yes, the idea quickly came up, but it was not the original intention behind introducing the yaml format for input.). Given this history, I am quite surprised, that there seems to be a push to deprecating the dat format quickly. I might have missed it, but at least I do not remember a consensus about this deprecation step.

2. The yam format might be nice, but from a user and performance perspective it remains a purely cosmetic change. The key problem of reading the mesh (cf. Allow to read the mesh from standard mesh formats #60) is not affected by the yaml format at all. I strongly believe that we should focus our efforts on Allow to read the mesh from standard mesh formats #60 instead of on reading a few input parameters from a different formatting of ASCII input.

3. Yes, I can see the value of using a standardized format. But if we want to do so, then we should stick to the standard to 100%. But we can already see that the yaml format is not well suited for storing mesh data, as we now have a "standard" format and an additional "inlined dat string" to save space.

4. Deprecation of well established features cannot be done ad hoc. Of course, I acknowledge the constant evolution of 4C that requires to adapt to recent changes on a regular basis (e.g. renaming input parameters, ...). Some (or maybe even many) of us will run "old" input files (e.g. when trying to reproduce results for writing up a PhD thesis). While small changes can be adapted in these old files, a complete rewrite puts an extreme burden onto the individual user. There might be scripts that can help with the conversion, but the additional work still exists and can be considerable. Let's remember how much work the change to FUNCTION_OF_SPACE_TIME was and still is, and that was a minor change compared to a change of the complete input file format. As a consequence, a deprecation of such large impact requires a sufficient grace period in order to alleviate the burden on 4C users as much as possible.

5. One thing that has not been discussed so far is what will happen to the pre-processor pre_exodus. Is the pre-processor already able to create yaml files?

6. This discussion opens with the goal to

   discuss if, how and when to deprecate the dat format.

   Yet, its course does not reflect this so far, but rather appears to me to assume that deprecation of the dat file format has already been carved in stone -- which at least I do not remember. We need a real discussion with an open outcome that works for all involved parties.

[amgebauer]

but from a user and performance perspective it remains a purely cosmetic change.

I think you can only come to this conclusion if you have never tried yaml as input for 4C. If you just look at the file itself, it seems like a pure cosmetic change. But if you add the schema file to your IDE, yaml is a boost in productivity since the IDE knows all valid options of your input file.

[sebproell]

@mayrmt

1. I see nothing wrong with realizing that a new solution is so nice that it can fully replace an older solution. Although it was not the initial goal to deprecate dat, I don't see how that is relevant if we nowadays think that yaml is strictly superior to dat. See also points 4 and 6.
2. Already answered multiple times by @gilrrei and @amgebauer. Please understand that this is not an issue of beauty for the people behind this change. Also, if you desire Allow to read the mesh from standard mesh formats #60 badly, this issue is open for anybody to be worked on.
3. We need to distinguish, on the one hand, between the standard of the format itself, where yaml has one and dat does not even exist in any relevant manner, and on the other hand, the way we interpret whatever we entered in a (standardized) format. Just because we use yaml does, of course, not magically give the code the power to know what MAT: 1 is supposed to mean. We still need to code that (in fact, this is done by the recent InputSpec). Our yaml files are compliant with the yaml standard. The "inlined dat style string" does not change that. I fully agree that yaml is not the correct ultimate format for meshes but we already discussed this (separate) point above.
4. (and also 6.) I opened this Github discussion in advance to discuss if, how and when we want to deprecate dat. We already gathered input by various tool maintainers. Reagrding the "if": So far, many people seem to be happy with deprecating dat but others would like to keep it (indefinitely?). Regarding the "how": we noted some key features we would like to have and test before deprecating dat, and acknowledge that we need helper scripts. Regarding the "when": we talked about the need for grace periods but many people indicated willingness to quickly adapt. IMO, deprecation means that this format is supported but you are warned to use it. New input features might not get added to it, and eventually we will remove dat file support entirely. Also the "old file rerun" example seems contrived because you would want to run this with the commit that was originally used. Of course, if you want to port this forward to a newer 4C version, you will run into bigger problems than just the input itself (e.g. massive Trilinos solver changes or the new solid elements (don't get me wrong, I agree we need these breaking changes :)))
5. That's a good point that I already talked over with @gilrrei. Of course, pre_exodus will get an option to emit yaml format .

[eulovi]

Personally, I don't like the idea of deprecating the data file in such a short time, as it is part of every developer's workflow. I wouldn't assume that all developers are ready to make such a change right now, as I don't know their personal schedules and current tasks. I would propose for a longer period of grace.

I haven't tried yaml as input for 4C, so I don't know its benefits for daily workflow. I think it would be nice if this could be shown to other developers (maybe in the next developer meeting?).

[gilrrei]

There seems to be a lot of confusion regarding different aspects of the input and timeline. As I already mentioned (also pointed out by @eulovi), I propose to do a TGM-like presentation (I guess the presenter should be one of the driving forces) with all the new input features, as most people are not familiar with them. I think this is particularly interesting for people who actually create input files, i.e., users, as this is a first glance at how the changes could improve their daily research. Not only is this information important to decide to switch workflows (for each user individually), but it also paves the way for an informed discussion of the next steps and possible improvements.

[lauraengelhardt]

I understand that the strongest argument against deprecating dat is the effort that comes with converting old dat files to yaml. Maybe the TGM could also include a short demonstration on how the conversion of existing dat files to yaml could happen.

[sebproell]

@lauraengelhardt I think porting various input generator scripts (like meshpy) to .dat is actually more work. Conversion of manually crafted .dat files will be made trivial (but sure, we can show it in the TGM 👍).

[mayrmt]

... and there are much more than just a handful of input generators! Yes, the larger ones like meshpy or lnmmeshio are one part of the story, the other part are the many scripts distributed across many users/developers that all would need to be converted as well -- and this number cannot even be counted, since we do not know about them.

[lauraengelhardt]

Coming from someone who worked with a lot of dat files during the recent clean-up of the input mechanism: The fact that dat is not a standardized format makes such changes very time-consuming. If the files were yaml, I would have simply written a python script to replace certain keys, values, a.s.o; dat files may have extra spaces, comments, whatsoever such that simple regex searches do not always work.

[gilrrei]

Thanks to the recent work from @sebproell and @lauraengelhardt, using 4C with yaml files just became even more user-friendly. Here is a demo for some recent sections for which we are currently enabling validation (not in main yet, wip):

Note: In dept tutorial will come in the TGM/Meeting

schema_demo.mp4

[sebproell]

We have made good progress to support standard input files (notably yaml) in the last few weeks. Also, @gilrrei and I will give a small TGM on the improvements in the input in the next weeks. With this progress in mind, I would like to propose a concrete plan for the deprecation of dat format:

1. Convert existing test files to yaml (now)
2. Warn if a dat file is used, point out how easily it can be converted with Allow 4C to convert dat to yaml #382 (this is the deprecation step) (now)
3. At some point: actually remove dat (future). I am fine with delaying this until we Allow to read the mesh from standard mesh formats #60. As discussed above, mesh input is not really related but this part of the input file is the one that is most probably created by lnmmehsio, meshpy, etc. so we could allow for dat here. Also, Allow to read the mesh from standard mesh formats #60 would make pre_exodus obsolete and further simplify input-related business.

Remember that deprecation does not mean that a feature is unusable. We are simply thinking forward and expressing the intent to replace a feature with a superior alternative. I see no point in misleading people into thinking that using .dat is a good idea. Not informing users that they should use .yaml/json for their input seems irresponsible at this point. I am open to discussing the time frame for eventual removal, but deprecation should happen basically now.

[mayrmt]

@sebproell, thanks for communicating your thoughts and for a concrete proposal for a path forward. I believe, that we can find common ground to move forward in that direction.

I am looking forward to the TGM. This will be a good opportunity to clarify questions and also to discuss "workflows" beside pre_exodus, meshpy, lnmmeshio.

[sebproell]

Recap

Developer point of view:

• dat is actively preventing progress in Versioning of our input file #74, Add input file update mechanism for YAML files #514, rework of sections to a more structured way, ...

User point of view:

• dat does not provide a benefit over yaml. On the contrary, it is not user-friendly.

Thus, we think it should be deleted in the long run.

Note: dat deprecation and deletion is independent of the changes to the mesh file #60. As there is no timeline for that, we do not want to be tied to this separate (but important!) project. Work on #60 is even possible in parallel to the points we list below.

Together with @lauraengelhardt and @gilrrei, we propose the following roadmap for dat file deprecation and deletion:

Phase 1 - Deprecation (now)

1. Convert existing test files to yaml (now, trivial)
2. Warn if a dat file is used, point out how easily it can be converted with Allow 4C to convert dat to yaml #382 (this is the deprecation step) (now): We can do this since every dat file is convertible to yaml.

Phase 2 - Adaptation

1. Public and open input generators (meshpy, lnmmeshio, cubitpy) should migrate to direct yaml support. Of course, initially any dat file can always be converted to yaml (see point above). If useful, common (python) infrastructure may emerge during these steps.
2. If pre-exodus is still desired, it should be ported to read a yaml header as indicated here: Boosting 4C Input and User Experience using YAML #490 (reply in thread)
3. Purge mentions of dat from the documentation and code base. Either use neutral wording or, where useful, refer to yaml.
4. Ensure that reading the large geometry sections from yaml files is performant for large examples. Preliminary testing gives no indication it shouldn't be the case but better be sure.

Note: All old dat files will continue to work until the end of this phase.

Phase 3 - Deletion (tbd)

1. Remove dat file support. Simplify implementation of input mechanism accordingly.

Note that starting at phase 2 we can tackle some of the planned additions. From our point of view, it is important to do phase 1 now. Phase 2 and 3 do not have a strict timeline. Nevertheless, there is overhead in maintaining code that is bound to be deleted.

This is a concrete proposal for action. Feel free to comment on specific points and add anything we might have forgotten.

[mayrmt]

Thanks for proposing a plan forward! I agree with some statements, but also want to add some further points. I am happy to discuss this.

Recap (cont'd)

Further developer point of views (for completeness):

• We have agreed to bundle maintenance and service efforts in task forces. So far, progress in task forces is less than optimal. Moving development activity to a completely new field is likely to reduce the time and effort spent on existing task forces.

Further user point of views (for completeness):

• All established work flows work with dat. Changing them requires non-negligible effort for each user.

Comments on the roadmap

I am afraid, that pulling the trigger on step 1 and not having an actionable timeline for steps 2 & 3 can be harmful, since it puts us in a state, where we encourage a new standard, but don't have the means to satisfy the new standard. We have done that in the past multiple times and it always had backfired. I know, that one has to start at some point and might not want to wait until all details have been clarified and all external tools have been updated. To this end, I'd prefer to have a clearer path and strategy on how to manage the transition (including #60 which is the most crucial part in input in my opinion).

[gilrrei]

@mayrmt

We have agreed to bundle maintenance and service efforts in task forces. So far, progress in task forces is less than optimal. Moving development activity to a completely new field is likely to reduce the time and effort spent on existing task forces.

I have to say I am disappointed by this kind of message. I thought these contributions were welcome as they improve usability for every user. As mentioned in the CONTRIBUTING.md:

We welcome all types of contributions, irrespective of size and complexity.

Personally, I am not part of any task force, and this work does not hinder other task forces (or other work) from advancing. Working on different things in parallel has worked (and still does), especially since we are a cool community interested in exchanging knowledge and/or just talking about nerdy things.

I invested some time in improving the input as the input is a major bottleneck for most users (including myself). Constructing the scripts/workflows you mentioned only costs a lot of time because the input is hard to handle the input in the first place. This is actively what we are trying to change, step by step.

[lauraengelhardt]

Thank you for your feedback. I’d like to address some of the points you raised:

We have agreed to bundle maintenance and service efforts in task forces. So far, progress in task forces is less than optimal. Moving development activity to a completely new field is likely to reduce the time and effort spent on existing task forces.

While I acknowledge the importance of the task force, it’s an open-source project, and I've already dedicated a significant amount of time to the task force I’m assigned to, and I’ll continue to support that work. However, switching to YAML is something I believe has long-term advantages, both for my research and for the wider community. I’ve weighed the costs, and for me, this transition is the right investment of time given its benefits. I truly believe that once the new input mechanism is in place, it will simplify workflows in a way that will pay off in the long term.

Changing them requires non-negligible effort for each user.

I understand the concern about the effort required for others to switch to YAML. That’s exactly why we’re proposing a reasonable transition period, and we can certainly discuss how long that should be. From what we’ve seen with, e.g. lnmeshio (and I believe meshpy from @davidrudlstorfer's experience), the switch to YAML is very manageable and shouldn’t take weeks or months. We’re also providing several tools to facilitate this transition, including a YAML converter, to minimize the burden on users.

All established work flows work with dat.

I don’t think our standard should be simply having something that works; we should aim for the most efficient and sustainable solution. Keeping our input generators up to date is time-consuming and inefficient in the long term. Automating tasks is also much more challenging with DAT. I am very confident that the transition to YAML, while requiring some initial effort, will in the end pay off.

I am afraid, that pulling the trigger on step 1 and not having an actionable timeline for steps 2 & 3 can be harmful, since it puts us in a state, where we encourage a new standard, but don't have the means to satisfy the new standard.

I understand your point about having an actionable timeline. However, in order to work on input file versioning (which has been requested multiple times), we need to complete step 1. Without this, we cannot move forward with things like file versioning, which will ultimately save users time by automating the process of keeping files up to date.

To this end, I'd prefer to have a clearer path and strategy on how to manage the transition

Could you provide any specific suggestions? I feel we’ve already outlined a clear approach, but I’m open to further clarifications if needed.

(including #60 which is the most crucial part in input in my opinion).

Regarding #60, I believe it’s an important issue, but it’s, as mentioned, not dependent on the transition to YAML. It can be tackled independently. Could you elaborate on why you think it should be addressed before?

One additional point at the end: We are an open source community, and if we like others to work with our software as well, the first point where they come in contact with the software is the input. If input is difficult, I will not use the software. The sooner we implement the new standard, the earlier new people start working with this new input mechanism.

[sebproell]

puts us in a state, where we encourage a new standard, but don't have the means to satisfy the new standard.

What is this supposed to mean? The new input mechanism is tested (way better than what we had before) and additionally requires less error-prone code since we use standardized formats. Since you can continue to convert dat to yaml until phase 3, nothing can break. There must be a misunderstanding here.

We have done that in the past multiple times and it always had backfired.

For me, this kind of statement does not carry any content. If mistakes happen, we try to learn from them. To make the transition as smooth as possible we have a plan in mind which we shared and asked for feedback. As @lauraengelhardt said, if this is not enough for you, please provide specific feedback on open points.

[gilrrei]

We've created a project with repo FourCIPP (coauthored by chatgpt):

The FourCIPP repository will hold a Python parser for 4C input data, designed to facilitate the transition to YAML for all 4C Python projects. This tool serves as the backend for 4C input Python projects, providing a streamlined approach to data handling.
It utilizes the 4C_metadata.yaml file to ensure that projects remain up-to-date and consistent with the latest features. FourCIPP aims to enhance the efficiency of 4C data processing while promoting the use of YAML in project workflows.

We'll add features step by step, but the goal is, of course, to handle everything 4C input. Once we have a good working version we'll provide more information, in the meantime, everyone who's interested in this tool can participate, from ideas to coding, contributions are welcome :)

[georghammerl]

In order to discuss especially Phase 2 (details, time line) of the Recap (#263 (comment)) and clarify potential misunderstandings, here is an invitation to a Zoom meeting on April 1st, 15:30 - 16:30:

https://tum-conf.zoom-x.de/j/69826674614?pwd=T2gYCfeW0RBHH1Avsli4mcLs6JXLOo.1

Please feel free to join. Meeting notes will be provided afterwards here.

[sebproell]

Thanks for setting up the meeting. Phase 1 is already well-defined and ready to start, and from the discussion so far, it seems we all agree on the necessity for deprecation, so I’d suggest we move forward here. Phase 1 will be resolved quickly within a few PRs. This allows us to advance to Phase 2 with a clear commitment away from dat. While working on it, we can continue refining Phase 2 based on new insights before eventually moving to Phase 3.

In order to discuss especially Phase 2 (details, time line) [...] and clarify potential misunderstandings,

I marked my availability in the poll. However, I personally don’t see the need for a meeting at this stage and not without a clear goal. If there are specific, unresolved concerns that could block progress, it would be helpful if they were summarized here in advance.

[isteinbrecher]

As far as I judge this thread, we have not reached an agreement yet, if we want to deprecate the dat format. There are still some important points to answer/discuss:

• For what type of input file items is the inlined dat string possible?
• Everything that we have seen up to now, that is supposed to show the advantages of the new input format is done for small examples. Is this ready for HPC? To assess that, some comparisons are needed. How does the input read time differ for yaml/dat? How does the file size differ for yaml/dat? This has to be verified with actual engineering examples of at least 2e7 nodes (and node sets that contain all nodes).
• How do the read in times (for large examples) differ for the proposed "Phase 1", when a dat file is internally converted to yaml.
• Is the proposed new format actually a "real" yaml format? I am no yaml expert but I think the hierarchical input files (as described in the slides of the TGM on slide 23) can not be read with a standard yaml library. Do we really want to introduce our own custom solution? If we would agree to tackle Allow to read the mesh from standard mesh formats #60, this feature would not be necessary and we could actually have a standard format.
• Why not include Allow to read the mesh from standard mesh formats #60 before going further? I think that would be something everyone can get behind and we don't postpone this (which usually means we never tackle it).
• I don't the rush to deprecate the dat format. What is the reason we cant have both solutions (at least for a transition period). Versioning of input files is no real reason for me, as this is only a minor issue that in my opinion does not outweigh the change of every existing input creation workflow for everyone on such short notice.
• If at some point we actually agree to deprecate the dat format:
  • I still believe think that replacing our input standard but keeping one of the main weaknesses (ASCII mesh format) is a bad decision. If we want to improve the way we do FE input for everyone in the community, we have to also tackle Allow to read the mesh from standard mesh formats #60. I don't agree that yaml is uncoupled from mesh format, as we already start to introduce out own special yaml features (see above) just to circumvent to work on an actual state of the art solution for inputting mesh data.
  • We need a (sufficiently long) transition period to allow the users to update their workflows. "Phase 1" is a bad start in my opinion, we can not decide on deprecating a feature and actually deprecate it the next day.
  • There seem to be a lot of new ideas floating around at the moment how to further proceed with the input. If we deprecate the dat format, how do we ensure that we don't have this discussion again in the near future? For example, do we allow the "inline dat-style" indefinitely?
  • (Continuation of the previous point) I think we have to define and document the way we expect our input to look in some document.
    • This will also allow new users with FE experience to adapt their meshing workflows to 4C.
      – I want to make sure that if we decide to deprecate the dat format, we do it in a thought trough and more or less final way, so we don’t end up repeatedly changing our workflows as we continuously redefine what we want from our input format.

Let me please put a perspective on why I am not as enthusiastic as some other people with switching to yaml and deprecating the dat format at the current pace. From everything I have seen so far, in the current state yaml helps with defining parameters in the input file. I have personally never had any relevant issues with that for the dat format, sure it is nicer if I automatically get suggestions and see if I wrongly put in the time step as a string, but in the end I still have to actually run the input file with 4C to see if everything was correct as there are a lot of consistency checks that can not be realized with the yaml checks. Also, these advantages don't work with the approach taken with CubitPy and MeshPy (I will talk to @davidrudlstorfer as he is one of the few who know the yaml input and MeshPy and listen to his take). My current point of view is: "yaml has some nice features but it does not help me with my workflow. The advantages we get from using yaml in it's current state don't outweight the drawback of updating all our tools and scripts. There are other more pressing needs." I surveyed a few 4C users at IMCS (mainly MeshPy and CubitPy users) and the feedback was more or less the same. I think such a large strategic change does not solely depend on the technical facts but on many other aspects as well, and should at least be discussed in the maintainer meeting.

I am curious on the answers and hope to get a constructive discussion going.

Also, I think we have to improve our communication in discussions like these and remember that we are a community and we should consider the wishes and needs of everyone in that community. I understand that the people behind the proposed changes are very enthusiastic and want to make progress, but I would also ask for some thoughtfulness for the people who only marginally benefit from the proposed changes, but need to adapt their whole input workflow that has been developed over a long time. Changes to such fundamental parts of 4C are bound to bring up different opinions and as we have seen in the past, a (virtual) discussion in person usually also helps to better understand conflicting opinions. Thus, I want to thank @georghammerl for setting up the meeting and I am looking forward to discussing future solutions for our input and the possible deprecation of the dat format there.

Finally, regarding fourcipp, I really like the idea of having a common base for our python tools! I am not quite sure yet what the exact idea behind it is, but I will have a look and reach out to the responsible people. I think it makes sense to develop this together, so as many existing 4C workflows as possible can benefit from it.

[sebproell]

Thanks for your feedback. Here is my answer co-authored with @gilrrei . To your points (in order):

1. Only the sections that will be deprecated with the Allow to read the mesh from standard mesh formats #60. They can be found in the 4C_metadata.yaml file generated with your build under the name "legacy_string_sections"

Click here for the current ones.

• STRUCTURE ELEMENTS
• FLUID ELEMENTS
• LUBRICATION ELEMENTS
• TRANSPORT ELEMENTS
• TRANSPORT2 ELEMENTS
• ALE ELEMENTS
• THERMO ELEMENTS
• ARTERY ELEMENTS
• REDUCED D AIRWAYS ELEMENTS
• PARTICLES
• PERIODIC BOUNDINGBOX ELEMENTS
• FLUID DOMAIN
• STRUCTURE DOMAIN
• NODE COORDS
• DNODE-NODE TOPOLOGY
• DLINE-NODE TOPOLOGY
• DSURF-NODE TOPOLOGY
• DVOL-NODE TOPOLOGY
• STRUCTURE KNOTVECTORS
• FLUID KNOTVECTORS
• ALE KNOTVECTORS
• TRANSPORT KNOTVECTORS
• TRANSPORT2 KNOTVECTORS
• THERMO KNOTVECTORS

2. Such tests have been performed by me, at least for 1e6 elements. The timing of reading elements here is virtually identical. I think the point is important and we monitor the performance.

3. There seems to be a confusion here. The file is not internally converted to yaml. You may convert it 4C --to-yaml.

4. The way we use yaml is standard conforming. The examples in the slides work (except for typos?). I think what your are referring to here is the fact that a yaml reader will not magically read the included file. Note that this feature is implemented in 4C for dat and yaml. Yes, it is useful to include a mesh, but it can also be interesting for the other use cases shown in the TGM.

5. As mentioned multiple times, Allow to read the mesh from standard mesh formats #60 is independent of yaml. Waiting would artificially slow down progress on yaml features. The input people are not planning to start this, at least not in the near future. Again, parallel work is welcome and possible. Let us move discussions about the mesh to Allow to read the mesh from standard mesh formats #60.

6. "I don't the rush to deprecate the dat format. What is the reason we cant have both solutions (at least for a transition period)." The process you describe is what we outlined in the roadmap, and what is meant by deprecation (Phase 1). It is not deletion, which does not have a timeline. We will have both options during Phase 2. "Versioning of input files is no real reason for me, " It is for us. We outlined why we think this is interesting for the community Versioning of our input file #74 Add input file update mechanism for YAML files #514. "change of every existing input creation workflow for everyone on such short notice" This discussion has been open for two months. What do you mean by short notice? There is no timeline for the eventual deletion.

deprecate the dat format, we do it in a thought trough and more or less final way

and should at least be discussed in the maintainer meeting

This is not how software development works. We are developing incrementally, with a clear direction but without overcommitting to details too early. Given the complexity of the codebase, the best approach is to iterate while staying as backward-compatible as possible. In fact, #514 could improve backward compatibility even further.

We reviewed the rest of your comment and believe we have addressed all technical points.

If you personally do not find this feature relevant, that's entirely fair. However, that should not prevent the project from moving forward.

[isteinbrecher]

Thanks for your answers, I have some follow up questions:

They can be found in the 4C_metadata.yaml file generated with your build under the name "legacy_string_sections"

Will this legacy string be supported indefinitely? Who decides this?

Such tests have been performed by me, at least for 1e6 elements. The timing of reading elements here is virtually identical. I think the point is important and we monitor the performance.

I think everyone who has worked with large examples in 4C knows that the fun things start to happen at 1e7 nodes and over 100 cores. So while it is good to check the performance with 1e6 elements, I would like to see the timings and file size comparisons for actual large examples.

There seems to be a confusion here. The file is not internally converted to yaml. You may convert it 4C --to-yaml.

Then this is a significant problem with "Phase 1". That means users have to adapt their input tools to convert dat files to yaml, then call the yaml with 4C. Once the input tools can create yaml, the previous workflow has to be adapted again. The workflows have to be adapted twice, which for me is already a reason to oppose the immediate start of phase 1 in the current form (the conversion should at least be done within 4C). However, the main drawback with "Phase 1" is that we have to convert the input file and then basically store the input file twice, thus basically doubling the storage and time required for input - this can easily become a bottleneck for scaling studies. These two points already show that if we decide to deprecate the dat format, we can not start with phase 1 right away but give users time to adapt their workflows.

The way we use yaml is standard conforming. The examples in the slides work (except for typos?). I think what your are referring to here is the fact that a yaml reader will not magically read the included file. Note that this feature is implemented in 4C for dat and yaml. Yes, it is useful to include a mesh, but it can also be interesting for the other use cases shown in the TGM.

I am pretty sure, that if I create the files (not including the typos) as proposed in the slides and use a standard yaml library to read the main file, I will not get what I would expect from the presentation. So while each file might technically be "true" yaml, what we expect from the combination of them is not, at least it is not readable without our own tweaks. I think this only has limited value for pure parameter files (you want to create those files with a python script anyway) we currently need this to circumvent the fact that we are not working on #60.

As mentioned multiple times, #60 is independent of yaml. Waiting would artificially slow down progress on yaml features. The input people are not planning to start this, at least not in the near future. Again, parallel work is welcome and possible. Let us move discussions about the mesh to #60.

Without a proper way to handle large data, the current state of yaml is only half of the solution of having a state of the art input format. For people who did not have issues with the way parameters are handled within dat, this means that they have to renew their whole workflow, for limited benefit.

From what I have seen, it seems to me that currently the yaml input is in a quite good state to handle parameters. So in my opinion the next logical step would be to tackle the one remaining huge issue we have with input. Don't get me wrong, versioning of input files surely is a nice feature, but is it really more important for our daily work and research than having a nice solution for inputting mesh data? I think this would really help everyone in our community.

This discussion has been open for two months. What do you mean by short notice? There is no timeline for the eventual deletion.

Yes, the discussion if we want to deprecate dat has been open for a while, but as far as I can we have reached an agreement on if / how / and when to deprecate the dat format. As I pointed out earlier I don't think that "Phase 1" is a suitable solution right away. If we agree to deprecate the dat format, then we have to give people enough time to switch their workflows and not leave them hanging in the middle with a less than optimal intermediate solution that also requires multiple changes to their workflows.

This is not how software development works. We are developing incrementally, with a clear direction but without overcommitting to details too early. Given the complexity of the codebase, the best approach is to iterate while staying as backward-compatible as possible. In fact, #514 could improve backward compatibility even further.

I don’t agree. First of all, if such a drastic change to how we use the software isn’t at least discussed in the maintainer meeting, then what’s the purpose of having one?

Also, I don’t see a clear direction here. Yes, we can now handle parameters with yaml (and I do see how that benefits some users), but the current proposed future direction feels more like an attempt to now squeeze out every possible feature from the existing intermediate state, rather than thinking about how we can improve the input for the whole community (you can see, we are gain at #60).

If you personally do not find this feature relevant, that's entirely fair. However, that should not prevent the project from moving forward.

As I said, this is not just my opinion but in some way or another shared by many developers and users that I have talked to about this. Also, just because I don't find the proposed features super relevant does not mean I want to prevent the project from moving forward. We have to actually discuss this topic and have to try to have the whole community on board.

We reviewed the rest of your comment and believe we have addressed all technical points.

I would still like to hear your answer to my previous question, how we avoid this discussion again in the future. How can we decide on a more or less fixed way we do input? Also the new input format should be documented so new users that want to try out 4C know what to expect.

I truly appreciate you answers and hope we can keep this discussion going so we can at some point reach a decision on the proposed topic.

[sebproell]

I think everyone who has worked with large examples in 4C knows that the fun things start to happen at 1e7 nodes and over 100 cores. So while it is good to check the performance with 1e6 elements, I would like to see the timings and file size comparisons for actual large examples.

Fair enough. I added this point to phase 2.

[gilrrei]

@isteinbrecher Thanks for the follow-up questions!

Legacy Section

Will this legacy string be supported indefinitely? Who decides this?

These sections are termed legacy string sections because they will eventually be treated differently. Here are some technical details:

• * ELEMENTS, PARTICLES, NODE COORDS, *-NODE TOPOLOGY, * KNOTVECTORS will become obsolete once Allow to read the mesh from standard mesh formats #60 is completed. This is a significant task currently without a roadmap, assignees, or promised resources. This is why we don't want to wait in the first place but encourage parallel work. If you have questions or ideas regarding Allow to read the mesh from standard mesh formats #60, please move the discussion there.
• The * DOMAIN sections and their functionality could be refactored and removed from the legacy sections; see issue Switch DOMAIN sections to InputSpec #571.

Who decides this?

The 4C community! 4C[2] would be a good name :D

Large Simulation Problems

While it is good to check performance with 1e6 elements, I would like to see timings and file size comparisons for actual large examples.

As mentioned by @sebproell this will be added to phase 2. It seems that you have some experience in this, we encourage you to try it out now (as always in coding, problems emerge and are fixed see #572) and bring your expertise.

Adapting Workflows Twice

Then this is a significant problem with "Phase 1". That means users have to adapt their input tools to convert dat files to yaml, then call the yaml with 4C.

• Right now: You can use both yaml and dat natively with 4C (no conversion needed), although dat is discouraged. So, no need to adapt any input files!
• 4C can already convert dat to yaml, as shown in the presentation. @sebproell provided this feature to ease the transition and encourage users to try out yaml.
• The dat format has changed over the years, requiring input generators to adapt each time. Cumulatively, these 'small' changes have required significant effort from individual community members. For example, Meshpy, lnmmeshio, and QUEENS have approximately 3000 lines of code to interpret dat, and none can currently interpret every section. This is why we introduced FourCIPP, where you raise some good points! We are providing infrastructure, presentations, and information to help users and developers cope with these changes.
• lnmmeshio is already YAML capable, QUEENS uses some yaml files, and meshpy is in progress. If other users need help, I’m happy to assist with Python-related queries!

INCLUDES

I am pretty sure, that if I create the files (not including the typos) as proposed in the slides and use a standard yaml library to read the main file, I will not get what I would expect from the presentation. So while each file might technically be "true" yaml, what we expect from the combination of them is not, at least it is not readable without our own tweaks. I think this only has limited value for pure parameter files (you want to create those files with a python script anyway) we currently need this to circumvent the fact that we are not working on #60.

Some points to consider:

• Every file of that slide is valid yaml and can be read natively with a yaml reader.
• 'What I expect' is subjective; we did not claim that a yaml reader would glue the files together. This is a feature of 4C, which also works for dat.
• If you don’t see the advantages of this feature, that’s okay, others do!
• The interpretation (i.e., gluing the info into a single object) will also be provided in FourCIPP.
• This feature is unrelated to yaml deprecation; it already exists, is tested, and works!

Large Data

Without a proper way to handle large data, the current state of yaml is only half of the solution of,
having a state of the art input format. For people who did not have issues with the way parameters are handled within dat, this means that they have to renew their whole workflow, for limited benefit.

• We’re glad to hear you agree that yaml is part of the solution for better input :)
• As outlined multiple times, we provide infrastructure, knowledge, and time to help those who may have concerns w.r.t. yaml.
• We encourage every datvocate to try out yaml and provide feedback.

From what I have seen, it seems to me that currently the yaml input is in a quite good state to handle parameters.

Thanks :)

So in my opinion the next logical step would be to tackle the one remaining huge issue we have with input.

The proposed depreaction isn't blocking these changes, we are also looking forward to an external mesh.

Don't get me wrong, versioning of input files surely is a nice feature, but is it really more important for our daily work and research than having a nice solution for inputting mesh data? I think this would really help everyone in our community.

• Yes, versioning of input files is nice :)
• We never claimed that is more important, we claim that it is a benefit
• This is an open source project, everybody is welcome to work on Allow to read the mesh from standard mesh formats #60

We appreciate your feedback and recognize the importance of #60. However, the deprecation of dat can proceed in parallel, and we’re excited about the positive impact it will have on the community. So, why not take a chance on yaml? It might just be the upgrade you didn’t know you needed!

[lauraengelhardt]

To determine which points are open for discussion in the meeting, we (@gilrrei, @sebproell) have summarized the key points we believe we all agree on. If anyone disagrees with any of the points listed below, please provide your comments and reasoning.

• 4C input in its current form is bad.
• We need a better input (see discussions in Herrsching).
• The new input should follow a standard format.
• In the long term, dat input will be phased out.
• The work done to improve the input is relevant.
• The level of relevance depends on individual priorities and projects.
• We ensure a smooth and well-communicated transition to an improved input, considering all related side projects by providing the necessary tools and support.
• The YAML format does not offer any advantages for Allow to read the mesh from standard mesh formats #60. Allow to read the mesh from standard mesh formats #60 is important - yes - but an independent topic.
• Incremental changes are preferable to a single large pull request.
• The timeline for transitioning to an improved input format should remain flexible (depending on progress), rather than being tied to a fixed date.

If you disagree with any of the points mentioned above, please provide your reasons below.

[georghammerl]

After discussion with @isteinbrecher and @mayrmt, I would like to suggest a modified roadmap

Phase 0 - Assess the technical capabilities of yaml

1. Ensure that reading the large geometry sections from yaml files is as performant as reading dat-files for large examples (approx 5e7 DOF, 2e7 nodes, 2e7 elements).
2. FourCIPP seems very helpful in a functional and sufficiently performant state (max performance overhead of 0.1s).

Phase 1 - yaml is the future, dat will be phased out; Both yaml & dat are fully operational (no forced conversion to yaml)

1. Convert some of the existing test cases to yaml while keeping some in dat format in order to test both input paths of 4C and thus avoid untested code.
2. Warn if a dat file is used, point out how easily it can be converted with Allow 4C to convert dat to yaml #382 (=deprecation)
3. Public and open input generators (meshpy, lnmmeshio, cubitpy) should migrate to direct yaml support. If useful, common (python) infrastructure may emerge during these steps.
4. If pre-exodus is still desired, it should be ported to read a yaml header as indicated here: Boosting 4C Input and User Experience using YAML #490 (reply in thread)
5. Purge mentions of dat from the documentation and code base. Either use neutral wording or, where useful, refer to yaml.
6. Make sure the documentation shows a way for users to create a working input file
7. Convert remaining test cases to yaml

Note: Steps 2...6 can be worked on in parallel.

Phase 2 - Deletion of dat support (tbd)

1. Simplify implementation of input mechanism accordingly.

I am looking forward whether this roadmap serves as a compromise achieving the same outcome as suggested in #263 (comment) .

So for now, the agenda for our meeting on April 1st is to discuss the content/order and a possible time frame of this proposed roadmap. Thoughts on how to efficiently update a Python based workflow are welcome and may be exchanged.

[sebproell]

I am looking forward whether this roadmap serves as a compromise achieving the same outcome as suggested in #263 (comment) .

Thanks for refining the roadmap with your concrete and actionable points! It closely follows our original proposal and I am happy to see that we aligned on the goal to deprecate dat in favor of yaml. The key changes seem to be:

• Introducing a performance assessment phase (Phase 0).
• Keeping some .dat test cases longer to ensure continued validation during the migration.
• Mentioning documentation more centrally.

These additions are reasonable and we are happy to incorporate them (plus we of course appreciate the zero-based indexing ;)). I am not sure if Phase 0 is meant to be fully completed before Phase 1, as the points in Phase 0 seem to work perfectly in parallel with the points in Phase 1. I would say these two phases can be combined. Just to clarify, there’s no open technical question about whether internal yaml strings will be as fast as dat strings -- they are. Of course, we should evaluate performance in the input mechanism because we might mess up some detail of the implementation. Also, FourCIPP will evolve based on the needs of our friend projects, right @gilrrei?

Let's see if we can come up with a time frame in the meeting 👍 That said, I think we’re ready to start even without a fixed timeline.

[mayrmt]

@georghammerl @sebproell Great, sounds like we are close to a common understanding!

@isteinbrecher and I have revisited the proposed plan and suggest some final adjustments:

---

Common understanding: we aim to use yaml as the only input format, dat will be phased at some point in the future (if Phase 0 is completed successfully).

Phase 0 - Assess the technical capabilities of yaml

1. Ensure that reading the large geometry sections from yaml files is as performant as reading dat-files for large examples (approx 5e7 DOF, 2e7 nodes, 2e7 elements).
2. FourCIPP seems very helpful in a functional and sufficiently performant state (max performance overhead of 0.1s --> completed within refactor: change yaml reader fourcipp#11). If useful, common (python) infrastructure may emerge during these steps.
3. Clarify and outline necessary steps to add/change parameters (input parameters, sections, definition of nodes/elements, ...): What has to be done in 4C? What has to be done in FourCIPP? What has to be done in input generators (MeshPy, lnmmeshio, CubitPy)? A TGM would be a great opportunity to educate everybody on the features and use cases of FourCIPP and its interplay with 4C and its various input generators.

Phase 1 - Both yaml & dat are fully operational (no forced conversion to yaml)

1. Convert some of the existing test cases to yaml while keeping some in dat format in order to test both input paths of 4C and thus avoid untested code.
2. Warn if a dat file is used, point out how easily it can be converted with Allow 4C to convert dat to yaml #382 (=deprecation)
3. Migrate public and open input generators (meshpy, lnmmeshio, cubitpy) to directly produce yaml input files. Goal: everybody needs to be able to create native yaml files from scratch.
4. If pre-exodus is still desired, it should be ported to read a yaml header as indicated here: Boosting 4C Input and User Experience using YAML #490 (reply in thread)
5. Purge mentions of dat from the documentation. Either use neutral wording (e.g. "input file") or, where useful, refer to yaml directly.
6. Specify the required input format in the documentation, so that users know what 4C expects (and do not have to reconstruct it from the source code).
7. Document necessary steps to create a yaml input file from scratch.

Phase 2 - Deletion of dat support

1. Remove capability to read dat files directly into 4C to start a simulation.
2. Convert remaining test cases from dat to yaml
3. Purge mentions of dat from the code base.
4. Simplify implementation of input mechanism accordingly.
5. Question: should we keep the dat-to-yaml converter to also be able to process legacy files and their migration to new input mechanisms in the future? For how long?

---

We are looking forward to tomorrow's meeting!

[georghammerl]

Minutes "Yaml input - dat deprecation meeting on April 1st - 15:30-16:30"

participants: @gilrrei @sebproell @mayrmt @ischeider @isteinbrecher @lauraengelhardt @ppraegla @davidrudlstorfer @eulovi @georghammerl
meeting notes: @ischeider @georghammerl

These notes refer to #263 (reply in thread) regarding the numbering.

Phase 0:

0.1: Discussion whether the requested performance tests can be done while already working on Phase 1 or if this is a prerequisit before launching Phase 1. @ppraegla mentions a particle input file with 1.7e6 particles , which has 4% file size overhead, but no significant reading speed reduction. For further testing, @mayrmt will provide timely a large input file to @ppraegla which will run tests and report timings. If timing deficiencies show up, @sebproell offers to work on those. (Timing analysis has already been done, see here)

0.2: No objections

0.3: FourCIPP may be used by input generators (MeshPy, lnmmeshio, CubitPy) to support yaml format, however this is optional and may also change in future. TGM is welcome.

Phase 1:

1.1: No objections

1.2 No objections

1.3: Three main input generators are named: MeshPy, lnmmeshio, CubitPy. lnmmeshio already has full support for yaml. The other two will be refactored to output yaml until 30.6.2025;. @davidrudlstorfer has already contacted the relevant persons directly. Other (possibly one-code-one-user scripts) have to be maintained by those users (@gilrrei offers help, especially for python based scripts).

1.4: The further use of pre-exodus is to be discussed. At least, one benefit of pre_exodus is that it has a binary input reader, which might even be helpful within 4C (or serve as a template for other binary inputs).

1.5 No objections

1.6 Identified as a very important aspect.

1.7 Other input generators (besides pre_exodus) should be mentioned in the documentation! Different tutorials may use different input generators. Ideally, a comparison of the input generators with their strenghts is provided to offer new users to optimally choose the input generator.

1.8 (new): The tutorials, which are as of now written as a sequence of cubit -> pre_exodus -> 4C should definitely be polished if pre_exodus will be kept. If pre_exodus will be dropped, the tutorials need to be migrated to describe the input path using a different input generator.

Final statement to Phase 1: All participants agree on the plan to complete Phase 1 until 30.06.2025.

Phase 2:

After completion of Phase1 and after 30.6.2025

2.1 – 2.4 No objections

2.5: After dropping the dat file support and having completed steps 2.1 to 2.3, the --to-yaml converter cannot work anymore. However, thanks to github, we can save the latest stage of dat file reader to have a working --to-yaml converter. @sebproell outlines an option to create a docker container with the last version which offers the conversion functionality together with a small script to enable single command line conversion. Starting from that point, ideally an update mechanism is available to provide seemless updates for the future (@georghammerl reports of exactly such a (very successful) approach in a different software project).

Final note

@georghammerl thanks the participants for the productive meeting. For future intensively discussed topics, a meeting may be scheduled slightly earlier to converge even faster and to a reach a common understanding.

[gilrrei]

Minor correction: Phase 2.5 the 'latest stage converter' docker idea, which I like a lot, comes from @sebproell :)

[amgebauer]

1.3: Three main input generators are named: MeshPy, lnmmeshio, CubitPy. These will be refactored to output yaml until 30.6.2025;. @davidrudlstorfer has already contacted the relevant persons directly. Other (possibly one-code-one-user scripts) have to be maintained by those users (@gilrrei offers help, especially for python based scripts).

lnmmeshio already has full support for yaml.

[ppraegla]

Here are the timings for

Ensure that reading the large geometry sections from yaml files is as performant as reading dat-files for large examples (approx 5e7 DOF, 2e7 nodes, 2e7 elements).

Thanks, @mayrmt, for providing the input file. After spending some time applying all input changes that happened within one year to the input file, I got the input file running with the latest 4C version (commit sha: 1626455bc7302f6125d87e112000d9053b5e04e5). The details of the input file are

• 1.6e7 elements
• 1.6e7 nodes

Comparison dat to yaml

• File size
  • dat: 3.4 GB
  • yaml: 3.4 GB
• time for the conversion: 53.917s

There is no noticeable difference in file size because the dat file had some extra whitespace due to formatting, e.g. some double spaces instead of one.

Reading time

• 1 node, with total of 64 processes
  • dat: 2.911e+02
  • yaml: 3.236e+02
• 2 nodes, total of 128 processes
  • dat: 2.966e+02
  • yaml: 3.224e+02

The runtime overhead for yaml is between 9 and 11 %.

Note that the simulation crashes some time after reading the input with the error

The computation of the euler-bernoulli beam nullspace in three dimensions requires five nullspace vectors per node, however the current node carries 6 vectors.

[gilrrei]

Thanks @ppraegla for doing the tesing.

[isteinbrecher]

Thanks for looking into this so quickly! I am fine with the 10% increase in runtime.

[sebproell]

Dat is bound to be deprecated. I created a milestone to track the progress: https://github.com/4C-multiphysics/4C/milestone/4

I will transfer the roadmap into a large issue for this milestone (see #598).