Release of Orocos Toolchain 2.10

[meyerj]

Follow-up on remaining issues and open questions for the release of Orocos Toolchain 2.9 2.10:

RTT

All GitHub issues and pull requests targeted for version 2.9
All GitHub issues and pull requests targeted for version 2.10

• Revert "Master introduce v1cmd syntax" (Master introduce v1cmd syntax rtt#84 (comment))
• Nested function calls on 2.9 (Nested function calls on 2.9 rtt#150)
• Inconsistent RTT_COMPONENT_PATH overlay semantics (Inconsistent RTT_COMPONENT_PATH overlay semantics rtt#161)
• thread safety may be broken for OwnThread operations if sequential activity used (thread safety may be broken for OwnThread operations if sequential activity used rtt#170) and Prevent changing the activity from a component's own thread (Prevent changing the activity from a component's own thread rtt#171)
• SlaveActivity: inconsistencies and potential segfaults due to dangling pointers (SlaveActivity: inconsistencies and potential segfaults due to dangling pointers rtt#178)
• Allow loading of global service plugins via the CORBA interface of each TaskContextProxy (Allow loading of global service plugins via the CORBA interface of each TaskContextProxy rtt#228)
• Fix multi-signal emission in case of fused functor data source callbacks (Fix multi-signal emission in case of fused functor data source callbacks rtt#293)
• Lost messages when a port hast multiple connections (one remote connection)(Lost messages when a port hast multiple connections (one remote connection) rtt#248)
• review and test Xenomai 3 support (add xenomai 3 support with xenomai 2 compatibility rtt#301)
• Massive deadlocking when disconnecting connections in parallel (Massive deadlocking when disconnecting connections in parallel rtt#300)
• FileDescriptorActivity locking and robustness (FileDescriptorActivity locking and robustness rtt#309)
• sample given to setDataSample seem to sometimes "leak" to the other side (sample given to setDataSample seem to sometimes "leak" to the other side rtt#319)

OCL

All GitHub issues and pull requests targeted for version 2.9
All GitHub issues and pull requests targeted for version 2.10

No open issues and pull requests that are not already merged into the toolchain-2.9 release branch.

Log4cpp

All GitHub issues and pull requests targeted for version 2.9
All GitHub issues and pull requests targeted for version 2.9

No open issues and pull requests that are not already merged into the toolchain-2.9 release branch.

Orogen and dependencies

• Decide whether to include it in the toolchain package or to decouple from Orocos core packages (RTT and OCL)?
  • Their release cycles seem to be coupled to Rock and have not been coordinated with RTT and OCL in recent years.
  • If included in Orocos Toolchain 2.9, which version?

orocos_toolchain meta package and installation

All GitHub issues and pull requests targeted for version 2.9

• New documentation, installation and setup scripts (New documentation, installation and setup scripts (work in progress) #13, replaced by Add/update setup scripts setup.sh and env.sh #33 and Add top-level CMakeLists.txt, configure script and Makefile #34)

Documentation

• (Preliminary) release notes have been merged in New Toolchain releasenotes #12.
• Amend release notes with changes in Orogen and dependencies.
• Update manuals in https://github.com/orocos-toolchain/rtt/tree/toolchain-2.9/doc, first of all the Component Builder's Manual (view online).
• Migrate documentation from http://www.orocos.org to https://orocos-toolchain.github.io/ and create an alias, so that http://www.orocos.org points to the new Github page. This is not directly coupled to the release.

History

• A first attempt to push forward a release has been made in February 2016, with limited to no response from the community.

[meyerj]

From orocos-toolchain/rtt#91 (comment):

Could you guys give some status on this ? It's been 2 years that this has been introduced, and it seems that it has been merged in toolchain-2.9 (without passing through master ???). What's up ?

Unfortunately there was no time yet to work on the remaining issues (e.g. #150) and to update the documentation like the component builder's manual.

I agree that ideally all new features would have been merged into master first and the toolchain-2.9 would have branched off from there. But some of the new features depend on another and could not have been merged independently. Or at least not without extensive merge conflict resolution. Furthermore some people apparently are using master as their stable branch and there are some caveats and breaking behavioral changes introduced in 2.9, as outlined in the release notes.

Today toolchain-2.9 contains all patches from master and from my point of view there is no reason to not merge it into master and hence indirectly close all open pull requests that have already been merged into toolchain-2.9.

[doudou]

Furthermore some people apparently are using master as their stable branch and there are some caveats and breaking behavioral changes introduced in 2.9, as outlined in the release notes.

Merging everything at once only make the situation worse.... Right now, there are so many changes in 2.9 that I'm going to have to plan for a week or two of integration tests. Which I don't have. Ergo, I'll probably go for branching off and praying that I'll get the time before we're too far off to stop even considering it.

The whole point of merging the PRs in master - and having master as the main developer branch - is that one doesn't get this integration cliff.

This is the same situation you guys have with orogen. You've been working for so long using branches of the orogen/typelib related stuff that you just don't want to update anymore.

[meyerj]

You are absolutely right. If we would have merged the new features into master directly as soon as they were considered finished, knowing that they have the potential to break existing applications, and there were no comments or concerns raised by the community, we would not have that problem now. Some of them actually did introduce bugs and unforeseen problems that have been fixed later, but probably they would have been found earlier in master (see orocos-toolchain/rtt#91 as an example).

In fact, we are already working on forked versions since a few years and also orogen did not accept patches for compatibility with ROS build systems (orocos-toolchain/orogen#23) or introduced new dependencies that could not be satisfied by Ubuntu package manager and basically blocked releases into ROS. See also related discussion at http://www.orocos.org/forum/rtt/rtt-dev/rocks-masternextstable-and-orocos-toolchains-masterstable. No offence, but there is simply nobody who has the time and oversight to review all suggested changes, to foresee potential issues for other parts of the community and to keep all the documentation up-to-date. That's why I would suggest to split off orogen and other Ruby dependencies from what is considered the "Orocos Toolchain", or at least to not release them anymore as a bundle.

Nevertheless, I would suggest to merge all pending patches into master now, maybe even upgrade the major version to 3, and to backport all bugfixes and other non-critical patches to either toolchain-2.8 or the stable branch. Everyone who is not willing to spent time for integration tests (yet) can switch to this branch.

[doudou]

That's why I would suggest to split off orogen and other Ruby dependencies from what is considered the "Orocos Toolchain", or at least to not release them anymore as a bundle.

This is obviously your choice, it has zero impact on my side. Furthermore, since the autoproj stuff is not supported by the "real" orocos toolchain developers, I would suggest deprecating it completely - I'd merge the orocos/autoproj packaging in the rock.core configuration and be done with it (which would simplify the latter quite a bit). The underlying question is: do you actually use typegen at all still ? If yes, what would be the impact of doing so ? If you do, if I were you, I would just try to get my application running on orogen/typelib/... master and start integrating the changes that accumulated on your fork PR by PR to realign.

I do miss updates on PRs from time to time ... (which seems to be what happened with the orogen issue you just pointed out). Pinging helps. I try to ping every once in a while as well.

As for the "orogen did not accept patches for compatibility with rosbuild". If that was so much of a problem, you could have raised a red flag instead of simply accepting it. I'm not a complete asshole (at least, I try not to), if you said that it was such a pain, I believe I would have reversed this policy (for what it's worth, we maintain 3rd party build changes with patches instead of forks for this very reason). Part of the reason I ripped nokogiri out of the orogen dependencies was to make things easier for everyone, not just Rock (which could handle nokogiri just fine).

A.k.a. "communication". I try to put all changes to orogen/typelib/... on pull requests for years now. Whose purpose was to allow people to ask questions before things land in master. So far, I merged probably all of them a month or more later with zero feedback.

Last point: it's good to care about not breaking anything from potential users, but given how little development effort we all have, I believe that there's a middle ground to find there as well.

[doudou]

Let's make something clear, since I haven't in the previous message: I don't intend to blame anyone (or, more accurately, I'm blaming everyone including myself) for the lack of cooperation between (mainly) the Intermodalics crowd and the Rock crowd.

The real question, that I believe I have been asking a few times already, is whether we want to try to fix this, or simply split, only cooperating over RTT's development.

I would personally prefer cooperation, but it has to be cooperation. Which means, that we all have to spend more time curating and handling PRs and issues on github, and communicating about developments. As an example - and I'm singling this one out because it has been painful for me, NOT because I want to blame you guys for anything - the Orocos CMake macros have been dumped on orogen in your fork without any kind of oversight on our side, and with zero knowledge of what these macros do. I do believe the orogen CMake code is absolutely horrible, but as a result, I always pushed back looking over the macros as I believe that (1) they do a lot of things behind the scenes and (2) it will be extremely time consuming.

If we do decide on cooperation, we all have to actively make our development branches converge. It will be painful at the beginning, with the need to establish how things will work, to move forward.

[meyerj]

I personally was not involved in the project before 2012/2013 nor am I the sole maintainer, and hence cannot make that decision alone. But in my opinion the best path forward would be to say goodbye to the "toolchain" concept (and this repository), leave maintenance of orogen, typelib and their dependencies completely to the Rock community and no longer support it for ROS or even consider it as part of Orocos. On the other hand, as pointed out by @doudou above, the cooperation and common understanding of each other's needs for the RTT and OCL development must and will be strengthened again, with the first goal to finally come up with a new 2.10 (or whatever) release that works for everyone, with up-to-date documentation and a fresh, but minimal web page. KDL is actively used and maintained, too. Not sure for other former Orocos projects that have been inactive for several years now, like rFSM and BFL...

@psoetens As the original maintainer and one of the most active developers back then, do you have an opinion on the future of "the Orocos toolchain" as it has been until at least the 2.6 release? Should we organize a video meeting in the coming weeks with whoever is interested to discuss the topic and finally make some decision?