Indentation reflecting graph structure in Python scenes

[leobois67]

Writing a scene in xml for the first time I noted how pleasant it was to have the structure of the graph reflected in the indentation of the code, compared to the flat implementation of the scenes I had in Python which makes understanding the graph and locating the different components a bit of a pain.

And so I looked for a way to have this indentation for the scenes in Python. After some thought I decided to use function definition as the means to indent new nodes, and to rely on decorators to make it work:

def createScene(root):
    set_current_node(root)  # initialize the global variable that tracks the current node

    # just a shortcut for root.gravity = xxx, root.time = xxx, root.dt = xxx...
    set_data(gravity=(0, -10, 0), time=0, dt=1e-3, bbox=[[-1, -3, -1], [4, 3, 1]])

    @sofa_node  # creates a child node called "plugins" in the current node
    def plugins():
        # the objects are added in the current node, here "root/plugins"
        add_object("RequiredPlugin", pluginName="Sofa.Component.Visual")
        add_object("RequiredPlugin", pluginName="Sofa.Component.ODESolver.Forward")
        add_object("RequiredPlugin", pluginName="Sofa.Component.StateContainer")
        add_object("RequiredPlugin", pluginName="Sofa.Component.Mass")
        add_object("RequiredPlugin", pluginName="Sofa.Component.MechanicalLoad")
        add_object("RequiredPlugin", pluginName="Sofa.Component.SolidMechanics.Spring")
        add_object("RequiredPlugin", pluginName="Sofa.Component.Mapping.NonLinear")

    add_object("DefaultAnimationLoop")
    add_object('VisualStyle', displayFlags='showBehavior showBehaviorModels showForceFields showMappings')

    @sofa_node  # creates a node "root/point"
    def point():
        add_object("MechanicalObject", template="Vec3d", position=(0, 2, 0))

    @sofa_node  # creates a node "root/rigid"
    def rigid():
        add_object("MechanicalObject", template="Rigid3d", position="1.5 0 0 0 0 0 1")
        add_object("UniformMass", template="Rigid3d", totalMass=1.0)

        @sofa_node  # creates a node "root/rigid/mapped"
        def mapped():
            add_object("MechanicalObject", template="Vec3d", position=(0, 0.5, 0))
            add_object("RigidMapping")

        add_object("SpringForceField", object1="@../point", object2="@mapped", indices1=0, indices2=0, length=2, stiffness=100)
        # @sofa_node turns the function into a node that can be used instead of the "@...":
        add_object("SpringForceField", object1=point.linkpath, object2=mapped.linkpath, indices1=0, indices2=0, length=2, stiffness=100)

    @sofa_node
    def solver():
        add_object("EulerExplicitSolver")
        add_child(rigid)  # nodes can also be used like this

This works thanks to the following code:

# Global variable with its getter and setter
_current_node: Sofa.Core.Node

def get_current_node():
    try:
        return _current_node
    except NameError as error:
        raise RuntimeError(
            "You must call 'set_current_node(root)' at the beginning of createScene()"
        ) from error

def set_current_node(node):
    global _current_node
    _current_node = node

# Editing the current node
def add_object(*args, **kwargs):
    return get_current_node().addObject(*args, **kwargs)

def add_child(*args, **kwargs):
    return get_current_node().addChild(*args, **kwargs)

def set_data(**kwargs):
    node = get_current_node()
    for data, value in kwargs.items():
        setattr(node, data, value)

# Instantiating a new node
def sofa_node(node_fn, name=None):
    name = node_fn.__name__ if name is None else name
    parent = get_current_node()
    child = add_child(name)
    set_current_node(child)
    node_fn()
    set_current_node(parent)
    return child

The name argument to sofa_node can be used to specify the name of the node with a string. This allows to have nodes whith names depending on a variable, for instance:

def my_node_fn():
    add_object(...)
    add_object(...)

# different nodes, with different names, from the same function:
child_nodes = [sofa_node(my_node_fn, name=f"node_{i}") for i in range(10)]

Some thoughts on my implementation:

• Usually a function definition does not do anything until the function is called. Here, not only is the function called instantly by the decorator, but it cannot be called later on since the decorator returns a Sofa.Core.Node instead of a function. Users would not be expecting this behavior.
• It relies on a global variable to track the current node, which may not be the most elegant way of doing that.

What do you think?

[damienmarchal]

Hello Leo,

Thank for sharing this.

I fully share the objective and support creative hacking of the python syntax. And you proposal is very interesting.
I agree with your "thoughts" about the implementation and this is in line with the experiments I, and other, did in the past.

There is a lot to say, if these kind of subject interest you I suggest to join the prefab work group animated by @bakpaul and @EulalieCoevoet, meeting are on their way before and during SofaWeek.

[damienmarchal]

In my past experiment focusing on the indendation aspect I was using the python feature called a Context Manager to make the code looking like the following:

@Sofa.Scene                                          
def createScene(root):
    with Node("Settings"):
        Object("VisualStyle", name="flags")

        with Node("Plugins"):
            Object("RequiredPlugin", name="Sofa.Component.Visual")
            Object("RequiredPlugin", name="Sofa.Component.ODESolver.Forward")
            Object("RequiredPlugin", name="Sofa.Component.StateContainer")
            Object("RequiredPlugin", name="Sofa.Component.Mass")
            Object("RequiredPlugin", name="Sofa.Component.MechanicalLoad")
            Object("RequiredPlugin", name="Sofa.Component.SolidMechanics.Spring")
            Object("RequiredPlugin", name="Sofa.Component.Mapping.NonLinear")

    with Node("Modelling"):
        with Node("Object1"):
            Object("MechanicalObject", name="test", position=1000, template="Rigid3")

            with Node("Mapping"):
                Object("OglModel", name="visual")
                Object("RigidMapping", name="visual")

Contexts manager have the advantage of:

• functional strong guarantees with enter/exit clause, in a similar way as RAII in C++ it allows to implement behavior that are exception's friendly
• indentation using a dedicated python statement: the ``ẁith XXX as YYY```.
• explicit semantic not relying on python's decorator magic/high order programming

Compared to your proposal I also use a stack so we can push/pop and have internal consistency and proper backtrace in case of exception.

Experimenting with context manager was very simple as it just need to add enter/exit methods in Binding_Node.cpp.

    p.def("__enter__", [](py::object self)
    {
        std::cout << "ENTER CTX MANAGER c++" << std::endl;
        //self.attr("onCtxEnter")(self);
        return self;
    });

    p.def("__exit__",
           [](py::object self, py::object type, py::object value, py::object traceback)
    {
        std::cout << "EXIT CTX MANAGER c++" << std::endl;
        //self.attr("onCtxExit")(self, type, value, traceback);
    });

In the following c++ code, un-commenting the self.attr("onCtxEnter") permit to forward to python method "onCtxEnter"/"onCtxExit" (as I do in some of the following example of use)

Even without the forwarding to python code, the c++ permits to write well indented code with the existing API and without any hacks/python magic:

def createSceneV2(root):
    with root.addChild("Settings") as settings:        
        settings.addObject("VisualStyle", name="flags")
        
        with settings.addChild("Plugins") as plugins:        
            plugins.addObject("RequiredPlugin", name="XXX")
            plugins.addObject("RequiredPlugin", name="YYY")

The drawback of the previous lines is that, despite it is well indented, it is also verbose. I think this is to
circumvente this verbosity issue that we want to solve by introducing an "hidden" global variable with the current/last object/node
and with helper functions (Object and Node in the following example) to hide its management.

Here is the helper functions I used for my first example... they are pretty straighforward except the MonkeyPatching to inject onCtxEnxter/onCtxExit:

class NodeStack(object):
    def __init__(self, *args, **kwargs):
        self.nodes = []
    
    def push(self, node):
        self.nodes.append(node)

    def pop(self):
        self.nodes = self.nodes[:-1]

    def top(self):
        return self.nodes[-1]

__nodestack__ = NodeStack()
def Node(name, *args, **kwargs):
    global __nodestack__ 
    node = __nodestack__.top().addChild(name, *args, **kwargs)

    def enter(self):
        __nodestack__.push(self)
    
    def exit(self, exc_type, exc_val, exc_tb):
        __nodestack__.pop()

    node.onCtxEnter = enter
    node.onCtxExit = exit     
    return node    

def Object(type, *args, **kwargs):
    global __nodestack__ 
    node =  __nodestack__.top().addObject(type, *args, **kwargs)

Up to now, I haven't concluded whether or not the use of a global variable to hide the "self/this" the current node was a clever idea or mis-design that would cause us lot of troubles. But, to avoid it, I didn't find better idea that having explicit "parenting" by propagating the "context" node as in the following example:

def createSceneV1(root):
    with Node("Settings", parent=root) as settings:        
        Object("VisualStyle", name="flags", parent= settings)
        
        with Node("Plugins", parent=setting) as plugins:        
            Object("RequiredPlugin", name="XXX", parent=plugins)
            Object("RequiredPlugin", name="YYY", parent=plugins)

I remember I also experimented if it was possible to as the "Settings" registering the "Settings" variable in the
python scope/context, but found that very hacky and its look fragile.

So ... maybe with more people interested on these issues, it is time to give all that a fresh look.

[leobois67]

Thank you for your answers! After we talked about it on Friday, I also made a context manager version that looks like yours:

def createScene(root):
    set_current_node(root)
    set_data(gravity=(0, -10, 0), time=0, dt=1e-3, bbox=[[-1, -3, -1], [4, 3, 1]])

    with SofaNode("plugins"):
        add_object("RequiredPlugin", pluginName="Sofa.Component.Visual")
        # [...]

    add_object("DefaultAnimationLoop")
    add_object('VisualStyle', displayFlags='showBehavior showBehaviorModels showForceFields showMappings')

    with SofaNode("point") as point:
        add_object("MechanicalObject", template="Vec3d", position=(0, 2, 0))

    with SofaNode("rigid") as rigid:
        add_object("MechanicalObject", template="Rigid3d", position="1.5 0 0 0 0 0 1")
        add_object("UniformMass", template="Rigid3d", totalMass=1.0)

        with SofaNode("mapped") as mapped:
            add_object("MechanicalObject", template="Vec3d", position=(0, 0.5, 0))
            add_object("RigidMapping")

        add_object("SpringForceField", object1="@../point", object2="@mapped", indices1=0, indices2=0, length=2, stiffness=100)
        # Nodes created by context manager can be used outside of it:
        add_object("SpringForceField", object1=point.linkpath, object2=mapped.linkpath, indices1=0, indices2=0, length=2, stiffness=100)

Compared to the previous version I just replaced the code for sofa_node by the following:

class SofaNode:
    def __init__(self, name):
        self.name = name
        self.parent = None

    def __enter__(self):
        self.parent = get_current_node()
        child = add_child(self.name)
        set_current_node(child)
        return child

    def __exit__(self, exc_type, exc_value, traceback):
        set_current_node(self.parent)

Comments:

• Is there a reason to use C++ code as you wrote rather than Python code as I did?
• Regarding the "explicit parenting" if we want to avoid a global variable, why not do as you wrote above?

with Node("my_node") as my_node:
    my_node.addObject(...)  # rather than Object(..., parent=my_node)

• I don’t understand your final consideration about "Settings".
• It is counter-intuitive (at least to me) that we can use the variable defined by the context manager (my_node above) outside of said context manager. However, this kind of usage is not unheard of; for instance that’s how Tensorflow handles its "tape" object:

with tf.GradientTape() as tape:
  y = layer(x)
  loss = tf.reduce_mean(y**2)

grad = tape.gradient(loss, layer.trainable_variables)  # uses 'tape' outside the context manager

• It is a detail but I don’t like the discrepancy between the root node being defined through a def createScene() and the other nodes being defined through a with Node().

[damienmarchal]

Hi @leobois67

1. The use of c++ was there because I wanted to be sure it would be implementable on the c++ side of the binding to avoid forcing ppl to use extra python library, and the python code... because it was much easier to prototype in python, eg: the exception handling -> sofa error message, etc... ). The two line of c++ also shows that, if it is ok to go with explicit parenting, then only a minor change in the binding is needed to get a very clear indentation scheme with context manager.

2. I'm not sure I understand your second comment, to me the two following versions fit in the explicit parenting scheme (where the attaching node is in fact not depending on the indentation level).

with Node("my_node") as my_node:
    my_node.addObject(...) 
    or Object(..., parent=my_node)

Implicit parenting being like:

with Node("my_node"):
     Object(RequiredObject(""))

3. The final consideration with "Setting" was reffering to the fact that in the following code

with Node("Settings") as Settings:
     Object(RequiredObject(""))

We are actually repeating the name "Settings". One for the sofa node, one to declare a python variable's name. So I was referring that in my pas experiment I tried to add variables with the name of the Node/Object that was created (to avoid this repetitions)

4. Thank for pointing the fact that when using with X() as Y the Y variable was still usable. I do share the concern it look really weird... but well... it is how it is.

5. The problem is that in createScene, the first node is created by runSofa.

6. In your initial proposal, in addition to use function definition to indent the code, you also used them to implement of prefab (reusable component) behavior. On my experiment I was more on using the "with" statement for indentation context and "def" for function declaration like in the following

def SofaPrefab(func):
    rs=inspect.stack()[1:2] 
    
    def wrapper(*args, **kwargs):
        with Node(*args) as node:
            ss=inspect.stack()[1:2]  
            node.setInstanciationSourceFileName(ss[0][1])    # This allows, in UX friendly  GUIs to right click to "Go to Scene ...."
            node.setInstanciationSourceFilePos(ss[0][2])
            node.setDefinitionSourceFileName(rs[0][1])          # This allows, in UX friendly GUIs to right click to "Go to Implementation ...." 
            node.setDefinitionSourceFilePos(rs[0][2]+1)
            func(*args,**kwargs)
        return node    
    return wrapper

@SofaPrefab
def MyObjectAssembly(self, n=0):
    Object("MechanicalObject", name="dofs", template="Rigid3")
    Object("UniformMass", name="mass", totalMass=1)

    for i in range(n):
        Node(f"Child{i}")

    with Node("Visual"):
        Object("OglModel")
        Object("RigidMapping")

@SofaScene
def createScene(root):
    set(gravity=[1,2,3])                                           # implicit use of this_node

    with Node("Settings"):
        Object("VisualStyle", name="flags")

        with Node("Plugins"):
            Object("RequiredPlugin", name="Sofa.Component.Visual")
            MyObjectAssembly("Prefab1")
            
        MyObjectAssembly("Prefab2", n=3)

        with MyObjectAssembly("Prefab3"):
               Node("AChildToAnExistingPrefab")