Add a `:LAYOUT` read/write table

[coretl]

After DiamondLightSource/pymalcolm#390 then *METADATA.LAYOUT will contain JSON like {"COUNTER1": {"x": 1.2, "y": 2.3}, "TTLIN1": {"x": 0.0, "y": 5.6}

This should be read into an NTTable with PV :LAYOUT, looking like:

Block	x	y
COUNTER1	1.2	2.3
TTLIN1	0.0	5.6

This should accept puts that will JSON encode the contents and write it down to *METADATA.LAYOUT

[davidpcls]

I started looking into this to see if I could contribute to the solution but was not able to fully understand the changes to be made. It looks like pandablocks-client already has the ability to push this data and so only the fastcs needs updating to handle it. I think the change needs to be to add support in the introspect method of client_wrapper.py:

        for field_name, value in field_data.items():
            if field_name.startswith("*METADATA"):
                field_name_without_prefix = field_name.removeprefix("*METADATA.")
                if field_name_without_prefix == "DESIGN":
                    continue  # TODO: Handle design.
                elif field_name_without_prefix.startswith("LABEL_"):
                    labels[
                        PandaName.from_string(
                            field_name_without_prefix.removeprefix("LABEL_")
                        )
                    ] = value
                elif field_name_without_prefix.startswith("LAYOUT"):
                    layout = create_table_object_from_field_data(field_name, value)
                else:
                    exception_string = ("Received metadata not corresponding to a known type: "
                        f"{field_name} = {value}.")
                    raise TypeError(exception_string)

            else:  # Field is a default value
                initial_values[PandaName.from_string(field_name)] = value

Although I suppose you could also just leave it as a long string here and pass it on if there are no plans to actually make this modifiable. If anyone has clues on this let me know, I'm new here so I'm not all that aware of how the entire codebase works.

[coretl]

The longer version is:

• Make an AttrRW(Table([("block", str), ("x", np.float64), ("y", np.float64)])) in the top level PandA controller called layout
• If the Attr is changed by pvput, then create a JSON encoded structure of the table, doing the same line splitting that the web UI does: https://github.com/DiamondLightSource/pymalcolm/blob/264ec4a319092a545853f41f89ddf1d21da56504/malcolm/modules/pandablocks/controllers/pandamanagercontroller.py#L327-L335
• If the Attr is changed by PandA (in your code snippet above), then contruct a new Table object and pass it to the Attr

None of this is very simple (I've got a few questions here myself), so I think it might be faster for us to handle this than explaining it here

[davidpcls]

Sounds good, I'll leave it to the experts! Thanks @coretl !

[evvaaaa]

My memory of this is a little hazy... I remember a lot of work around to deal with the top-down introspection, I'd imagine a lot of it could be refactored now from this PandaController containing a wrapper class to give it block sub controllers.

I'm assuming on update from the panda you don't mean updating the datatype of the Attr post introspection like we can do here in CA. This isn't currently supported by FastCS pva. Otherwise...

1

fastcs-PandABlocks/src/fastcs_pandablocks/panda/client_wrapper.py

Lines 85 to 107 in f2bf17e

	if field_name_without_prefix == "DESIGN":
	continue # TODO: Handle design.
	elif not field_name_without_prefix.startswith("LABEL_"):
	raise TypeError(
	"Received metadata not corresponding to a `LABEL_`: "
	f"{field_name} = {value}."
	)
	labels[
	PandaName.from_string(
	field_name_without_prefix.removeprefix("LABEL_")
	)
	] = value
	else: # Field is a default value
	initial_values[PandaName.from_string(field_name)] = value
	formatted_initial_values = pformat(initial_values, indent=4).replace(
	"\n", "\n "
	)
	logging.debug(f"INITIAL VALUES:\n {formatted_initial_values}")
	formatted_labels = pformat(labels, indent=4).replace("\n", "\n ")
	logging.debug(f"LABELS:\n {formatted_labels}")
	return blocks, fields, labels, initial_values

Goes to

         layout_dtype, layout_value = [], []
         if field_name_without_prefix == "DESIGN": 
             continue  # TODO: Handle design. 
         elif field_name_without_prefix.startswith("LABEL_"): 
             labels[ 
                 PandaName.from_string( 
                     field_name_without_prefix.removeprefix("LABEL_") 
                 ) 
             ] = value
         elif field_name_without_prefix == "LAYOUT":
             # Check it's not malformed
             different_dimensions = set(tuple(dimensions) for dimensions in value.values())
             assert len(different_dimensions) == 1
             dimensions = list(different_dimensions)[0]
             layout_dtype = [("block", "<U20")] + [(dim, np.float64) for dim in dimensions]]
             layout_value = [(block,) + tuple(dim.values()) for block, dim in value.items()]
              
         else:
             raise TypeError( 
                 "Received metadata not corresponding and unknown field" 
                 f"{field_name} = {value}." 
             )
     else:  # Field is a default value 
         initial_values[PandaName.from_string(field_name)] = value 
  
 formatted_initial_values = pformat(initial_values, indent=4).replace( 
     "\n", "\n    " 
 ) 
 logging.debug(f"INITIAL VALUES:\n    {formatted_initial_values}") 
 formatted_labels = pformat(labels, indent=4).replace("\n", "\n    ") 
 logging.debug(f"LABELS:\n    {formatted_labels}") 
  
 return blocks, fields, labels, initial_values, layout_dtype, layout_value

2

Create a table attribute using the dtype here, add a callback which writes the json, and puts it to the panda with self._raw_panda.

fastcs-PandABlocks/src/fastcs_pandablocks/panda/blocks/blocks.py

Line 159 in f2bf17e

(similar to this)

fastcs-PandABlocks/src/fastcs_pandablocks/panda/blocks/blocks.py

Line 389 in f2bf17e

handler=DefaultFieldHandler(panda_name, self._raw_panda.put_value_to_panda),

The attribute should be added to the PandaController from the _blocks wrapper here.

fastcs-PandABlocks/src/fastcs_pandablocks/panda/panda_controller.py

Lines 43 to 44 in f2bf17e

	for block_name, block in self._blocks.controllers():
	self.register_sub_controller(block_name, block)

3

Update the attributes' value here if it changes.

fastcs-PandABlocks/src/fastcs_pandablocks/panda/panda_controller.py

Line 47 in f2bf17e

attribute = self._blocks.get_attribute(panda_name)

get_attribute will have to be modified to access the "top level" attribute, instead of an attribute from a controller.

fastcs-PandABlocks/src/fastcs_pandablocks/panda/blocks/blocks.py

Lines 74 to 77 in f2bf17e

	def get_attribute(self, panda_name: PandaName) -> Attribute:
	return self._introspected_controllers[
	panda_name.up_to_block()
	].panda_name_to_attribute[panda_name]

I don't believe it's currently doing this with labels (this is only done on introspection).

fastcs-PandABlocks/src/fastcs_pandablocks/panda/client_wrapper.py

Line 83 in f2bf17e

if field_name.startswith("*METADATA"):

so these should probably be exposed too.