Generic Hub Allocation/Deallocation Ports

[teddylambert]

In the generic hub, where should the bufferDeallocate, dataInDeallocate, and dataOutAllocate ports connect (to other hubs or other components)?

In the dataIn_handler, once the buffer gets deserialized on the receiving hub and the data is determined to have come through portIn on the sending hub, the dataInDeallocate_out(0, fwBuffer) function gets run. What exactly is getting deallocated? The initial buffer was allocated in send_data on the sending hub, but the deallocation call is now happening on the receiving hub with a different address space.

Additionally, is there an example deployment that uses the generic hub?

[LeStarch]

To take a step back, Fw::Buffer wraps (but does not own) a buffer to hold data. It is our way of passing U8* around without losing track of the size. The data in these buffers is owned by something else (component with stack memory, StaticMemory component, and BufferManager component. Thus the buffer must be returned to its source or memory will be leaked. This what the deallocates do.

Let us look at each port:

1. dataOutAllocate needed allocate a buffer for the data out port as we need a buffer to write things into. It will be returned/deallocated by a component later in the chain. This is usually hooked to the StaticMemory component or the BufferManager component.
2. dataInDeallocate is used to return the buffer incoming from the "wire" side of the hub to ensure that memory allocated there is released. This is usually hooked to the StaticMemory component or the BufferManager component.
3. bufferDeallocate is used to deallocate/return buffers sent in from the fprime side of the hub. This is usually hooked up to the filedownlink's buffer return port, as file downlink is (usually) the source of buffers in the system.

Typically, the hub sits on-top of a downlink/uplink stack (Framer, Deframer, StaticMemory, and Tcp driver) and it can be hooked-up in a similar fashion.

We do not have an example of the hub, but I will see if I can look-up something.

[LeStarch]

Here is example XML on how the hub is hooked-up. In this case our uplink/downlink wire interface is a Tcp Server, hence that name. I did have to obfuscate this a bit, so there may be typos. As can be seen, it mirrors the uplink downlink interface to ground.

Also note, the first 2 slots of StaticMemory are already used in this example, so those allocate/deallocate pairs start at index 2.

Also also note, the Framer and Deframer in this hub configuration is just processing Buffers as the hub dataIn and dataOut lines fill buffers.

       <!-- HUB Uplink -->
       <!-- Allocate memory for the tcpServer to fill with data from memory source (StaticMemory port 2)-->
        <connection name = "UplinkDriverAllocate">
            <source component = "tcpServer" port = "allocate" type = "Fw::BufferGet" num = "0"/>
            <target component = "staticMemory" port = "bufferAllocate" type = "Fw::BufferGet" num = "2"/>
        </connection>
       <!-- Tcp Server hands data (after allocation and reading from socket) to deframer to unpack packers -->
        <connection name = "UplinkDrvToDeframe">
            <source component = "tcpServer" port = "recv" type = "Drv::ByteStreamRecv" num = "0"/>
            <target component = "deframer" port = "framedIn" type = "Drv::ByteStreamRecv" num = "0"/>
        </connection>
       <!-- Memory allocated by tcp server must be returned to the source (StaticMemory port 2) -->
        <connection name = "UplinkDrvDeallocate">
            <source component = "deframer" port = "framedDeallocate" type = "Fw::BufferSend" num = "0"/>
            <target component = "staticMemory" port = "bufferDeallocate" type = "Fw::BufferSend" num = "2"/>
        </connection>

       <!-- Deframer needs to allocate memory for processing file packets.  Source is the buffer manager for uplink files. -->
        <connection name = "UplinkDeframerAllocate">
            <source component = "deframer" port = "bufferAllocate" type = "Fw::BufferGet" num = "0"/>
            <target component = "fileUplinkBufferManager" port = "bufferGetCallee" type = "Fw::BufferGet" num = "0"/>
        </connection>
        <!-- deframer's bufferOut is the hub's dataIn.  Deframed messages go to the hub -->
        <!-- Note: deframer Fw::Com port not hooked up, neither is bufferDeallocate called only on Fw::Com buffer types -->
        <connection name = "UplinkDeframerHub">
            <source component = "deframer" port = "bufferOut" type = "Fw::BufferSend" num = "0"/>
            <target component = "hub" port = "dataIn" type = "Fw::BufferSend" num = "0"/>
        </connection>
       <!-- Hub buffer out connected to fileUplink such that files can be sent across the hub -->
        <connection name = "UplinkfileUplink">
            <source component = "hub" port = "buffersOut" type = "Fw::BufferSend" num = "0"/>
            <target component = "fileUplink" port = "bufferSendIn" type = "Fw::BufferSend" num = "0"/>
        </connection>
       <!-- Deallocate the buffer allocated in the deframer by returning to the source buffer manager. -->
        <!-- This is only used when a port call is outcoming.  We have a second deallocate next for the File buffers-->
        <connection name = "UplinkDeallocateHub">
            <source component = "hub" port = "dataInDeallocate" type = "Fw::BufferSend" num = "0"/>
            <target component = "fileUplinkBufferManager" port = "bufferSendIn" type = "Fw::BufferSend" num = "0"/>
        </connection>
        <!-- File uplink must deallocate buffers to, as hub just passes them through -->
        <connection name = "UplinkDeallocateFileUp">
            <source component = "fileUplink" port = "bufferSendOut" type = "Fw::BufferSend" num = "0"/>
            <target component = "fileUplinkBufferManager" port = "bufferSendIn" type = "Fw::BufferSend" num = "0"/>
        </connection>

       <!-- Hub downlink (outgoing) --> 
        <!-- File downlink sends data out the hub -->
        <connection name = "DownlinkfileDownlink">
            <source component = "fileDownlink" port = "bufferSendOut" type = "BufferSend" num = "0"/>
            <target component = "hub" port = "buffersIn" type = "BufferSend" num = "0"/>
        </connection>
        <!-- Return buffer to file downlink after hub is done from it -->
        <connection name = "DownlinkHubReturnInBuffer">
            <source component = "hub" port = "bufferDeallocate" type = "Fw::BufferSend" num = "0"/>
            <target component = "fileDownlink" port = "bufferReturn" type = "Fw::BufferSend" num = "0"/>
        </connection>
       <! -- Allocate memory for buffers leaving hub going through frammer/deframer -->
        <connection name = "DownlinkHubAllocate">
            <source component = "hub" port = "dataOutAllocate" type = "Fw::BufferGet" num = "0"/>
            <target component = "staticMemory" port = "bufferAllocate" type = "Fw::BufferGet" num = "3"/>
        </connection>
       <!-- Connection from hub to framer -->
        <connection name = "DownlinkHubToFramer">
            <source component = "hub" port = "dataOut" type = "Fw::BufferSend" num = "0"/>
            <target component = "framer" port = "bufferIn" type = "Fw::BufferSend" num = "0"/>
        </connection>
        <!-- Framer deallocates memory allocated in hub -->
        <connection name = "DownlinkHubDeallocate">
            <source component = "framer" port = "bufferDeallocate" type = "Fw::BufferSend" num = "0"/>
            <target component = "staticMemory" port = "bufferDeallocate" type = "Fw::BufferSend" num = "{3"/>
        </connection>
        <!-- framer allocates memory for passing to driver -->
        <connection name = "DownlinkFramerAllocate">
            <source component = "framer" port = "framedAllocate" type = "Fw::BufferGet" num = "0"/>
            <target component = "staticMemory" port = "bufferAllocate" type = "Fw::BufferGet" num = "4"/>
        </connection>
        <!-- Framer passes data to the tcpServer -->
        <connection name = "DownlinkFramerToTcp">
            <source component = "framer" port = "framedOut" type = "Drv::ByteStreamSend" num = "0"/>
            <target component = "tcpServer" port = "send" type = "Drv::ByteStreamSend" num = "0"/>
        </connection>
        <!-- driver deallocate's data allocated in framer -->
        <connection name = "DownlinkFramerDeallocate">
            <source component = "tcpServer" port = "deallocate" type = "Fw::BufferSend" num = "0"/>
            <target component = "staticMemory" port = "bufferDeallocate" type = "Fw::BufferSend" num = "4"/>
        </connection>

        <!-- hub telemertry connection -->
        <connection name = "hubTlm">
            <source component = "hub" port = "portOut" type = "Serial" num = "1"/>
            <target component = "chanTlm" port = "TlmRecv" type = "Tlm" num = "0"/>
        </connection>

[LeStarch]

The reason this is so complicated is each stage of uplink or downlink add/subtracts items and thus needs to copy memory.

From wire to fprime:

TcpServer: allocates for wire data, deframer consumes the tcp allocated memory, and allocates smaller buffer (no headers/footers) going to hub and through hub to file uplink. Hub and file uplink deallocate that buffer.

From fprime to wire:
Hub allocates data to write serialized hub call, framer consumes this deallocates that packet, and allocates bigger packet (with headers/footers). Tcp server consumes that and deallocates it.

This is a "full" example as we have the framer and deframer here to ensure that the headter/footer/check summing is protecting out hub pair transmissions. In this example we are leaving through a network to a different device, thus data integrity is needed.

[teddylambert]

Is there any documentation on the data_type="Serial" and Fw::SerializeBufferBase& type? Haven't seen that type before and am interested in how they work. The hub doesn't handle any allocation or deallocation with this port type (which I am assuming in my system comes from a component producing data)--is it statically created by the component that is sending the data to the hub? Or in the case of the hub sending deserialized data to another component, I see in the above example that the hub portOut, which is of type serial, is connected to a tlm type port. How does this work? Can a Serial port be connected to any other port types?

[LeStarch]

Ports in F´ have the ability to serialize arguments into a buffer such that a port call can be stored in a queue (think async ports for example).

The "Serial" data type is a special data type that just reveals that buffer as a byte array. This means you can treat an input from any component's output port as a generic array of bytes in the receiving component. In the case of the generic hub, the portIn calls are of type serial because we want to handle any output port of any data type. Thus we take in an array of bytes (technically it is a class wrapping a byte array) and handle it by sending whatever it was to the other side...thus the hub can send anything from one side to the other.

[SMorettini]

Thanks for all the explanation. I'm not sure I understand everything correctly and I need some more clarifications:

considering two deployments in two devices(DEVICE_A and DEVICE_B) and I want them to communicate together with TCP

1. the topology in the example shows I should implement only one device, right?
2. if I use the topology in the example for DEVICE_A, for DEVICE_B I need a similar topology but with TcpClient. Am I right?