Using GXF codelets/graphs with NITROS

[3473f]

Hi, i am currently looking into using pre-existing codelets in GXF with NITROS. For this purpose I built a simple graph consisting of two codelets: a random number generator and an adder. The generated codelets and their graph run without any problems in graph composer.

For the NITROS integration, I replaced the bazel files with colcon CMakeLists.txt

project(sample_extn LANGUAGES C CXX)

# Dependencies
find_package(GXF ${ISAAC_ROS_GXF_VERSION} MODULE REQUIRED
    COMPONENTS
    std
    core
)
find_package(yaml-cpp)

# Sample extensions
add_library(sample_extn SHARED
  sample_extn.cpp
  adder.cpp
  rng.cpp
)
set(CMAKE_INCLUDE_CURRENT_DIR TRUE)
target_include_directories(sample_extn PRIVATE
    ${CMAKE_CURRENT_SOURCE_DIR}/include
    ${CMAKE_CURRENT_SOURCE_DIR}/include/adder
    ${CMAKE_CURRENT_SOURCE_DIR}/include/rng
)

target_link_libraries(sample_extn
  PUBLIC
    GXF::std
    GXF::core
    yaml-cpp
)

and wrote the following ROS node to load these graph and extensions in NITROS

#include "test_gxf_node/test_gxf_node.hpp"
#include "rclcpp_components/register_node_macro.hpp"

namespace kion {
namespace test {

const std::vector<std::pair<std::string, std::string>> EXTENSIONS = {
  {"isaac_ros_gxf", "gxf/lib/std/libgxf_std.so"},
  {"isaac_ros_gxf", "gxf/lib/multimedia/libgxf_multimedia.so"},
  {"isaac_ros_gxf", "gxf/lib/serialization/libgxf_serialization.so"},
  {"isaac_ros_gxf", "gxf/lib/cuda/libgxf_cuda.so"},
  {"test_gxf", "gxf/lib/sample_extn/libsample_extn.so"},
};
const std::vector<std::string> PRESET_EXTENSION_SPEC_NAMES = { };
const std::vector<std::string> EXTENSION_SPEC_FILENAMES = {};
const std::vector<std::string> GENERATOR_RULE_FILENAMES = {};
constexpr char APP_YAML_FILENAME[] = "config/app.yaml";
constexpr char PACKAGE_NAME[] = "test_gxf";
const nvidia::isaac_ros::nitros::NitrosPublisherSubscriberConfigMap CONFIG_MAP = {};

TestGxfNode::TestGxfNode(const rclcpp::NodeOptions & options)
: nvidia::isaac_ros::nitros::NitrosNode(options,
    APP_YAML_FILENAME,
    CONFIG_MAP,
    PRESET_EXTENSION_SPEC_NAMES,
    EXTENSION_SPEC_FILENAMES,
    GENERATOR_RULE_FILENAMES,
    EXTENSIONS,
    PACKAGE_NAME)
{
  RCLCPP_DEBUG(get_logger(), "[TestGxfNode] Constructor");
  startNitrosNode();
}

void TestGxfNode::preLoadGraphCallback()
{
  RCLCPP_DEBUG(get_logger(), "[TestGxfNode] pretLoadGraphCallback().");
}

void TestGxfNode::postLoadGraphCallback()
{
  RCLCPP_DEBUG(get_logger(), "[TestGxfNode] postLoadGraphCallback().");
}

TestGxfNode::~TestGxfNode() = default;


}
}

RCLCPP_COMPONENTS_REGISTER_NODE(kion::test::TestGxfNode)

However, I am getting the following error when I start my node

[INFO] [launch]: All log files can be found below /home/admin/.ros/log/2024-01-12-14-48-41-105961-AImonster-32372
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [32387]
[component_container_mt-1] [INFO] [1705067321.510065487] [test_gxf_container.test_gxf_container]: Load Library: /workspaces/isaac_ros-dev/install/test_gxf/lib/libtest_gxf_node.so
[component_container_mt-1] [INFO] [1705067321.534779360] [test_gxf_container.test_gxf_container]: Found class: rclcpp_components::NodeFactoryTemplate<kion::test::TestGxfNode>
[component_container_mt-1] [INFO] [1705067321.534844533] [test_gxf_container.test_gxf_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<kion::test::TestGxfNode>
[component_container_mt-1] [INFO] [1705067321.540196827] [NitrosContext]: [NitrosContext] Creating a new shared context
[component_container_mt-1] [INFO] [1705067321.540336188] [test_gxf_node]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1705067321.541086865] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so
[component_container_mt-1] [INFO] [1705067321.549216950] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_gxf_helpers.so
[component_container_mt-1] [INFO] [1705067321.552674968] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_sight.so
[component_container_mt-1] [INFO] [1705067321.556679384] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_atlas.so
[component_container_mt-1] [INFO] [1705067321.560924079] [NitrosContext]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_nitros/share/isaac_ros_nitros/config/type_adapter_nitros_context_graph.yaml'
[component_container_mt-1] [INFO] [1705067321.561560202] [NitrosContext]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1705067321.566332341] [NitrosContext]: [NitrosContext] Running application...
[component_container_mt-1] 2024-01-12 14:48:41.566 WARN  gxf/std/program.cpp@514: No system specified. Nothing to do
[component_container_mt-1] [INFO] [1705067321.566736366] [test_gxf_node]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1705067321.566752675] [test_gxf_node]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1705067321.568415816] [test_gxf_node]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1705067321.568448035] [test_gxf_node]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1705067321.568454805] [test_gxf_node]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1705067321.568460113] [test_gxf_node]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1705067321.568465127] [test_gxf_node]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1705067321.568717857] [test_gxf_node]: [NitrosContext] Loading extension: gxf/lib/libgxf_message_compositor.so
[component_container_mt-1] [INFO] [1705067321.570049932] [test_gxf_node]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1705067321.571156760] [test_gxf_node]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1705067321.574914102] [test_gxf_node]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1705067321.577963505] [test_gxf_node]: [NitrosContext] Loading extension: gxf/lib/sample_extn/libsample_extn.so
[component_container_mt-1] [INFO] [1705067321.579680526] [test_gxf_node]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] 2024-01-12 14:48:41.580 ERROR extensions/gxf_optimizer/core/configuration_graph.cpp@254: Failed to create the port groups for Vertex V4(name="rng2", type="sample::sample_extn::rng"): GXF_QUERY_NOT_FOUND
[component_container_mt-1] 2024-01-12 14:48:41.580 ERROR extensions/gxf_optimizer/core/configuration_graph.cpp@212: Failed to create a vertex based on a given YAML node
[component_container_mt-1] 2024-01-12 14:48:41.580 ERROR extensions/gxf_optimizer/core/configuration_graph.cpp@1928: Failed to create groups and vertices from YAML graph
[component_container_mt-1] 2024-01-12 14:48:41.580 ERROR external/com_nvidia_gxf/common/expected.hpp@343: Failed to load subgraphs
[component_container_mt-1] 2024-01-12 14:48:41.580 ERROR external/com_nvidia_gxf/common/expected.hpp@343: Failed to create interfaces
[component_container_mt-1] 2024-01-12 14:48:41.580 ERROR external/com_nvidia_gxf/common/expected.hpp@343: Failed to create GXF connections
[component_container_mt-1] 2024-01-12 14:48:41.580 ERROR external/com_nvidia_gxf/common/expected.hpp@343: Failed to create intra-vertex connections
[component_container_mt-1] 2024-01-12 14:48:41.580 ERROR external/com_nvidia_gxf/common/expected.hpp@343: Failed to create intra-group connections
[component_container_mt-1] 2024-01-12 14:48:41.580 ERROR extensions/gxf_optimizer/core/configuration_graph.cpp@1889: Failed to load a graph from YAML: GXF_QUERY_NOT_FOUND
[component_container_mt-1] 2024-01-12 14:48:41.580 ERROR extensions/gxf_optimizer/core/optimizer.cpp@158: Failed to load a graph from YAML nodes
[component_container_mt-1] [ERROR] [1705067321.580261280] [test_gxf_node]: [NitrosNode] optimizer_.loadGraph Error: GXF_QUERY_NOT_FOUND
[component_container_mt-1] [INFO] [1705067321.580330192] [test_gxf_node]: [NitrosNode] Terminating the running application
[component_container_mt-1] [INFO] [1705067321.580340348] [test_gxf_node]: [NitrosContext] Interrupting GXF...
[component_container_mt-1] 2024-01-12 14:48:41.580 ERROR gxf/std/program.cpp@533: Attempted interrupting when not running (state=0).
[component_container_mt-1] 2024-01-12 14:48:41.580 ERROR gxf/core/runtime.cpp@1400: Graph interrupt failed with error: GXF_INVALID_EXECUTION_SEQUENCE
[component_container_mt-1] [ERROR] [1705067321.580354692] [test_gxf_node]: [NitrosContext] GxfGraphInterrupt Error: GXF_INVALID_EXECUTION_SEQUENCE
[component_container_mt-1] [INFO] [1705067321.580360806] [test_gxf_node]: [NitrosContext] Waiting on GXF...
[component_container_mt-1] [INFO] [1705067321.580368168] [test_gxf_node]: [NitrosContext] Deinitializing...
[component_container_mt-1] [INFO] [1705067321.580375440] [test_gxf_node]: [NitrosContext] Destroying context
[component_container_mt-1] [INFO] [1705067321.580743619] [test_gxf_node]: [NitrosNode] Application termination done
[component_container_mt-1] [ERROR] [1705067321.582869578] [test_gxf_container.test_gxf_container]: Component constructor threw an exception: [NitrosNode] optimizer_.loadGraph Error: GXF_QUERY_NOT_FOUND
[ERROR] [launch_ros.actions.load_composable_nodes]: Failed to load node 'test_gxf_node' of type 'kion::test::TestGxfNode' in container '/test_gxf_container/test_gxf_container': Component constructor threw an exception: [NitrosNode] optimizer_.loadGraph Error: GXF_QUERY_NOT_FOUND

Two lines from the log are interesting to me: first, it seems that Nitros is able to find the compiled library with my custom extensions, second, the the optimizer is unable to create the vertex for my extensions throwing GXF_QUERY_NOT_FOUND. I assume this means that it is unable to find the given extension

[component_container_mt-1] [INFO] [1705067321.577963505] [test_gxf_node]: [NitrosContext] Loading extension: gxf/lib/sample_extn/libsample_extn.so
...
[component_container_mt-1] 2024-01-12 14:48:41.580 ERROR extensions/gxf_optimizer/core/configuration_graph.cpp@254: Failed to create the port groups for Vertex V4(name="rng2", type="sample::sample_extn::rng"): GXF_QUERY_NOT_FOUND

I would like to note that I am using the detectnet decoder node as a reference to my package, with the only difference being the PRESET_EXTENSION_SPEC_NAMES constant. There on L67 it is defined as the package name, however, if I define it similarly, I get the following error

[component_container_mt-1] 2024-01-12 15:13:23.633 ERROR extensions/gxf_optimizer/spec_manager/spec_manager.cpp@91: Could not find preset specs with the specified key "test_gxf"

but leaving it empty seems to work (no idea why this is the case).

So I currently have the following questions:

1. is it possible to list all the available extensions? similar to using the registry cli tool with graph composer
2. what is actually meant by the preset extension spec. names
3. if you have pointers to the right direction of getting my code to work

Here is the code to the entire ROS package and the GXF codelets: test_gxf.zip

[hemalshahNV]

This is really great work! Could you describe a bit more about what you want to do with GXF? If it is to use NITROS, we developed CUDA with NITROS for everyone to gain the benefits of NITROS without having to know GXF at all which we have used internally for some of our nodes. We do not support incorporating new GXF codelets into ROS 2 using a NITROS node. For example, in your logs above, the gxf_optimizer extension requires a spec file that there is no documentation available for.

[3473f]

It's a bit of a niche usecase really. We have lots of code developed in ISAAC SDK and since it is no longer supported, we are looking at an alternative at the moment. We want to use both DeepStream/Graph Composer and ISAAC ROS depening on the application. For machine vision applications, it would be enough to use DS and for robotics, we will have to use ROS2. That's why we want to use GXF for our core vision modules to maintain cross compatibility across both platforms and develop the robotic application specific nodes using ROS2 and NITROS.

without having to know GXF at all which we have used internally for some of our nodes.

It was great to see that the sourcecode for gxf is now available as it helped during porting :)

For example, in your logs above, the gxf_optimizer extension requires a spec file that there is no documentation available for.

I assumed it was trying to find a .yaml file, but I wasn't able to find such file in the packages or share directory.

[ZhenshengLee]

We do not support incorporating new GXF codelets into ROS 2 using a NITROS node.

@hemalshahNV
Do you mean that the open sourced gxf is limited to only update the prebuilt binaries in isaac_ros_gxf repo?
It it's true, the scalability of nitros graph acceleration is limited too.

This GXF repository contains the buildable source code for the GXF framework and its deployment for Isaac ROS NITROS. Developers could use the scripts here to build the GXF source and install the updated binaries and headers for all supported platforms in the isaac_ros_gxf package locally.
from https://github.com/NVIDIA-ISAAC-ROS/gxf/blob/862d25e756827af53059df6722b4d4ad2d6cf3a2/README.md?plain=1#L6