How to modify the position and orientation of a camera in windowed app

[ZZJzhou]

Hello,

I use the script editor in isaacsim windowed app and set up a scene, which includes a world, 3 different robot arms imported by 3 urdf files. And I want to attach a camera on one of the robot arm so that i can get the view of the other robot arms.

However, when I set up the scene, the camera can not see the target and I can not modify the pose of the camera.

Can anyone help me solve this issue?

Here is my code:

import asyncio
import numpy as np
from isaacsim.core.api.world import World
from isaacsim.core.prims import RigidPrim
from isaacsim.core.api.objects import DynamicCuboid

import omni.kit.commands
import omni.usd
from isaacsim.asset.importer.urdf import _urdf

import omni
from pxr import Gf, Sdf, UsdPhysics
from omni.isaac.sensor import Camera
import omni.isaac.core.utils.numpy.rotations as rot_utils
from omni.kit.viewport.utility import get_viewport_from_window_name

async def set_scene():

    if World.instance():
        World.instance().clear_instance()
    world=World()
    await world.initialize_simulation_context_async()
    world.scene.add_default_ground_plane(z_position=-1.0)

    # Acquire the URDF extension interface for parsing and importing URDF files
    urdf_interface = _urdf.acquire_urdf_interface()

    # Configure the settings for importing the URDF file
    import_config = _urdf.ImportConfig()
    import_config.convex_decomp = False  # Disable convex decomposition for simplicity
    import_config.fix_base = True       # Fix the base of the robot to the ground
    import_config.make_default_prim = True  # Make the robot the default prim in the scene
    import_config.self_collision = False  # Disable self-collision for performance
    import_config.distance_scale = 1     # Set distance scale for the robot
    import_config.density = 0.0          # Set density to 0 (use default values)

    # Retrieve the path of the URDF file
    urdf_list = [
                 "/home/oem/xxx/isaacsim/urdf_5_0_0/rmbf8_description/urdf/instr_RMBF-8_mesh.urdf",
                 "/home/oem/xxx/isaacsim/urdf_5_0_0/rlnd8_description/urdf/instr_RLND-8_mesh.urdf",
                 "/home/oem/xxx/isaacsim/urdf_5_0_0/arm_description/robot_G2_2_2_FleshColor.urdf"
                 ]
    for urdf_path in urdf_list:
        # Parse the robot's URDF file to generate a robot model
        result, robot_model = omni.kit.commands.execute(
            "URDFParseFile",
            urdf_path=urdf_path,
            import_config=import_config
        )

        # Update the joint drive parameters for better stiffness and damping
        for joint in robot_model.joints:
            robot_model.joints[joint].drive.strength = 1047.19751  # High stiffness value
            robot_model.joints[joint].drive.damping = 52.35988    # Moderate damping value

        # Import the robot onto the current stage and retrieve its prim path
        result, prim_path = omni.kit.commands.execute(
            "URDFImportRobot",
            urdf_robot=robot_model,
            import_config=import_config,
        )
        print("result is ", result)
        print("prim path is ", prim_path)


    camera = Camera(
        prim_path="/SetupArmG2_2/L5/Camera",  
        translation=np.array([-0.65, 0, -0.8]),
        frequency=60,
        resolution=(1920, 1080),
        orientation=rot_utils.euler_angles_to_quats(np.array([0, 89, 129]), degrees=True)
    )
        
    # # create a physical scene
    stage = omni.usd.get_context().get_stage()
    # Add a physics scene prim to stage
    scene = UsdPhysics.Scene.Define(stage, Sdf.Path("/World/physicsScene"))
    # Set gravity vector
    scene.CreateGravityDirectionAttr().Set(Gf.Vec3f(0.0, 0.0, -1.0))
    scene.CreateGravityMagnitudeAttr().Set(981.0*0.01)

    world.reset()
    camera.initialize()
    camera.set_resolution([1920, 1080])

    # Get viewport api, bind camera and adjust resolution 
    viewport_api = get_viewport_from_window_name("Viewport")
    viewport_api.set_active_camera("/SetupArmG2_2/L5/Camera")   
    viewport_api.resolution=(1920, 1080)
    
asyncio.ensure_future(set_scene())

[zwdoescode]

when I set up the scene, the camera can not see the target and I can not modify the pose of the camera

What is the target that you want the camera to see? The other robot arms or do you have a cube as a target in the scene?
What error does it give when you modify the pose of the camera? I see you have already set the pose of the camera in your script. Does this generate any error?

camera = Camera(
        prim_path="/SetupArmG2_2/L5/Camera",  
        translation=np.array([-0.65, 0, -0.8]),
        frequency=60,
        resolution=(1920, 1080),
        orientation=rot_utils.euler_angles_to_quats(np.array([0, 89, 129]), degrees=True)
    )

[ZZJzhou]

I want the camera to see the end-effectors of another two robot arms, but I do not know how to set the camera properly neither in windowed app nor in python script.
Besides, the orientation data defined in python script does not match the data from the panel in the windowed app.

And when I want to save the image of viewport, but the image shows nothing but black.
Here is my following code:

    # Get viewport api, bind camera and adjust resolution 
    viewport_api = get_viewport_from_window_name("Viewport")
    viewport_api.set_active_camera("/SetupArmG2_2/L5/Camera") 
    viewport_api.resolution=(1920, 1080)
    from omni.kit.viewport.utility import capture_viewport_to_file
    imgae_path = "/home/oem/xxx/isaacsim/utils_5_0_0/img.png"
    capture_viewport_to_file(viewport_api, imgae_path)
    print("img saved")

[ZZJzhou]

And I also tried the example of Core Functions - “Hello World” of Replicator, the saved images are all black and data are void.
But no error occurred in log.

[zwdoescode]

@ZZJzhou could you please share your files for us to investigate?

[ZZJzhou]

Thks, my issue has been solved. The reason is that the scale of the robot arm does not match the camera. When I rescale the imported urdf, it solved.