Skip to content

enable preview for oak camera #189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shaosu-nvidia wants to merge 1 commit into
base: main
Choose a base branch
from
Open

enable preview for oak camera #189

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions src/plugins/oak/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ message(STATUS "Configuring DepthAI...")
set(DEPTHAI_BUILD_EXAMPLES OFF CACHE BOOL "" FORCE)
set(DEPTHAI_BUILD_TESTS OFF CACHE BOOL "" FORCE)
set(DEPTHAI_BUILD_DOCS OFF CACHE BOOL "" FORCE)
set(DEPTHAI_OPENCV_SUPPORT OFF CACHE BOOL "" FORCE)

FetchContent_Declare(
depthai
Expand All @@ -28,6 +29,22 @@ FetchContent_MakeAvailable(depthai)

message(STATUS "Building OAK camera plugin with DepthAI ${depthai_VERSION}")

# ==============================================================================
# SDL2 (for live preview window, bundled via FetchContent)
# ==============================================================================
set(SDL2_VERSION "2.30.11")

set(SDL_TEST OFF CACHE BOOL "" FORCE)
set(SDL_SHARED ON CACHE BOOL "" FORCE)
set(SDL_STATIC OFF CACHE BOOL "" FORCE)

FetchContent_Declare(
sdl2
URL "https://github.com/libsdl-org/SDL/releases/download/release-${SDL2_VERSION}/SDL2-${SDL2_VERSION}.tar.gz"
)
FetchContent_MakeAvailable(sdl2)
message(STATUS "SDL2 ${SDL2_VERSION} — live preview support enabled")
Comment on lines +35 to +46
Copy link

@coderabbitai coderabbitai bot Feb 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

🧩 Analysis chain

🌐 Web query:

SDL2 2.30.11 release date latest version

💡 Result:

  • SDL2 2.30.11 release date: January 1, 2025. [1][2]
  • Latest SDL2 version (as of Feb 27, 2026): 2.32.10 (release artifacts dated September 1, 2025 on libsdl.org). [3]

Sources:
[1] https://libsdl.org/release/ (SDL2-2.30.11 files dated 2025-01-01)
[2] https://discourse.libsdl.org/t/sdl-updated-to-version-2-30-11-for-release/56485 (commit/update for 2.30.11 on 01 Jan 2025)
[3] https://libsdl.org/release/ (SDL2-2.32.10 files dated 2025-09-01)

Consider updating to SDL2 2.32.10 or document the reason for pinning to 2.30.11.

SDL2 2.30.11 is available from libsdl.org, but version 2.32.10 was released in September 2025 and is currently the latest stable release. If remaining on 2.30.11 is intentional, add a comment explaining the rationale; otherwise, consider upgrading to the latest version.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@src/plugins/oak/CMakeLists.txt` around lines 35 - 46, The CMakeLists
currently pins SDL2 via the SDL2_VERSION variable set to "2.30.11" (used by
FetchContent_Declare sdl2 and the URL) without explanation; either update
SDL2_VERSION to "2.32.10" to use the latest stable release or add a short
comment above the SDL2_VERSION declaration explaining why 2.30.11 is required
(e.g., compatibility, API/ABI constraints, or CI/test matrix). Ensure the change
touches the SDL2_VERSION definition and the FetchContent_Declare URL reference
so they remain consistent.


# ==============================================================================
# Build OAK Plugin
# ==============================================================================
Expand All @@ -36,6 +53,7 @@ add_executable(camera_plugin_oak
core/oak_camera.cpp
core/rawdata_writer.cpp
core/frame_sink.cpp
core/preview_stream.cpp
)

target_link_libraries(camera_plugin_oak
Expand All @@ -45,8 +63,12 @@ target_link_libraries(camera_plugin_oak
mcap::mcap
oxr::oxr_core
pusherio::pusherio
SDL2::SDL2
)

# Bundle SDL2 shared library alongside the binary
install(TARGETS SDL2 LIBRARY DESTINATION plugins/oak_camera)

# Set RPATH to find bundled libraries in same directory
set_target_properties(camera_plugin_oak PROPERTIES
INSTALL_RPATH "$ORIGIN"
Expand Down
2 changes: 1 addition & 1 deletion src/plugins/oak/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ DepthAI is fetched and built automatically via FetchContent. The first build tak
cd IsaacTeleop

# Configure and build
cmake -B build
cmake -B build -DBUILD_PLUGIN_OAK_CAMERA=ON
cmake --build build --target camera_plugin_oak --parallel
```
Comment on lines +25 to 27
Copy link

@coderabbitai coderabbitai bot Feb 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Missing documentation for the new --preview feature.

The README documents all CLI options but does not mention the new --preview flag introduced in this PR. Consider adding documentation for this feature, including:

  • The --preview option in the Configuration table
  • Any SDL2 system dependencies that might be needed
  • Example usage with --preview
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@src/plugins/oak/README.md` around lines 25 - 27, Update the
src/plugins/oak/README.md to document the new CLI flag --preview: add an entry
for --preview in the Configuration/options table describing its behavior, list
any SDL2 system dependencies required to enable preview mode (e.g., libsdl2-dev
or equivalent), and add a short example command line showing how to build/run
with --preview (alongside the existing cmake examples). Ensure the text clearly
notes that preview requires SDL2 and any platform-specific notes so users know
to install the dependency before using --preview.


Expand Down
55 changes: 34 additions & 21 deletions src/plugins/oak/core/oak_camera.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
#include "oak_camera.hpp"

#include "frame_sink.hpp"
#include "preview_stream.hpp"

#include <algorithm>
#include <iostream>
Expand Down Expand Up @@ -42,18 +43,32 @@ OakCamera::OakCamera(const OakConfig& config, const std::vector<StreamConfig>& s
for (const auto& [socket, name] : sensors)
std::cout << " Socket " << static_cast<int>(socket) << ": " << name << std::endl;

create_pipeline(config, streams, sensors);
if (sensors.find(dai::CameraBoardSocket::CAM_A) == sensors.end())
throw std::runtime_error("Color sensor not found on CAM_A");
auto color_sensor_name = sensors.find(dai::CameraBoardSocket::CAM_A)->second;
auto color_resolution = color_sensor_name == "OV9782" ? dai::ColorCameraProperties::SensorResolution::THE_800_P :
dai::ColorCameraProperties::SensorResolution::THE_1080_P;

m_device->startPipeline(*m_pipeline);
static constexpr const char* kPreviewStreamName = "ColorPreview";

auto pipeline = create_pipeline(config, streams, color_resolution);

if (config.preview)
m_preview = PreviewStream::try_create(kPreviewStreamName, pipeline, color_resolution);
Comment on lines +56 to +57
Copy link

@coderabbitai coderabbitai bot Feb 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Consider logging when preview creation fails.

If PreviewStream::try_create returns nullptr (e.g., SDL initialization fails), the pipeline continues without preview but there's no indication to the user that their --preview flag was ineffective. The error is logged inside try_create, but an additional message at this level would improve user experience.

Proposed enhancement 
     if (config.preview)
+    {
         m_preview = PreviewStream::try_create(kPreviewStreamName, pipeline, color_resolution);
+        if (!m_preview)
+            std::cerr << "Warning: Preview requested but could not be initialized" << std::endl;
+    }
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@src/plugins/oak/core/oak_camera.cpp` around lines 56 - 57, If config.preview
is true but PreviewStream::try_create(kPreviewStreamName, pipeline,
color_resolution) returns nullptr, add a log message indicating preview creation
failed so users know their --preview flag had no effect; check the result
assigned to m_preview and when it is null call an appropriate logger (e.g.,
warn/error) with context including kPreviewStreamName and that SDL/preview
initialization failed to help debugging.


m_device->startPipeline(pipeline);

for (const auto& s : streams)
{
m_queues[s.camera] = m_device->getOutputQueue(core::EnumNameStreamType(s.camera), 8, false);
}

if (m_preview)
m_preview->setOutputQueue(m_device->getOutputQueue(kPreviewStreamName, 4, false));

std::cout << "OAK camera pipeline started" << std::endl;
}

OakCamera::~OakCamera() = default;

dai::DeviceInfo OakCamera::find_device(const std::string& device_id)
{
auto devices = dai::Device::getAllAvailableDevices();
Expand Down Expand Up @@ -83,27 +98,27 @@ dai::DeviceInfo OakCamera::find_device(const std::string& device_id)
// Pipeline construction
// =============================================================================

void OakCamera::create_pipeline(const OakConfig& config,
const std::vector<StreamConfig>& streams,
const std::unordered_map<dai::CameraBoardSocket, std::string>& sensors)
dai::Pipeline OakCamera::create_pipeline(const OakConfig& config,
const std::vector<StreamConfig>& streams,
dai::ColorCameraProperties::SensorResolution color_resolution)
{
m_pipeline = std::make_shared<dai::Pipeline>();
dai::Pipeline pipeline;

bool need_color = has_stream(streams, core::StreamType_Color);
bool need_mono_left = has_stream(streams, core::StreamType_MonoLeft);
bool need_mono_right = has_stream(streams, core::StreamType_MonoRight);

auto create_h264_output = [&](dai::Node::Output& source, const char* stream_name)
{
auto enc = m_pipeline->create<dai::node::VideoEncoder>();
auto enc = pipeline.create<dai::node::VideoEncoder>();
enc->setDefaultProfilePreset(config.fps, dai::VideoEncoderProperties::Profile::H264_BASELINE);
enc->setBitrate(config.bitrate);
enc->setQuality(config.quality);
enc->setKeyframeFrequency(config.keyframe_frequency);
enc->setNumBFrames(0);
enc->setRateControlMode(dai::VideoEncoderProperties::RateControlMode::CBR);

auto xout = m_pipeline->create<dai::node::XLinkOut>();
auto xout = pipeline.create<dai::node::XLinkOut>();
xout->setStreamName(stream_name);

source.link(enc->input);
Expand All @@ -113,16 +128,9 @@ void OakCamera::create_pipeline(const OakConfig& config,
// ---- Color camera ----
if (need_color)
{
auto it = sensors.find(dai::CameraBoardSocket::CAM_A);
if (it == sensors.end())
throw std::runtime_error("Color stream requested but no sensor found on CAM_A");

auto resolution = it->second == "OV9782" ? dai::ColorCameraProperties::SensorResolution::THE_800_P :
dai::ColorCameraProperties::SensorResolution::THE_1080_P;

auto camRgb = m_pipeline->create<dai::node::ColorCamera>();
auto camRgb = pipeline.create<dai::node::ColorCamera>();
camRgb->setBoardSocket(dai::CameraBoardSocket::CAM_A);
camRgb->setResolution(resolution);
camRgb->setResolution(color_resolution);
camRgb->setFps(config.fps);
camRgb->setColorOrder(dai::ColorCameraProperties::ColorOrder::BGR);

Expand All @@ -132,7 +140,7 @@ void OakCamera::create_pipeline(const OakConfig& config,
// ---- Mono cameras ----
if (need_mono_left)
{
auto monoLeft = m_pipeline->create<dai::node::MonoCamera>();
auto monoLeft = pipeline.create<dai::node::MonoCamera>();
monoLeft->setBoardSocket(dai::CameraBoardSocket::CAM_B);
monoLeft->setResolution(dai::MonoCameraProperties::SensorResolution::THE_400_P);
monoLeft->setFps(config.fps);
Expand All @@ -142,13 +150,15 @@ void OakCamera::create_pipeline(const OakConfig& config,

if (need_mono_right)
{
auto monoRight = m_pipeline->create<dai::node::MonoCamera>();
auto monoRight = pipeline.create<dai::node::MonoCamera>();
monoRight->setBoardSocket(dai::CameraBoardSocket::CAM_C);
monoRight->setResolution(dai::MonoCameraProperties::SensorResolution::THE_400_P);
monoRight->setFps(config.fps);

create_h264_output(monoRight->out, core::EnumNameStreamType(core::StreamType_MonoRight));
}

return pipeline;
}

// =============================================================================
Expand Down Expand Up @@ -182,6 +192,9 @@ void OakCamera::update()
m_sink->on_frame(frame);
++m_frame_counts[type];
}

if (m_preview)
m_preview->update();
}

// =============================================================================
Expand Down
15 changes: 10 additions & 5 deletions src/plugins/oak/core/oak_camera.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,9 @@ namespace plugins
namespace oak
{

// Forward declaration -- FrameSink is defined in frame_sink.hpp.
// Forward declarations
class FrameSink;
class PreviewStream;

// ============================================================================
// Stream configuration
Expand All @@ -41,6 +42,7 @@ struct OakConfig
int bitrate = 8'000'000;
int quality = 80;
int keyframe_frequency = 30;
bool preview = false;
};

struct OakFrame
Expand All @@ -65,6 +67,7 @@ class OakCamera
{
public:
OakCamera(const OakConfig& config, const std::vector<StreamConfig>& streams, std::unique_ptr<FrameSink> sink);
~OakCamera();

OakCamera(const OakCamera&) = delete;
OakCamera& operator=(const OakCamera&) = delete;
Expand All @@ -79,15 +82,17 @@ class OakCamera

private:
dai::DeviceInfo find_device(const std::string& device_id);
void create_pipeline(const OakConfig& config,
const std::vector<StreamConfig>& streams,
const std::unordered_map<dai::CameraBoardSocket, std::string>& sensors);
dai::Pipeline create_pipeline(const OakConfig& config,
const std::vector<StreamConfig>& streams,
dai::ColorCameraProperties::SensorResolution color_resolution);

std::shared_ptr<dai::Pipeline> m_pipeline;
std::shared_ptr<dai::Device> m_device;
std::map<core::StreamType, std::shared_ptr<dai::DataOutputQueue>> m_queues;

std::unique_ptr<FrameSink> m_sink;
std::map<core::StreamType, uint64_t> m_frame_counts;

std::unique_ptr<PreviewStream> m_preview;
};

} // namespace oak
Expand Down
Loading