Skip to content

feat(visualizer): device-aware shell rendering with Remotion Player#2023

Merged
quanru merged 26 commits intomainfrom
feat/device-shell-rendering
Mar 4, 2026
Merged

feat(visualizer): device-aware shell rendering with Remotion Player#2023
quanru merged 26 commits intomainfrom
feat/device-shell-rendering

Conversation

@quanru
Copy link
Collaborator

@quanru quanru commented Feb 24, 2026

Summary

  • Replace pixi.js canvas with Remotion Player for the main visualizer Player component
  • Add device-aware shell rendering (desktop-browser, desktop-app, iPhone, Android) based on deviceType prop
  • Implement cyberpunk visual effects (glitch, scanlines, cursor trail, ripple) for branded video export
  • Add opening/ending scenes with animated transitions
  • Embed toolbar controls (download, export, settings) directly into Remotion Player control bar via renderCustomControls API
  • Refactor frame state derivation with accumulator pattern for cleaner code
  • Add effectsEnabled toggle and playback speed settings

Test plan

  • Verify Player renders correctly with different device types (desktop-browser, iPhone, Android, desktop-app)
  • Verify device shell overlays (title bar, status bar, navigation bar) display correctly
  • Verify video export produces correct output with device shells
  • Verify toolbar buttons (download, export, settings, fullscreen) appear in Remotion control bar
  • Verify effects toggle, playback speed, and focus-on-cursor settings work
  • Verify opening/ending scenes animate correctly
  • Verify clean mode (effects disabled) works correctly

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates the visualizer playback pipeline from a Pixi.js canvas renderer to a Remotion-based player, adding device-aware shells and branded cyberpunk visual effects for both preview and video export. It also plumbs deviceType through execution dumps into the visualizer so the correct shell can be selected.

Changes:

  • Replaced the Pixi.js visualizer Player with @remotion/player and introduced Remotion compositions/scenes (opening, steps, ending, progress bar).
  • Added a shared frame-timeline model (ScriptFrame/FrameMap) and shared state derivation for Remotion preview + Canvas-based export.
  • Plumbed deviceType from core dumps → report store → visualizer player, and added a persisted effectsEnabled preference.

Reviewed changes

Copilot reviewed 26 out of 27 changed files in this pull request and generated 11 comments.

Show a summary per file
File Description
pnpm-lock.yaml Adds Remotion dependencies and removes Pixi-related packages from the lockfile.
packages/visualizer/src/utils/replay-scripts.ts Extends replay metadata to include deviceType.
packages/visualizer/src/utils/pixi-loader.ts Removes the Pixi texture loader (Pixi renderer is removed).
packages/visualizer/src/store/store.tsx Adds persisted effectsEnabled preference.
packages/visualizer/src/hooks/usePlaygroundExecution.ts Refactors hook signature to an options object and plumbs deviceType into grouped dumps.
packages/visualizer/src/component/universal-playground/index.tsx Updates usePlaygroundExecution callsite to pass options object + deviceType.
packages/visualizer/src/component/playground-result/index.tsx Passes deviceType through to the visualizer Player.
packages/visualizer/src/component/player/remotion/visual-effects.ts Adds pure computation utilities for effects + device shell resolution/layout constants.
packages/visualizer/src/component/player/remotion/frame-calculator.ts Introduces FrameMap/ScriptFrame timeline calculation for Remotion + export.
packages/visualizer/src/component/player/remotion/export-branded-video.ts Adds Canvas+MediaRecorder-based branded replay export with shells and effects.
packages/visualizer/src/component/player/remotion/derive-frame-state.ts Shared accumulator-based frame state derivation used by Remotion preview + export.
packages/visualizer/src/component/player/remotion/StepScene.tsx Implements the Remotion “steps” scene with shells/effects/cursor/ripple/overlays.
packages/visualizer/src/component/player/remotion/ProgressBar.tsx Adds a Remotion progress bar overlay.
packages/visualizer/src/component/player/remotion/OpeningScene.tsx Adds cyberpunk opening scene.
packages/visualizer/src/component/player/remotion/EndingScene.tsx Adds cyberpunk ending scene.
packages/visualizer/src/component/player/remotion/CyberOverlays.tsx Adds shared overlay components/icons used across scenes.
packages/visualizer/src/component/player/remotion/BrandedComposition.tsx Wires opening/steps/ending/progress into a single Remotion composition.
packages/visualizer/src/component/player/index.tsx Replaces Pixi player with Remotion player; adds export, effects toggle, speed control integration.
packages/visualizer/src/component/player/index.less Updates player styling for Remotion output + custom controls.
packages/visualizer/src/component/blackboard/index.tsx Replaces Pixi-based blackboard overlays with DOM/CSS overlays and click coordinate mapping.
packages/visualizer/src/component/blackboard/index.less Adds CSS overlay styling + pulse animation for blackboard highlights/points.
packages/visualizer/package.json Removes Pixi deps, adds remotion and @remotion/player.
packages/core/src/types.ts Adds optional deviceType to IGroupedActionDump + serialization.
packages/core/src/agent/agent.ts Populates deviceType on grouped dumps from interfaceType.
apps/report/src/components/store/index.tsx Stores/propagates deviceType from scripts info into report UI state.
apps/report/src/components/detail-panel/index.tsx Passes deviceType into the visualizer Player.
apps/report/src/App.tsx Passes deviceType into the visualizer Player for the main report view.
Files not reviewed (1)
  • pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented Feb 24, 2026

Deploying midscene with  Cloudflare Pages  Cloudflare Pages

Latest commit: 8977df1
Status: ✅  Deploy successful!
Preview URL: https://245e2ca7.midscene.pages.dev
Branch Preview URL: https://feat-device-shell-rendering.midscene.pages.dev

View logs

@quanru quanru force-pushed the feat/device-shell-rendering branch from eae82e7 to 88d2c2c Compare March 2, 2026 11:06
@quanru quanru marked this pull request as ready for review March 3, 2026 07:55
quanru added 20 commits March 4, 2026 11:27
…functionality

chore(deps): remove unnecessary pixi dependencies from package.json and pnpm-lock.yaml
Use renderCustomControls API to place download, export, and settings
buttons directly in the player's built-in control bar instead of a
separate toolbar section below the video.
- Remove all cyberpunk visual effects (opening/ending scenes, glitch,
  ripple, cursor trail, scanlines, HUD corners, 3D transforms)
- Delete OpeningScene, EndingScene, ProgressBar, CyberOverlays components
- Simplify frame-calculator, StepScene, export-branded-video
- Replace effectsEnabled store state with subtitleEnabled toggle
- Add subtitle visibility toggle to player settings panel
- Unify control bar button styles (play/pause, fullscreen, settings)
- Improve sidebar icon-button hover effect to match theme toggle
Add min-width constraints and proper overflow handling to prevent
page-side and main-right panels from overlapping at narrow widths.
@quanru quanru force-pushed the feat/device-shell-rendering branch from 53fb434 to d0062c0 Compare March 4, 2026 03:27
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 21 out of 22 changed files in this pull request and generated 7 comments.

Files not reviewed (1)
  • pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

const mainLayoutChangedRef = useRef(false);
const [sidebarWidth, setSidebarWidth] = useState(() => {
const saved = localStorage.getItem(SIDEBAR_WIDTH_KEY);
return saved ? Number(saved) : DEFAULT_SIDEBAR_WIDTH;
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sidebarWidth is initialized from localStorage using Number(saved) without validation. If the stored value is missing/invalid, this can become NaN and result in an invalid style.width. Consider validating/clamping the parsed value (and falling back to DEFAULT_SIDEBAR_WIDTH when Number.isFinite is false).

Suggested change
return saved ? Number(saved) : DEFAULT_SIDEBAR_WIDTH;
const parsed = saved !== null ? Number(saved) : NaN;
// Fallback to default if the stored value is missing, invalid, or non-positive.
return Number.isFinite(parsed) && parsed > 0
? parsed
: DEFAULT_SIDEBAR_WIDTH;

Copilot uses AI. Check for mistakes.
a.href = url;
a.download = 'midscene_report.html';
a.click();
URL.revokeObjectURL(url);
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

URL.revokeObjectURL(url) is called immediately after a.click(). In some browsers (notably Safari), revoking the object URL synchronously can cancel the download before it starts. Consider revoking asynchronously (e.g., via setTimeout) or after the navigation/download has been initiated.

Suggested change
URL.revokeObjectURL(url);
setTimeout(() => {
URL.revokeObjectURL(url);
}, 0);

Copilot uses AI. Check for mistakes.
Comment on lines +276 to +277
a.click();
URL.revokeObjectURL(url);
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After recording stops, the captureStream() tracks are never stopped, and the object URL is revoked immediately after a.click(). Stopping the stream tracks helps avoid leaking resources, and delaying URL.revokeObjectURL avoids flaky downloads in some browsers.

Suggested change
a.click();
URL.revokeObjectURL(url);
// Append to DOM to improve cross-browser reliability of the click-triggered download
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
// Stop all tracks from the capture stream to avoid leaking resources
stream.getTracks().forEach((track) => track.stop());
// Delay revoking the object URL to avoid flaky downloads in some browsers
setTimeout(() => {
URL.revokeObjectURL(url);
}, 1000);

Copilot uses AI. Check for mistakes.
quanru added 2 commits March 4, 2026 15:02
…n up player

- Rewrite Timeline component from pixi.js to Canvas 2D API
- Remove pixi.js and pixi-filters dependencies
- Delete pixi-loader utility module
- Hide resize handle visual indicator completely
- Simplify StepScene layout calculations for portrait mode
- Add Remotion license acknowledgment to suppress console warning
@quanru quanru requested a review from Copilot March 4, 2026 08:16
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated 10 comments.

Files not reviewed (1)
  • pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.


<span className="time-display">
{formatTime(player.currentFrame, frameMap.fps)} /{' '}
{formatTime(totalFrames, frameMap.fps)}
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The total time display uses totalFrames, but the player’s last reachable frame index is totalFrames - 1 (your seek percent calculation also uses totalFrames - 1). This causes the displayed duration to overshoot by ~1 frame (often showing 00:01 when the last frame is still 00:00). Consider formatting the end time using totalFrames - 1 (or compute duration seconds explicitly) to align UI with the actual timeline range.

Suggested change
{formatTime(totalFrames, frameMap.fps)}
{formatTime(totalFrames - 1, frameMap.fps)}

Copilot uses AI. Check for mistakes.
Comment on lines +52 to +68
while (accumulated >= frameDuration) {
accumulated -= frameDuration;
const next = frameRef.current + 1;
if (next >= durationRef.current) {
if (loopRef.current) {
frameRef.current = 0;
setCurrentFrame(0);
} else {
frameRef.current = durationRef.current - 1;
setCurrentFrame(durationRef.current - 1);
setPlaying(false);
return;
}
} else {
frameRef.current = next;
setCurrentFrame(next);
}
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This while loop can call setCurrentFrame many times in a single RAF tick (e.g., when the tab was backgrounded or playbackRate is high), causing unnecessary React renders. Consider computing the final frame advance in local variables (and whether playback should stop/loop), then calling setCurrentFrame at most once per tick.

Suggested change
while (accumulated >= frameDuration) {
accumulated -= frameDuration;
const next = frameRef.current + 1;
if (next >= durationRef.current) {
if (loopRef.current) {
frameRef.current = 0;
setCurrentFrame(0);
} else {
frameRef.current = durationRef.current - 1;
setCurrentFrame(durationRef.current - 1);
setPlaying(false);
return;
}
} else {
frameRef.current = next;
setCurrentFrame(next);
}
if (accumulated >= frameDuration) {
const framesToAdvance = Math.floor(accumulated / frameDuration);
accumulated = accumulated % frameDuration;
let newFrame = frameRef.current;
let shouldStop = false;
for (let i = 0; i < framesToAdvance; i++) {
const next = newFrame + 1;
if (next >= durationRef.current) {
if (loopRef.current) {
newFrame = 0;
} else {
newFrame = durationRef.current - 1;
shouldStop = true;
break;
}
} else {
newFrame = next;
}
}
if (newFrame !== frameRef.current) {
frameRef.current = newFrame;
setCurrentFrame(newFrame);
}
if (shouldStop) {
setPlaying(false);
return;
}

Copilot uses AI. Check for mistakes.
Comment on lines +35 to +51
function deriveTaskId(
scriptFrames: ScriptFrame[],
stepsFrame: number,
): string | null {
let taskId: string | null = null;
for (const sf of scriptFrames) {
if (sf.durationInFrames === 0) {
if (sf.startFrame <= stepsFrame) {
taskId = sf.taskId ?? taskId;
}
};

// Add error handler for MediaRecorder
mediaRecorder.onerror = (event) => {
console.error('MediaRecorder error:', event);
message.error('Video recording failed. Please try again.');
this.recording = false;
this.mediaRecorder = null;
};

this.mediaRecorder = mediaRecorder;
this.recording = true;
return this.mediaRecorder.start();
}

stop() {
if (!this.recording || !this.mediaRecorder) {
console.warn('not recording');
return;
continue;
}

// Bind onstop handler BEFORE calling stop() to ensure it's attached in time
this.mediaRecorder.onstop = () => {
// Check if we have any data
if (this.chunks.length === 0) {
console.error('No video data captured');
message.error('Video export failed: No data captured.');
return;
}

const blob = new Blob(this.chunks, { type: 'video/webm' });

// Check blob size
if (blob.size === 0) {
console.error('Video blob is empty');
message.error('Video export failed: Empty file.');
return;
}

const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = 'midscene_replay.webm';
a.click();
URL.revokeObjectURL(url);
};

this.mediaRecorder.stop();
this.recording = false;
this.mediaRecorder = null;
if (stepsFrame < sf.startFrame) break;
taskId = sf.taskId ?? taskId;
}
return taskId;
}
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deriveTaskId linearly scans scriptFrames. Since it’s invoked from an effect that runs every frame update, this becomes O(N) work per frame and can get expensive for larger replays. Consider precomputing a frame→taskId lookup (e.g., an array sized totalDurationInFrames or a list of segment boundaries with binary search), or track the current script index incrementally as playback progresses.

Copilot uses AI. Check for mistakes.
Comment on lines +129 to +136
const [controlsVisible, setControlsVisible] = useState(true);
const hideTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);

const fadeOutItem = async (
graphics: PIXI.Container | PIXI.Graphics | PIXI.Text,
duration: number,
frame: FrameFn,
): Promise<void> => {
return fadeInGraphics(graphics, duration, frame, 0);
};
const showControls = useCallback(() => {
setControlsVisible(true);
if (hideTimerRef.current) clearTimeout(hideTimerRef.current);
hideTimerRef.current = setTimeout(() => setControlsVisible(false), 3000);
}, []);
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The auto-hide timer isn’t cleared on unmount. If the component unmounts while a timeout is pending, it may attempt to call setControlsVisible after unmount. Add an effect cleanup that clears hideTimerRef.current (and consider reusing the same cleanup for the mouse-leave timer).

Copilot uses AI. Check for mistakes.
Comment on lines +251 to +253
const canvas = document.createElement('canvas');
canvas.width = W;
canvas.height = H;
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Export hard-codes a 16:9 canvas (960x540). This will distort or crop portrait replays (e.g., iPhone/Android), which is a key scenario in this PR. Consider deriving export dimensions from frameMap.imageWidth/imageHeight (or from a device-aware composition size) and updating the camera math accordingly, so exported video matches the same aspect ratio/device shell as the preview.

Copilot uses AI. Check for mistakes.
Comment on lines +198 to +202
const hasPtrData =
Math.abs(camera.pointerLeft - Math.round(baseW / 2)) > 1 ||
Math.abs(camera.pointerTop - Math.round(baseH / 2)) > 1 ||
Math.abs(prevCamera.pointerLeft - Math.round(baseW / 2)) > 1 ||
Math.abs(prevCamera.pointerTop - Math.round(baseH / 2)) > 1;
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hasPtrData compares pointer positions against baseW/baseH centers, but pointer coordinates are derived from script/image-specific dimensions (imgW/imgH and camera). When scripts contain mixed image sizes (or when base dimensions differ), this can incorrectly hide/show the cursor. Consider comparing to the effective image center for the current frame (e.g., Math.round(imgW/2) / Math.round(imgH/2)) or a dedicated 'no pointer data' sentinel.

Suggested change
const hasPtrData =
Math.abs(camera.pointerLeft - Math.round(baseW / 2)) > 1 ||
Math.abs(camera.pointerTop - Math.round(baseH / 2)) > 1 ||
Math.abs(prevCamera.pointerLeft - Math.round(baseW / 2)) > 1 ||
Math.abs(prevCamera.pointerTop - Math.round(baseH / 2)) > 1;
const centerX = Math.round(imgW / 2);
const centerY = Math.round(imgH / 2);
const hasPtrData =
Math.abs(camera.pointerLeft - centerX) > 1 ||
Math.abs(camera.pointerTop - centerY) > 1 ||
Math.abs(prevCamera.pointerLeft - centerX) > 1 ||
Math.abs(prevCamera.pointerTop - centerY) > 1;

Copilot uses AI. Check for mistakes.
Comment on lines +256 to +257
const stream = canvas.captureStream(fps);
const recorder = new MediaRecorder(stream, { mimeType: 'video/webm' });
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a fixed mimeType: 'video/webm' can throw at construction time if unsupported (and some browsers don’t support WebM/MediaRecorder at all). Consider guarding with MediaRecorder.isTypeSupported(...) and falling back to a supported type (or surfacing a clear message). Also, revoking the object URL immediately after a.click() can cancel downloads in some browsers; consider deferring URL.revokeObjectURL(url) (e.g., via setTimeout) or revoking after a user gesture completes.

Copilot uses AI. Check for mistakes.
Comment on lines +175 to +186
allScreenshots.map(async (shot, index) => {
if (imgCache.has(shot.img)) {
// Compute layout from cached image
const img = imgCache.get(shot.img)!;
const w = Math.floor(
(screenshotMaxHeight / img.naturalHeight) * img.naturalWidth,
);
allScreenshots[index].x = leftForTimeOffset(shot.timeOffset);
allScreenshots[index].y = screenshotTop;
allScreenshots[index].width = w;
allScreenshots[index].height = screenshotMaxHeight;
return;
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mutates allScreenshots items (which originate from props.screenshots). Mutating props can cause hard-to-track rendering inconsistencies and breaks React’s immutability expectations. Consider keeping a separate internal layout array/map (e.g., in stateRef.current) keyed by screenshot id, or copying props.screenshots into a new array before assigning layout fields.

Copilot uses AI. Check for mistakes.
Comment on lines +278 to +286
<img
src={img}
style={{
width: w,
height: h,
transformOrigin: '0 0',
transform: transformStyle,
}}
/>
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decorative <img> elements used purely for visual playback should include alt=\"\" (empty alt) so screen readers don’t announce the raw URL / redundant content. Apply the same to other cursor/spinner image elements in this component.

Copilot uses AI. Check for mistakes.
Comment on lines +46 to +54
const handleClick = (e: React.MouseEvent<HTMLDivElement>) => {
if (!props.onCanvasClick || !containerRef.current) return;
const rect = containerRef.current.getBoundingClientRect();
const scaleX = screenWidth / rect.width;
const scaleY = screenHeight / rect.height;
const x = Math.round((e.clientX - rect.left) * scaleX);
const y = Math.round((e.clientY - rect.top) * scaleY);
props.onCanvasClick([x, y]);
};
Copy link

Copilot AI Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Click→image coordinate mapping uses the container’s bounding box, but the screenshot <img> has its own border and the container may include overlay/layout differences. This can introduce small but consistent coordinate offsets. Consider measuring the actual screenshot image element bounds (e.g., attach a ref to .blackboard-screenshot) and compute coordinates relative to that rect for more accurate hit positions.

Copilot uses AI. Check for mistakes.
@quanru quanru merged commit b133a65 into main Mar 4, 2026
8 checks passed
@quanru quanru deleted the feat/device-shell-rendering branch March 4, 2026 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants