Support Kitty graphics protocol mvp #5619

anthonykim1 · 2026-01-22T19:57:18Z

Part of #5592

WIP

Plan:

(Done with Add register apc handler #5617) First PR: Add registerAPCHandler to parser and expose in API, add tests
Scaffold addon:
- Create addon-kitty-graphics, copy structure from other addons
- Hook up in demo client.ts
  - Search addon-progress and ProgressAddon for how to do this
- Get it working and activating
  - (Checked, remember to remove comments) Add a console.log to the addon's activate function
Come up with a simple test command to verify in kitty or ghostty (see send-png)
Set up APC handler, get it to trigger with the test
Create a playwright test (test folder) in the addon that automates it for a simple image
- Use single 1x1 black pixel png image, write it to the terminal in your test, verify the image we see is black
- Use 3x1 png image, 255 red first, 255 green second, 255 blue third
Understand what APC sequences are being sent by the program
Implement the handlers
Draw to a canvas layer
Make sure we get the playwright tests to pass.

(Jan 22, 4:00 PM PST) There's a problem currently where kitty kitten +icat wont return image but using send-image file will.

Copilot's research on why normal kitty kitten +icat doesnt work on xterm demo rn:

kitty icat uses TIOCGWINSZ ioctl to get pixel dimensions from the PTY before sending any graphics. node-pty doesn't set pixel values in that ioctl, so it returns 0. kitty icat sees 0 pixels and errors out before even trying the graphics protocol.

(Jan 22, 6:30 PM PST)
Things like: kitty +kitten icat --use-window-size=80,24,1280,768 --transfer-mode=stream ~/Desktop/xterm.js/demo/logo.png work atm

TODO since image goes on top of text rn 🥶:

Calculate how many rows/cols the image spans
Move the cursor accordingly (or reserve blank space)

TODOs:

Fix broken placement.
Chunking for large image is broken.
Make sure cursor is on the right place
Make sure replacement, deletion work
Perf concern below
Make sure all the other feedbacks are addressed.

jerch · 2026-01-23T09:00:01Z

...node-pty doesn't set pixel values in that ioctl...

Imho this should be fixed in node-pty to populate the pixel dimensions where possible (I think there was an issue with conpty not providing it, but for other platforms it can be populated)

Another workaround for console apps is to use WinOps sequences like CSI 14t (see here for an example https://github.com/jerch/sixel-puzzle/blob/56129538af70fa4ec9441d1d3553398dc8e66f1f/termlib.py#L95) But of couse this is beyond xterm.js and has to be implemented by the console app itself.

jerch · 2026-01-23T09:13:18Z

addons/addon-kitty-graphics/src/KittyGraphicsAddon.ts

+    const binaryString = atob(base64Data);
+    const bytes = new Uint8Array(binaryString.length);
+    for (let i = 0; i < binaryString.length; i++) {
+      bytes[i] = binaryString.charCodeAt(i);
+    }


This has very bad runtime and is a nightmare creating memory pressure with huge images.

You might want to check the wasm based base64 decoder I used for the IIP protocol. It works on chunks, thus should be much friendlier to chunked data ingestion.

For basic usage see https://github.com/jerch/xterm-wasm-parts, for chunked usage addons/addon-image/src/IIPHandler.ts holds the needed bits.

jerch · 2026-01-23T09:32:18Z

addons/addon-kitty-graphics/src/KittyGraphicsAddon.ts

+    if (format === KittyFormat.PNG) {
+      // PNG: create blob and decode
+      const blob = new Blob([bytes], { type: 'image/png' });
+      return createImageBitmap(blob);


I dont know if this is still an issue - older Safari versions could not do this and needed a workaround with new Image() instead. (also see IIP handler)

It seems to work on my work mac, safari: Version 26.2 (21623.1.14.11.9) . Will check again with my personal mac at home that should be more outdated!

jerch · 2026-01-23T09:38:34Z

@anthonykim1 I start to wonder, if the kitty graphics handler should be integrated into the image addon. I don't know the requirements of this protocol for the render layering, but maybe things could be reused and unified? It would make the lifecycling easier and would not introduce another image layer.

But as I said - i dont know the kitty protocol requirements, so fusing it might be a futile attempt. Thoughts?

jerch · 2026-01-23T11:26:08Z

addons/addon-kitty-graphics/src/KittyGraphicsAddon.ts

+    this._apcHandler = terminal.parser.registerApcHandler(0x47, (data: string) => {
+      return this._handleKittyGraphics(data);
+    });


Since image data tend to be much bigger than 10 MB, the string based handler interface is not a good choice here (the string handler interface is limited to 10 MB of string data). Also the string forth and back conversion and aggregation exhibits really bad runtime here. Better - implement a full chunk based handler class and register an instance of it directly on the parser. This is much more efficient, as the utf32 codepoints emitted by the parser can be handled borrowed without copying them. (See the IIP and Sixel handlers for how to create a proper chunk handler class.)

addons/addon-kitty-graphics/src/Types.ts

addons/addon-kitty-graphics/src/KittyGraphicsAddon.ts

addons/addon-kitty-graphics/src/KittyImageRenderer.ts

Tyriar · 2026-01-23T19:50:15Z

addons/addon-kitty-graphics/test/KittyGraphicsAddon.test.ts

+// Load test images as base64
+const BLACK_1X1_BASE64 = readFileSync('./addons/addon-kitty-graphics/fixture/black-1x1.png').toString('base64');
+const RGB_3X1_BASE64 = readFileSync('./addons/addon-kitty-graphics/fixture/rgb-3x1.png').toString('base64');


You also want tests for f=24 and 32

send-png

addons/addon-kitty-graphics/src/KittyGraphicsAddon.ts

jerch

A few comments from my side...

jerch · 2026-01-28T11:04:50Z

addons/addon-kitty-graphics/src/KittyApcHandler.ts

+    // TODO: This atob + charCodeAt loop has bad runtime and creates memory pressure with large
+    // images. Consider using the wasm-based base64 decoder from xterm-wasm-parts which also
+    // supports chunked data ingestion. See addon-image/src/IIPHandler.ts for chunked usage.
+    const binaryString = atob(base64Data);


Note on this - the wasm based base64 decoder works best, if the decoded payload size is known (it does a sanity check with this). Unfortunately the kitty protocol does not announce the expected byte size, which is bad as the payload can be any size. In JS we have only 2GB at hand, therefore I strongly suggest to cap the payload at a certain size (the image addon does this at 20MB for IIP data payload and 16M pixels by default, both is adjustable)

For the decoder it means, that you prolly should do the base64 decoding in 1 MB shards (1048576 --> 786432) and add those shards manually to the full byte data.

@anthonykim1 let's move it into addon-image for this, just make sure all the kitty stuff is clearly separated in a separate folder or something

jerch · 2026-01-28T11:31:25Z

addons/addon-kitty-graphics/src/KittyApcHandler.ts

+    for (let i = 0; i < pixelCount; i++) {
+      data[dstOffset    ] = bytes[srcOffset    ]; // R
+      data[dstOffset + 1] = bytes[srcOffset + 1]; // G
+      data[dstOffset + 2] = bytes[srcOffset + 2]; // B
+      data[dstOffset + 3] = isRgba ? bytes[srcOffset + 3] : ALPHA_OPAQUE;
+      srcOffset += bytesPerPixel;
+      dstOffset += BYTES_PER_PIXEL_RGBA;
+    }


This is still subpar - for RGBA data you can simply use the bytes directly, for RGB interleaving with the 4th byte in uint32 blocks might be faster, schematically:

if (format === KittyFormat.RGBA) { imgData = new ImageData(bytes, width, height); } else { const bytes32 = new Uint32Array(BYTES.buffer); const data32 = new Uint32Array(DATA.buffer); let dstOffset = 0; let srcOffset = 0; // assuming little endian, PIXELS must be 4-aligned for (let i = 0; i < PIXELS; i += 4) { const bloc1 = bytes32[srcOffset++]; const bloc2 = bytes32[srcOffset++]; const bloc3 = bytes32[srcOffset++]; data32[dstOffset++] = bloc1 | 0xFF000000; data32[dstOffset++] = (bloc1 >> 24) | (bloc2 << 8) | 0xFF000000; data32[dstOffset++] = (bloc2 >> 16) | (bloc3 << 16) | 0xFF000000; data32[dstOffset++] = (bloc3 >> 8) | 0xFF000000; } // handle leftover here... }

This is 28 vs. 7 reads/writes per 4 pixels (memory access is a lot heavier than the bit manipulations).

On a sidenote - this byte interleaving is a typical task where wasm-simd would outperform by far (can do it with 3 instructions - load, swizzle, store).

Edit: Fixing the bit shifts. Timing is 60 ms vs. 10 ms for 4M pixels (speedup 5-6x, or 0.2 GB/s vs. 1.3 GB/s, wasm-simd would reach here >4GB/s).

Here is a gist of different versions: https://gist.github.com/jerch/e2f7695b887228d9e703c53af23c0737

Results:

$> node lib/test.wasm.js original 50.4 ms, 250 MB/s slightly optimized 23.9 ms, 526 MB/s 4 byte blocks 8.1 ms, 1553 MB/s wasm simd 2.65 ms, 4748 MB/s

This is crazy.. 50 -> 2.6 🤯🤩🔥

anthonykim1 added 2 commits January 22, 2026 10:44

Support Kitty graphics protocol

3049436

Add some playwright tests that will first fail

6a02d7c

anthonykim1 self-assigned this Jan 22, 2026

anthonykim1 marked this pull request as draft January 22, 2026 19:57

anthonykim1 changed the title ~~Support Kitty graphics protocol~~ Support Kitty graphics protocol mvp Jan 22, 2026

anthonykim1 added 10 commits January 22, 2026 11:58

npm install to update pckage-lock.json

4477463

Edit ci.yml ?

bea1628

Dont mess with og formatting for ci.yml

fc0dc3f

ci.yml

8f0e711

Renderer + og formatting for ci.yml

ec5b421

Add rendering, tweak send-png

a507afb

param in kittyGraphicsAddon.ts , ci.yml

27825f9

Rename

8b6f8f3

add query to say OK responses

849110d

comments

bc1d87d

jerch reviewed Jan 23, 2026

View reviewed changes

more comments

5e95ef6

Tyriar requested changes Jan 23, 2026

View reviewed changes

Tyriar reviewed Jan 23, 2026

View reviewed changes

addons/addon-kitty-graphics/src/KittyGraphicsAddon.ts Outdated Show resolved Hide resolved

Tyriar reviewed Jan 23, 2026

View reviewed changes

addons/addon-kitty-graphics/src/KittyGraphicsAddon.ts Outdated Show resolved Hide resolved

anthonykim1 mentioned this pull request Jan 23, 2026

Add pixel size support to resize() microsoft/node-pty#881

Merged

anthonykim1 added 5 commits January 27, 2026 17:21

Fix chunking for large data, allow pixels

1442f84

remove send-png

40342ec

Prefer ?? and move disposable location

e2dabc7

Dont forget about action o for compression

ae171ac

switch from triggerDataEvnet to this._terminal.input

17d4395

anthonykim1 added 8 commits January 27, 2026 19:04

Use Terminal.dimensions intead

d58e859

Conste num for keys, describe in jsdoc

b0b0868

Handle actions, compression first, then parseInt before Switch

259cc67

Dont use magic number, use symbol

fd64137

Replace more magic numbers with readable symbols

15c71e2

Assemble and use dataArray for ImageData, make hot-loop faster

a1c1702

Create KittyApchandler.ts

f89524e

Outsource a bunch to KittyApcHandler. TODO: go over this

8c9afdb

anthonykim1 mentioned this pull request Jan 28, 2026

Add API to expose screenElement #5629

Open

anthonykim1 added 4 commits January 27, 2026 22:20

Small details

3e8fc48

stale comments

5a79b08

comments on file system

d777119

TODO on wasm stuff

a1b69e5

jerch reviewed Jan 28, 2026

View reviewed changes

Add TODOs from jerch's comments

492d950

Support Kitty graphics protocol mvp #5619

Are you sure you want to change the base?

Support Kitty graphics protocol mvp #5619

Conversation

anthonykim1 commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jerch commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jerch Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jerch Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

anthonykim1 Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

jerch commented Jan 23, 2026

Uh oh!

jerch Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tyriar Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jerch left a comment

Choose a reason for hiding this comment

Uh oh!

jerch Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Tyriar Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

jerch Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jerch Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

anthonykim1 Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

anthonykim1 commented Jan 22, 2026 •

edited

Loading

jerch commented Jan 23, 2026 •

edited

Loading

jerch Jan 23, 2026 •

edited

Loading

jerch Jan 23, 2026 •

edited

Loading

jerch Jan 28, 2026 •

edited

Loading