Embedded raster graphics images

[peterbrittain]

Some terminals support embedding graphical images in the terminal. For example:

• iTerm2: https://www.iterm2.com/documentation-images.html
• Sixel graphics: https://github.com/saitoha/PySixel

It would be nice to be able to make use of this in a cross platform manner.

[peterbrittain]

Not enough standard support across terminals yet... Deferring for some time.

[garfieldnate]

blessed supports this using w3mimgdisplay (packaged with Ranger).

[peterbrittain]

Looks like that w3m library is drawing directly to the screen in some cases, which will never work on a true terminal, so I'm not sure I'd want it in by default. Possibly an optional extra?

[peterjschroeder]

Here is my semi-working version of sixel support. It only seems to work correctly when the terminal is full screen. I believe it may have something to do with using _width and _height, but it may also be the use of TextBox for it. Any pointers would be appreciated.

`def sixel(image, x, y, max_w, max_h):
# Translate character count for max width and height to pixels.
# This may be the wrong way to go about this
max_w = int(max_w * (shutil.get_terminal_size()[0] / 12.7))
max_h = int(max_h * (shutil.get_terminal_size()[1] / 2.11))

# Convert to sixel
with Image(filename=image) as file:
    # Resize image keeping ratio
    width = file.width
    height = file.height

    if width > max_w:
        height = (height * max_w) // width
        width = max_w
    if height > max_h or height < max_h and width < max_w:
        width = (width * max_h) // height
        height = max_h

    file.sample(width, height)

    # Save as sixel
    if not os.path.exists("/tmp/sixel"):
        os.makedirs("/tmp/sixel")
    file.format = 'six'
    file.save(filename='/tmp/sixel/%s' % os.path.basename(os.path.splitext(image)[0]+'.six'))

# Display sixel
with open('/tmp/sixel/%s' % os.path.basename(os.path.splitext(image)[0]+'.six'), 'r') as file:
    image = file.read()

    # Center the image
    x += ((max_w - width) / 2) / (shutil.get_terminal_size()[0] / 12.7)
    y +- ((max_h - height) / 2) / (shutil.get_terminal_size()[1] / 2.11)

    sys.stdout.write("\x1b7\x1b[%d;%df%s\x1b8" % (y, x, image))`

Example function call.

sixel("/home/peter/.local/share/Cockatrice/Cockatrice/pics/downloadedPics/OP1/%s.jpg" % (cards[self._list.value]['name']), self._cardimage.get_location()[0]+1, self._cardimage.get_location()[1]+1, self._cardimage._w, self._cardimage._h)

[peterbrittain]

That's a good start! The worry I have is how to decide whether to allow an application to use something like that API or not. How do you detect the safety of that feature on Linux (all terminal types), MacOS and Windows (native, CygWin, or even cases like PuTTY connecting to a Linux server)?

[peterjschroeder]

As far as I know, it would have to be on a case by case basis.

if "mlterm" in os.environ['TERM'] or "xterm" in os.environ['TERM']:

I would have it fallback to a text image using libcaca or something similar.

[peterbrittain]

Asciimatics already supports image display using text images (without falling back to libcaca). Check out the renderer docs, or images.py sample.

My big worry with this feature is that until it is easy for an average user to set up and use on all basic systems, it is going to cause more problems than it solves. We can't insist on an average user doing a custom build of terminals or install non-standard terminals for their OS. And if most users just see the existing feature, is that worth the time and effort?

There will also need to be some extra logic added to the Screen objects to recognise when sixel is in use, so that it properly integrates the image with the terminal double-buffering.

[peterjschroeder]

Nice, I hadn't seen the ImageFile function.

A universal Image function would make it easy for the user.
In this Image function, it could check for sixel compatible terminals. If found it uses the sixel function, if not it falls back to ColourImageFile. Than check if the terminal supports color, if not fall back to ImageFile.

I did try to use Screen to draw the sixel, but for some reason it strips the escape codes so it just shoots out text to the screen.

[peterbrittain]

Yeah - you can't print escape codes in the Screen as it is already handling that for you in a cross-platform manner.

I need to think about this some more, but I suspect there would need to be some logic for the Screen to be aware which parts are reserved for sixel (and a way to link to the data stream for the image while it should be drawn). There also needs to be some way to spot when the sixel image needs removing (as it gets overwritten) or moving (as the screen scrolls).

[peterjschroeder]

I used textbox for the reserved space, but a dedicated widget for images would be preferable.
Is _w and _h suppose to store the current width and height of the widget. Or am I misusing these? Should there be something like get_location but for width and height?
I haven't tested scrolling yet, but if you cat a sixel on a supported terminal it scrolls properly.
As you said, there should still be a way to clear it.

[peterbrittain]

I don't quite understand how you got your custom widget to draw the image at the right time. Could you explain, or share the code?

[peterjschroeder]

It actually tries to draw it before it starts, but that is most likely a simple fix as it didn't do it before I started adjusting some things. Attached is my code.

ccg.zip