Video Streaming #100
Replies: 29 comments 21 replies
-
|
Yes, you'd just use |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, ill try it out! |
Beta Was this translation helpful? Give feedback.
-
With |
Beta Was this translation helpful? Give feedback.
-
|
So, i tried doing this pr, pw := io.Pipe()
bufpipe := bufio.NewReadWriter(bufio.NewReader(pr),bufio.NewWriter(pw))
video := gtk.NewVideoForMediaStream(gtk.NewMediaFileForInputStream(gioutil.NewInputStream(bufpipe)))
video.SetVExpand(true)
video.SetHExpand(true)While in another part of my application bytes of the video are being written into the ReadWriter func (w *WeylusClient) Run() {
// avoid racecondition
time.Sleep(time.Second)
for {
select {
case <-w.ctx.Done():
log.Ctx(w.ctx).Err(errors.Wrap(w.ctx.Err(), "closed context")).Msg("closed context")
return
case msg := <-w.msgs:
switch msg.Type {
case websocket.MessageText:
log.Ctx(w.ctx).Info().RawJSON("data", msg.Data).Msg("received data")
for response, callbacks := range w.callbacks {
if strings.Contains(string(msg.Data), string(response)) {
for _, callback := range callbacks {
callback(msg)
}
}
}
case websocket.MessageBinary:
if _, err := w.BufPipe.Write(msg.Data); err != nil {
log.Ctx(w.ctx).Err(err).Msg("error on write data")
}
}
}
}
}And when running this, i get |
Beta Was this translation helpful? Give feedback.
-
|
Seems like that is because internally, Gtk actually checks if the input stream belongs to a file, but implementing my own MediaStream doesnt seem to work as the interface contains unexported methods. How do i do this? |
Beta Was this translation helpful? Give feedback.
-
|
Ah, I think I remember why I don't actually do this now. AFAIK, GTK never actually implemented video streaming on their code. You should double-check this with |
Beta Was this translation helpful? Give feedback.
-
|
So, is the only way to do this with gstreamer and the gtk4 gstreamer plugin, is there another way to do it or does that also not work? |
Beta Was this translation helpful? Give feedback.
-
|
It would work if you download it as a file and play that and implement your own My only concern is that subclassing is broken in gotk4 right now. I haven't touched that code in a bit, and I was still in the process of implementing it, so that unfortunately will likely not work. Maybe |
Beta Was this translation helpful? Give feedback.
-
|
Hm, i dont think downloading it is really possible, as the video is live and should be played with minimal latency, this is essentially a remote desktop app, everything except for the video is working so far, and the video aswell if i output it to stdout and pipe my app into mpv |
Beta Was this translation helpful? Give feedback.
-
|
How many frames per second is the video? Is it a low FPS one like a CCTV camera or is it a 1080p60 live stream? |
Beta Was this translation helpful? Give feedback.
-
|
It's as high fps as possible, with arbitrary resolutions. This is a native linux client for the program Weylus, as all linux browsers I've tested barely support stylus input |
Beta Was this translation helpful? Give feedback.
-
|
Hmm, from reading the latest GTK source code, the Gst backend will play if you give it a URI somehow, so you can try and fiddle with that. The FFmpeg backend should work for |
Beta Was this translation helpful? Give feedback.
-
|
Where are the docs of those backends? I haven't found those backends yet in my reading of the MediaStream and GtkVideo docs |
Beta Was this translation helpful? Give feedback.
-
|
I don't really know where the docs for those are. I'm just Here's the line of code that's causing this issue: https://gitlab.gnome.org/GNOME/gtk/-/blob/main/modules/media/gtkgstmediafile.c#L299 |
Beta Was this translation helpful? Give feedback.
-
|
I just asked about this on the GTK matrix rooms, and was pointed towards this https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/tree/main/video/gtk4 which provides a gdk.Paintable for the video |
Beta Was this translation helpful? Give feedback.
-
|
Aha! I found a hack to play back an HTTP video stream: package main
import (
"fmt"
"os"
"github.com/diamondburned/gotk4/pkg/gio/v2"
"github.com/diamondburned/gotk4/pkg/gtk/v4"
)
func main() {
app := gtk.NewApplication("com.github.diamondburned.gotk4.giotest", 0)
app.ConnectActivate(func() { activate(app) })
app.Run(os.Args)
}
func activate(app *gtk.Application) {
vfs := gio.VFSGetLocal()
file := vfs.FileForURI("https://cdn.discordapp.com/attachments/342633910322069505/1087990964372647978/LLOGE.mp4")
fmt.Println(file.URI())
video := gtk.NewVideo()
video.SetFile(file)
win := gtk.NewApplicationWindow(app)
win.SetChild(video)
win.Show()
}I will definitely be using this trick for my applications from now on. |
Beta Was this translation helpful? Give feedback.
-
|
That requires an http video tho, ill still have to use gstreamer for my app |
Beta Was this translation helpful? Give feedback.
-
|
I can't really tell exactly how the application works. I'd imagine whatever else you're using for your app will also work, as long as it has a valid and well-supported URI format for it. If not, you can always have a running FFmpeg process to transform it into a pipe. |
Beta Was this translation helpful? Give feedback.
-
|
I have a websocket on which i send input events and also send requests for frames in json, and then get Binary Messages back containing the frame encoded in h264, how would that work with ffmpeg? Gstreamer should probably work, but it's alot of setup and alot of additional complexity, as I haven't used Gstreamer before. When i get back a GtkPaintable from go-gst, can i just use type assertion to get the gotk4 type, or is there something else needed to get the proper type? |
Beta Was this translation helpful? Give feedback.
-
No, you'll have to effectively "pass ownership" from go-glib to gotk4/glib, since they're completely different libraries. You can also use gotk4 to autogenerate your own Gst bindings.
I'm not sure if
I'm actually unsure if go-gst can do this. go-gst doesn't have bindings to GTK, and Gst doesn't use GTK (it's the case the other way around), so how would it give you a |
Beta Was this translation helpful? Give feedback.
-
|
Seems like even with RTMP there is no way around writing a custom MediaStream or GFile implementation for it. How would I go about generating my own Gst bindings with gotk4? With go-gst I thought of maybe just hoping that go lets me force the variable into the correct type somehow, but that probably wouldn't work |
Beta Was this translation helpful? Give feedback.
-
You're not doing either. You'd probably be starting a local HTTP or TCP streaming server using one of those protocols and hope that Gst understands the URL. The FFmpeg implementation itself understands
It wouldn't, no. Each library has to do its own memory management that effectively involves "keep some internal state with this and take its memory ownership"; using it in both libraries would very likely cause memory leaks. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
After more testing, it does not seem to be working or viable |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
After alot of talk in the GTK matrix room, it definetly seems like the best way would be to generate bindings for gstreamer, how would i go about doing that? |
Beta Was this translation helpful? Give feedback.
-
|
Here's some code that reads a BMP from a tmpfs file: https://github.com/diamondburned/bmp-stream/blob/main/main.go. It achieves reasonable frame rates sometimes, but there's still some latency involved. It might be worth writing a bit of C code to implement your own type FramePaintable struct { ptr *C.app_frame_paintable }
// Free frees the FramePaintable from memory. You would probably call unref() in here.
func (*FramePaintable) Free()
// Swap swaps the internal buffer with buf, returning the internal to the caller.
// This operation invalidates the Paintable.
func (*FramePaintable) Swap(buf *byte) *byteThen, similar to the above code, you can swap the written buffer in, get the internal buffer out, and write to that the next time (double buffering). |
Beta Was this translation helpful? Give feedback.
-
|
It might also be worth it to do a CPU profile on the |
Beta Was this translation helpful? Give feedback.
-
|
For completeness, here's bmp-stream implemented as a DrawingArea using Cairo instead of using the newer MemoryTexture API: https://github.com/diamondburned/bmp-stream/tree/cairo. This performs a bit worse on my machine, likely because it's drawn purely using software, despite having less allocations (I'm guessing). |
Beta Was this translation helpful? Give feedback.
-
|
So, after quite a while I finally got around to testing this, and it works, somewhat 2023-03-30.20-22-17.mp4The latency is quite high, but at least its better than not working, i used the code from https://github.com/diamondburned/bmp-stream/blob/main/main.go |
Beta Was this translation helpful? Give feedback.
-
|
With hardware decoding i've been able to get it a bit faster, closer to what your bmp-stream code achieves, but still alot more latency than wanted |
Beta Was this translation helpful? Give feedback.
-
|
A CPU Profile of bmp-stream seems to show that most of the time is spent on runtime.cgocall |
Beta Was this translation helpful? Give feedback.
-
|
Another option might be to write c code that uses gstreamer and the gtk4 gstreamer plugin and then returns the gdk paintable back to go and then just use that if thats possible, it would probably be alot simpler than trying to optimize streaming bitmaps to close to zero latency for arbitrary resolutions |
Beta Was this translation helpful? Give feedback.
-
|
How would i do that casting? I haven't quite been able to figure that out |
Beta Was this translation helpful? Give feedback.
-
|
Casting? If you can get the C code to return a ptr := C.custom_get_paintable(&v.native) // function takes ref before returning
obj := coreglib.AssumeOwnership(unsafe.Pointer(ptr))
return obj.Cast().(gdk.Paintabler) |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Now. in the latest commit of my project I added enough working go bindings for Gstreamer to get the same thing to run using gotk4 https://github.com/OmegaRogue/weylus-desktop/tree/main/gstreamer |
Beta Was this translation helpful? Give feedback.
-
|
nice job , @OmegaRogue , would you consider making your work as a independent package , or an additional subpackage to gotk4 @diamondburned , this would be a really useful feature to this project , i.e. a hardware accelerated media/streaming player , i have not seen any go UI toolkit/lib offers such a feature , most of them could not event deliver a simple video player . |
Beta Was this translation helpful? Give feedback.
-
|
@OmegaRogue i could not get example running gotk4-gstreamer from both main and develop branches (on Apple M1 Pro) , it complains a lot about type mismatch in assignment and conversion issue , though i can successfully run gotk4 examples , did i miss something ? thanks |
Beta Was this translation helpful? Give feedback.
-
|
This sounds like some difference in architecture/platform. Perhaps you might want to re-run |
Beta Was this translation helpful? Give feedback.
-
|
go: downloading github.com/kr/pretty v0.3.1 |
Beta Was this translation helpful? Give feedback.
-
|
Are you already in the Nix shell? Try |
Beta Was this translation helpful? Give feedback.
-
|
I think know that issue, there should be code to manually fix it after generation in there, maybe its commented out |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Is there a way to display a video from an h264 byte stream on a video widget or does this work together with https://github.com/tinyzimmer/go-gst ?
Beta Was this translation helpful? Give feedback.
All reactions