Adding new windows dynamically

[jamierobertson1]

Hi,

I'm wondering if it's currently possible to create and open new windows dynamically, and if so where is the best time and place place to do it?

I've experimented by modifying the vsgdynamicview example (unsuccessfully so far), by creating and adding a new Window and CommandGraph in a vsg::Operation and compiling the commandGraph, then addding it with addRecordAndSubmitTaskAndPresentation before calling updateViewer, but I get the impression from the errors though that there's a bit more to it when it comes to adding windows/ CommandGraph's than rather than just views.

Any advice welcome.

Thanks

Jamie

[robertosfield]

It's hard to provide advice without seeing the code to be able to comment on. Code for adding/removing windows is a very recent addition so there isn't yet a neat example illustrating/testing it.

The vsgdynamcview example is focused on just adding new vsg::View, but is probably a good starting place. Perhaps a vsgdynamicwindows would be a good follow up example. If you've modified vsgdynamicview for your own testing could you post these changes so we can review the code and test it out ourselves.

[jamierobertson1]

Here's the modified vsgdynamicviews example I've been testing with.
vsgdynamicviews

When loading the teapot model in the 2nd window, the window does pop up at least (although frozen), but if loading the lz model it crashes due to missing depth bins.

[robertosfield]

I have merged the changes and checked them into vsgExamples as a DynamicViews branch:

https://github.com/vsg-dev/vsgExamples/tree/DynamicViews

It works for me right now, but I see two teapots. I tried use lz.vsgt in the second window and it worked fine too, no crashes.

The second window isn't frozen, it just wasn't getting the events to control it's trackball because the main window's trackball hadn't been assigned a specific window so was eating all the events for both windows so the second window's trackball ignored them as they were early handled.

I have checked in a small fix to the DynamicView branch to make sure the main windows' trackball on handles events for the window it's associated with, The change is: vsg-dev/vsgExamples@1fb5bc7

However, there is still an outstanding issue that I haven't gotten to the bottom of - I get Vulkan debug errors, I presume because the compile traversal happening in the background thread is causing problem:

$ vsgdynamicviews  -d 
VUID-vkDestroyFence-fence-01120(ERROR / SPEC): msgNum: 1562993224 - Validation Error: [ VUID-vkDestroyFence-fence-01120 ] Object 0: handle = 0x40000000004, type = VK_OBJECT_TYPE_FENCE; | MessageID = 0x5d296248 | VkFence 0x40000000004[] is in use. The Vulkan spec states: All queue submission commands that refer to fence must have completed execution (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-vkDestroyFence-fence-01120)
    Objects: 1
        [0] 0x40000000004, type: 7, name: NULL
VUID-vkDestroyFence-fence-01120(ERROR / SPEC): msgNum: 1562993224 - Validation Error: [ VUID-vkDestroyFence-fence-01120 ] Object 0: handle = 0x50000000005, type = VK_OBJECT_TYPE_FENCE; | MessageID = 0x5d296248 | VkFence 0x50000000005[] is in use. The Vulkan spec states: All queue submission commands that refer to fence must have completed execution (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-vkDestroyFence-fence-01120)
    Objects: 1
        [0] 0x50000000005, type: 7, name: NULL
VUID-vkDestroyFence-fence-01120(ERROR / SPEC): msgNum: 1562993224 - Validation Error: [ VUID-vkDestroyFence-fence-01120 ] Object 0: handle = 0x60000000006, type = VK_OBJECT_TYPE_FENCE; | MessageID = 0x5d296248 | VkFence 0x60000000006[] is in use. The Vulkan spec states: All queue submission commands that refer to fence must have completed execution (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-vkDestroyFence-fence-01120)
    Objects: 1
        [0] 0x60000000006, type: 7, name: NULL
VUID-vkDestroyBuffer-buffer-00922(ERROR / SPEC): msgNum: -464217071 - Validation Error: [ VUID-vkDestroyBuffer-buffer-00922 ] Object 0: handle = 0x370000000037, type = VK_OBJECT_TYPE_BUFFER; | MessageID = 0xe4549c11 | Cannot free VkBuffer 0x370000000037[] that is in use by a command buffer. The Vulkan spec states: All submitted commands that refer to buffer, either directly or via a VkBufferView, must have completed execution (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-vkDestroyBuffer-buffer-00922)
    Objects: 1
        [0] 0x370000000037, type: 9, name: NULL
VUID-vkFreeMemory-memory-00677(ERROR / SPEC): msgNum: 1214025378 - Validation Error: [ VUID-vkFreeMemory-memory-00677 ] Object 0: handle = 0x561741023c10, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x485c8ea2 | Cannot call vkFreeMemory on VkDeviceMemory 0x380000000038[] that is currently in use by a command buffer. The Vulkan spec states: All submitted commands that refer to memory (via images or buffers) must have completed execution (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-vkFreeMemory-memory-00677)
    Objects: 1
        [0] 0x561741023c10, type: 3, name: NULL
VUID-vkFreeCommandBuffers-pCommandBuffers-00047(ERROR / SPEC): msgNum: 448332540 - Validation Error: [ VUID-vkFreeCommandBuffers-pCommandBuffers-00047 ] Object 0: handle = 0x56174115c090, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x1ab902fc | Attempt to free VkCommandBuffer 0x56174115c090[] which is in use. The Vulkan spec states: All elements of pCommandBuffers must not be in the pending state (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-vkFreeCommandBuffers-pCommandBuffers-00047)
    Objects: 1
        [0] 0x56174115c090, type: 6, name: NULL
VUID-vkDestroyBuffer-buffer-00922(ERROR / SPEC): msgNum: -464217071 - Validation Error: [ VUID-vkDestroyBuffer-buffer-00922 ] Object 0: handle = 0x3c000000003c, type = VK_OBJECT_TYPE_BUFFER; | MessageID = 0xe4549c11 | Cannot free VkBuffer 0x3c000000003c[] that is in use by a command buffer. The Vulkan spec states: All submitted commands that refer to buffer, either directly or via a VkBufferView, must have completed execution (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-vkDestroyBuffer-buffer-00922)
    Objects: 1
        [0] 0x3c000000003c, type: 9, name: NULL
VUID-vkFreeMemory-memory-00677(ERROR / SPEC): msgNum: 1214025378 - Validation Error: [ VUID-vkFreeMemory-memory-00677 ] Object 0: handle = 0x561741023c10, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x485c8ea2 | Cannot call vkFreeMemory on VkDeviceMemory 0x3d000000003d[] that is currently in use by a command buffer. The Vulkan spec states: All submitted commands that refer to memory (via images or buffers) must have completed execution (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-vkFreeMemory-memory-00677)
    Objects: 1
        [0] 0x561741023c10, type: 3, name: NULL
VUID-vkDestroySemaphore-semaphore-01137(ERROR / SPEC): msgNum: -1588160456 - Validation Error: [ VUID-vkDestroySemaphore-semaphore-01137 ] Object 0: handle = 0x561741023c10, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0xa1569838 | Cannot call vkDestroySemaphore on VkSemaphore 0x3b000000003b[] that is currently in use by a command buffer. The Vulkan spec states: All submitted batches that refer to semaphore must have completed execution (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-vkDestroySemaphore-semaphore-01137)
    Objects: 1
        [0] 0x561741023c10, type: 3, name: NULL
VUID-vkFreeCommandBuffers-pCommandBuffers-00047(ERROR / SPEC): msgNum: 448332540 - Validation Error: [ VUID-vkFreeCommandBuffers-pCommandBuffers-00047 ] Object 0: handle = 0x561740f48fb0, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x1ab902fc | Attempt to free VkCommandBuffer 0x561740f48fb0[] which is in use. The Vulkan spec states: All elements of pCommandBuffers must not be in the pending state (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-vkFreeCommandBuffers-pCommandBuffers-00047)
    Objects: 1
        [0] 0x561740f48fb0, type: 6, name: NULL
VUID-vkDestroyBuffer-buffer-00922(ERROR / SPEC): msgNum: -464217071 - Validation Error: [ VUID-vkDestroyBuffer-buffer-00922 ] Object 0: handle = 0x410000000041, type = VK_OBJECT_TYPE_BUFFER; | MessageID = 0xe4549c11 | Cannot free VkBuffer 0x410000000041[] that is in use by a command buffer. The Vulkan spec states: All submitted commands that refer to buffer, either directly or via a VkBufferView, must have completed execution (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-vkDestroyBuffer-buffer-00922)
    Objects: 1
        [0] 0x410000000041, type: 9, name: NULL
VUID-vkFreeMemory-memory-00677(ERROR / SPEC): msgNum: 1214025378 - Validation Error: [ VUID-vkFreeMemory-memory-00677 ] Object 0: handle = 0x561741023c10, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x485c8ea2 | Cannot call vkFreeMemory on VkDeviceMemory 0x420000000042[] that is currently in use by a command buffer. The Vulkan spec states: All submitted commands that refer to memory (via images or buffers) must have completed execution (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-vkFreeMemory-memory-00677)
    Objects: 1
        [0] 0x561741023c10, type: 3, name: NULL
VUID-vkDestroySemaphore-semaphore-01137(ERROR / SPEC): msgNum: -1588160456 - Validation Error: [ VUID-vkDestroySemaphore-semaphore-01137 ] Object 0: handle = 0x561741023c10, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0xa1569838 | Cannot call vkDestroySemaphore on VkSemaphore 0x400000000040[] that is currently in use by a command buffer. The Vulkan spec states: All submitted batches that refer to semaphore must have completed execution (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-vkDestroySemaphore-semaphore-01137)
    Objects: 1
        [0] 0x561741023c10, type: 3, name: NULL
VUID-vkFreeCommandBuffers-pCommandBuffers-00047(ERROR / SPEC): msgNum: 448332540 - Validation Error: [ VUID-vkFreeCommandBuffers-pCommandBuffers-00047 ] Object 0: handle = 0x561741017280, type = VK_OBJECT_TYPE_COMMAND_BUFFER; | MessageID = 0x1ab902fc | Attempt to free VkCommandBuffer 0x561741017280[] which is in use. The Vulkan spec states: All elements of pCommandBuffers must not be in the pending state (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-vkFreeCommandBuffers-pCommandBuffers-00047)
    Objects: 1
        [0] 0x561741017280, type: 6, name: NULL
VUID-vkDestroySemaphore-semaphore-01137(ERROR / SPEC): msgNum: -1588160456 - Validation Error: [ VUID-vkDestroySemaphore-semaphore-01137 ] Object 0: handle = 0x561741023c10, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0xa1569838 | Cannot call vkDestroySemaphore on VkSemaphore 0x360000000036[] that is currently in use by a command buffer. The Vulkan spec states: All submitted batches that refer to semaphore must have completed execution (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-vkDestroySemaphore-semaphore-01137)
    Objects: 1
        [0] 0x561741023c10, type: 3, name: NULL
VUID-vkDestroySemaphore-semaphore-01137(ERROR / SPEC): msgNum: -1588160456 - Validation Error: [ VUID-vkDestroySemaphore-semaphore-01137 ] Object 0: handle = 0x561741023c10, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0xa1569838 | Cannot call vkDestroySemaphore on VkSemaphore 0x30000000003[] that is currently in use by a command buffer. The Vulkan spec states: All submitted batches that refer to semaphore must have completed execution (https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VUID-vkDestroySemaphore-semaphore-01137)
    Objects: 1
        [0] 0x561741023c10, type: 3, name: NULL
    Disabling compile for view ref_ptr<vsg::View>(vsg::View 0x7f53543c29f0)
    Enabling compile for view ref_ptr<vsg::View>(vsg::View 0x7f53543c29f0)
Merge::run() Add window

Adding windows multi-threaded is pushing things to breaking point, I'm not sure yet whether we should just declate this unsupported and require users to do the compile and merge in the main thread for new windows.

[robertosfield]

I am still investigating the problems that are happening when adding windows on the fly, it currently looks like the assignment of new RecordAndSubmitTasks during a frame() is causing problems with initialization of tracking of swap buffers, fences and associated semaphores.

In Vulkan you need to explicitly manage how you render to the swap chain, and each window has it's own swap chain, and up until recently there was no way to add new windows after the initial viewer setup. I have tested the addition of new windows but only in the context of doing it before the first frame being rendered, so before the frame loop has started.

The first frame rendered initializers several data members of the RecordAndSubmitTasks but in this new usage case we're actually adding new Window and associated RecordAndSubmitTasks during a frame so the initialization is not being done in the way it's been design/implemented for.

So... turns out this all rather more complicated than just adding a new View to an existing Window.... I will need to make changes to the include/vsg/app classes to handle the usage case of adding new Windows and associated RecordAndSubmitTasks, or possible come up with an official technique for adding new windows after the viewer is in flight.

[jamierobertson1]

Thanks for investigating and the explanation. I do still get the same behavior as I originally did with the modified example (perhaps as I'm using windows). I also attempted to get it working single threaded, but got crashes in RecordAndSubmitTask, no doubt for for the reasons you describe

It'd certainly be a useful feature for me. My use case is optionally opening windows dynamically to display selected pairs of pointclouds together on the fly. Unfortunately in this case, it's difficult to know how many windows might be needed later before starting the viewers main rendering frame loop.

[robertosfield]

I have done some further work and got a solution that works without any Vulkan debug errors, but requires a bit of awkward finishing and beginning of the new frame to cope with how this example uses an update operation to do the merge:

vsg-dev/vsgExamples@ce9ad31

I think the best way to handle the addition of the new window to viewer is to do the merge after the present frame's viewer.present() call. Modifying the main frame loop loop to have a list of post viewer.present() update operations would be a bit cumbersome but would properly be the cleanest way to do things.

[robertosfield]

I have got the multi-thread load and window creation working well enough now that I've made it an official vsgdynamicwindows example, which can now part of vsgExamples master:

vsg-dev/vsgExamples#237

I added two models & window combination to the multi-thread load part and found that when one of the windows was a paged database the DatabasePager could be assigned to the Viewer then subsequently discarded by the next load. To fix bug this I had to modify the Viewer::assignRecordAndSubmitTaskAndPresentation(..) method to assign any DatabasePager that had been previously assigned. This fix is merged with VSG master: 5dcb64f.

With the latest incarnation of the vsgdynamicwindows code I use a vsg::OperationQueue to schedule merge operations that are then called in the frame loop just after the viewer.present().

        // rendering main loop
        while (viewer->advanceToNextFrame() && (numFrames < 0 || (numFrames--) > 0))
        {
            // pass any events into EventHandlers assigned to the Viewer
            viewer->handleEvents();
            viewer->update();
            viewer->recordAndSubmit();
            viewer->present();

            // do any viewer additions after present
            while(auto op = postPresentOperationQueue->take()) op->run();
        }

This is a bit clunky so I'm now considering whether the vsg::UpdateOperation to is current just used for Viewer::update() operations could be generalized so that the we could have vsg::FrameOperations that we can specify which point in the frame the operation should be run, rather than tying it just to update operations.

I will ponder on this, and update the VSG and the new vsgdynamicwindows example if I decide to go this route.

[jamierobertson1]

Maybe because I'm using windows, but I do still get a frozen windows (little blue spinning cursor icon, when the window should have focus) when running the example, and I get the following vulkan error when the program ends.

VUID-vkQueueSubmit-pWaitSemaphores-03238(ERROR / SPEC): msgNum: -1258007888 - Validation Error: [ VUID-vkQueueSubmit-pWaitSemaphores-03238 ] Object 0: handle = 0xa21a4e0000000030, type = VK_OBJECT_TYPE_SEMAPHORE; Object 1: handle = 0x1fa02555e40, type = VK_OBJECT_TYPE_QUEUE; | MessageID = 0xb50452b0 | vkQueueSubmit(): pSubmits[0].pWaitSemaphores[1] Queue VkQueue 0x1fa02555e40[] is waiting on semaphore (VkSemaphore 0xa21a4e0000000030[]) that has no way to be signaled. The Vulkan spec states: All elements of the pWaitSemaphores member of all elements of pSubmits created with a VkSemaphoreType of VK_SEMAPHORE_TYPE_BINARY must reference a semaphore signal operation that has been submitted for execution and any semaphore signal operations on which it depends (if any) must have also been submitted for execution (https://vulkan.lunarg.com/doc/view/1.3.246.1/windows/1.3-extensions/vkspec.html#VUID-vkQueueSubmit-pWaitSemaphores-03238) Objects: 2 [0] 0xa21a4e0000000030, type: 5, name: NULL [1] 0x1fa02555e40, type: 4, name: NULL

It does work perfectly when the LoadWindowOperation is not run from a thread which is all I needed, so problem solved for me.

[robertosfield]

When you get the frozen windows have you tried to break the application in the debugger to see what it's doing? I suspect the Vulkan debug error is happening on exit because somewhere the application is stalled elsewhere.

I have tried multiple running vsgdynamicwindows -d on a loop but have yet to provoke any Vulkan debug warnings.

I think it might be nice to have an example that adds/removes windows based on keyboard events rather than just having them load in a background thread. I have too many other tasks to get on with now though so will have to leave this to another day/week,

[jamierobertson1]

I have tried, but not managed to pinpoint anything so far (just normal looking calls to Queue::present). I can see that the trackballs assigned to the additional window aren't getting any events. I'm struggling to break at the point where the additional windows have focus though.

I'm travelling the next couple of days, but will have another look at it when I get a chance. It must be a consequence of adding windows from threads, as it works ok without the threads.

[robertosfield]

The revision to vsgdynamicviews moves the addition of the windows to main thread as part of the merge operation.

Does vsgwindows work OK for you? Do all the windows get events?

[jamierobertson1]

No, the new windows freeze in the new vsgwindows example . But as a test if I just do:

        int count = 0;
        // rendering main loop
        while (viewer->advanceToNextFrame() && (numFrames < 0 || (numFrames--) > 0))
        {
            // pass any events into EventHandlers assigned to the Viewer
            viewer->handleEvents();

            if (count == 600)
            {
                auto loadWindow = LoadWindowOperation::create(postPresentOperationQueue, observer_viewer, window, 500, 50, 512, 480, "models/lz.vsgt", options);
                loadWindow->run();
            }
            count++;
            viewer->update();
            viewer->recordAndSubmit();
            viewer->present();

            // do any viewer additions after present
            while(auto op = postPresentOperationQueue->take()) op->run();
        }

Then that window works ok with events