DescriptorSet - debugging a validation error

[gwaldron]

I have a DescriptorSet that contains several DescriptorImages. I create this and then compile it in a background thread, as in the vsgdynamicload example. I am getting a validation error during viewer->compileManager->compile():

Validation Error: [ VUID-vkDestroyImageView-imageView-01026 ] Object 0: handle = 0x22c252d1480, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x63ac21f0 |
Cannot call vkDestroyImageView on VkImageView 0x146e160000001683[] that is currently in use by a command buffer.
The Vulkan spec states: All submitted commands that refer to imageView must have completed execution
(https://vulkan.lunarg.com/doc/view/1.3.231.1/windows/1.3-extensions/vkspec.html#VUID-vkDestroyImageView-imageView-01026)

I discovered that the DescriptorSet I just created already had a set of descriptors in it's Implementation object, and is un-referencing and deleting those old instances during compilation (in DescriptorSet::Implementation::assign). The eventual call to vkDestroyImageView is resulting in the validation error.

Was my DescriptorSet recycled from a descriptor pool, preserving the old Implementation object and its Descriptor references?
Any suggestions for getting around this?

[robertosfield]

Without reviewing the code and what is happening in various threads it's not possible to suggest exactly is wrong and what to do about it.

Broadly speaking, from the Vulkan debug error it would look like the DescriptorSet is still being used by another thread other than the one you are doing the compile from, so it's really not ready to be reused. It's possible to have several frames worth of command bufffers recorded and submitted to a Vulkan queue and awaiting execution on the GPU, if you are reusing the DescriptorSet while the Vulkan objects associated with them are being actively referenced by a command buffer then this will cause this error.

The solution for this type of issue would be delaying reuse until the Descriptor and associated Descriptors have all completed rendering.

[gwaldron]

Thanks Robert. I am not re-using the DescriptorSet -- I am creating a brand new one -- but the underlying DescriptorSet::Implementation is getting recycled internally by VSG. So I am not sure how to prevent that or plan around it.

There's a comment in DescriptorSet.cpp with a great question :)

void DescriptorSet::Implementation::assign(Context& context, const Descriptors& in_descriptors)
{
    // should we do anything about previous _descriptors that may have been assigned?
    _descriptors = in_descriptors;

[timoore]

I encountered this sort of thing in vsgCs. The reference counting mechanism isn't quite sufficient to protect against recycling of resources that are still being used. When a tile is deleted in vsgCs, it is put on a list of objects to be deleted "several" frames later.

[gwaldron]

Yeah... I would really like to avoid this sort of thing :)

[gwaldron]

Besides. When is "several frames" sufficient? No way to know. Not only would you need to queue up all your to-be-deleted VSG objects for later, but you would also need to mutex any access to the Compiler so that it can't run while you are deleting the objects in that list. Which also means, the user of your SDK can never call viewer->compileManager->compile directly in a safe way. Am I wrong?

[robertosfield]

The VSG sync's the frame based on the swap chain so there a limit on how long record command buffers and the objects they are associated with need to be kept around. So "several frames" is 2 or 3 based on how many swap chain images there are.

The VSG uses mutexes around reuse of the DescriptorSet::Implementation, but doesn't mutex the Commands being recorded the command buffers as this would be too expensive for record and resetting the command resources. For the later one will need to use a thread safe mechanism for prevent deletion of subgraphs that contributed to the command buffers that are still in use.

Tim and I have previously discussed the desirability of a mechanism in the scheduling deletion of subgraphs, but I haven't had a chance to implement support for it yet. I have many tasks all vying for my time so it's a matter of bit by bit working through them.

[timoore]

Besides. When is "several frames" sufficient? No way to know. Not only would you need to queue up all your to-be-deleted VSG objects for later, but you would also need to mutex any access to the Compiler so that it can't run while you are deleting the objects in that list. Which also means, the user of your SDK can never call viewer->compileManager->compile directly in a safe way. Am I wrong?

@robertosfield gave the answer about the number of frames necessary needed to hold the objects before their resuse. I don't understand your concern about the compiler and this issue; the objects are still deleted and returned to VSG via the ref_ptr refcount mechanism, which is atomic.

[gwaldron]

Tim, I could very well be missing something. If you put an object on the "to delete later" list (as in Robert's DeleteManager mock-up), it may still be referenced in the DescriptorSet::Implementation recycle cache and therefore won't get deleted by the DeleteManager. I think the DeleteManager woulld need to verify that refcount == 1 before clearing an object from its list. Thoughts?

[robertosfield]

The delete manager wouldn't directly delete objects, it would simply unref them and if they weren't holding the only reference to the object it wouldn't get deleted. The DescriptorSet::Implementation won't get recycled until the parent DescriptorSet is deleted so I think there should be a conflict.

In cases where an application assigns a subgraph to the DeleteManager but retains a reference to the subgraph or objects within it the DeleteManager will just clear it's cached and those references will remain valid, in essence it's just a extra reference to an object hanging around in the DeleteManager for a few frames for no benefit, but also no risk.

[gwaldron]

OK I think I get it. Thanks for the discussion gents!

[robertosfield]

While I haven't spent the time on the design side, I think something as simple as a circular buffer to cache objects that are to be deleted. Psuedo code

class DeleteManager
{

    std::vector<vsg::ref_ptr<vsg::Objects>> deleteCache;
    size_t currentFrame = 0;

    DeleteManager(size_t numFramesToCache) 
    {
          for(size_t i=0; i < cnumFramesToCache; ++i) cache.push_back(Objets::create());
   }

    void advance()
    {
          currentFrame = (currentFrame+1) % (cache.size());
          cache[currentFrame]->clear();
    }

    void sheduleDelete(ref_ptr<vsg::Object> objectToDelete)
    {
        cache[currentFrame]->addChild(object);
    }
};

Something like this might be added to vsg::Viewer like the CompileManager has been. One would also want to make sure it's thread safe, and potentially you could move the actual delete to a dedicated delete thread.