obs-nvenc: destroy NVENC resources before session cleanup

[aleksandr-voitenko]

Description

This PR fixes NVENC teardown ordering and adds checks to prevent NVENC API calls with an invalid/destroyed session handle.

Problem

At the moment, destroy sequence calls nvEncDestroyEncoder before releasing all NVENC-created input resources (registered textures/surfaces).
That violates the NVENC API contract and can lead to invalid session usage during cleanup.

See for details in nvenc.c function nvenc_destroy().

static void nvenc_destroy(void *data)
{
...
	// Destroys the encoder session
	if (enc->session)
		nv.nvEncDestroyEncoder(enc->session);

#ifdef _WIN32
	d3d11_free_textures(enc);
	d3d11_free(enc);
#else
	cuda_opengl_free(enc);
#endif
	cuda_free_surfaces(enc);
	cuda_ctx_free(enc);
...

Inside d3d11_free_textures:

static void d3d11_texture_free(struct nvenc_data *enc, struct nv_texture *nvtex)
{
	if (nvtex->res) {
...
		// BOOM! Use of already destroyed session
		nv.nvEncUnregisterResource(enc->session, nvtex->res);
		nvtex->tex->lpVtbl->Release(nvtex->tex);
	}
}

void d3d11_free_textures(struct nvenc_data *enc)
{
	for (size_t i = 0; i < enc->textures.num; i++) {
		d3d11_texture_free(enc, &enc->textures.array[i]);
	}
}

Inside cuda_free_surfaces:

static void cuda_surface_free(struct nvenc_data *enc, struct nv_cuda_surface *nvsurf)
{
	if (nvsurf->res) {
...
		// BOOM again!
		nv.nvEncUnregisterResource(enc->session, nvsurf->res);
		cu->cuArrayDestroy(nvsurf->tex);
	}
}

void cuda_free_surfaces(struct nvenc_data *enc)
{
...
	for (size_t i = 0; i < enc->surfaces.num; i++) {
		cuda_surface_free(enc, &enc->surfaces.array[i]);
	}
...
}

How Has This Been Tested?

Manually, Windows only.

Types of changes

• Bug fix (non-breaking change which fixes an issue)