Scene Setup Enhancement: Generate at Keyframe & Disable Cameras

[entropy010]

Hi,
Thanks again for this awesome tool. I can see it become more amd more powerful with every update :)
I hope the following suggestions are rather easy to implement (they probably aren't though - things that look simple can be very tricky under the hood)

Feature 1:
Generating at specific time without jumping to first frame:
Hitting "Generate" does not cause an animation to jump to the first frame. If a scene is at a specific keyframe then it generates at that keyframe.

StableGen Aspects that are affected:

• [ x ] Scene Setup (Cameras, Object preparation)

Additional context:
Currently the only way to put an object into different states and render it with StableGen is to export the model at a desired keyframe state as FBX into a new scene or lock/delete keyframes and copy the desired keyframe to frame 1.
This has a strong impact on a concurrent scene.

This feature would tremendously enhance generation capabilities.
For example you have a straight setup of a jacket at frame 1 and its wrinkeled at frame 20.
Rendering it at 1st and 20th frame gives you the ability to interpolate both textures with RIFE VFI frame interpolation and have an animated texture transition on the model.

---

Feature 2:
Disabling one or more cameras (hide or disable) in Viewport:
When a camera is hidden or disabled it would be removed from the array of cameras and not be considered.

StableGen Aspects that are affected:

• [ x ] Scene Setup (Cameras, Object preparation)

Additional context:
Sometimes generation of a paricular camera needs readjustment or re-rendering an Img2Img variant of a certain view. Currently the only way to realize this is to bake the solution, keep the needed cameras and rerender.

Additional / Alternative approach idea via paint mask:
This is probably an obsure and cumbersome solution but I just thought I'd give some ideas.
What if we would be a able to set up a black-white texture mask in which black represents 0% denoise and white 100% denoise. Then masked parts of a model could be rerendered in various strengths.

Anthother thing to consider is how hard it is to manage this from an end user perspective. Probably not everyone would feel the need to paint a texture mask in texture paint in oder to control certain denoise levels on the model. The inpaint/camera blending solution in its current state does a great job already.

Even though the mask would be very simple with just a blotch here and there it could be too much for many since managing textures in Blender is not very straight forward.

An automated solution idea to this approach:
Setting a distance field parameter to the camera control how strong the denoise is at a certain distance.
Similar to view distance where an object becomes invisible the denoise would become less and less.

I hope this helps finding good future ideas for future updates.
^^

[sakalond]

Thanks a lot for the ideas.

Generating at specific time without jumping to first frame

That shouldn't be too hard to implement. The only reason I fixed it at the 1st frame is that Blender automatically appends the frame number to all renders and I wanted to simplify the file management. But I get it would probably be useful to have the option to use any frame in some scenarios.

Disabling one or more cameras (hide or disable) in Viewport

Also a good idea. I certainly already ran into some cases, where I wanted to keep some cameras, but didn't want them to be used for generation as well.

Currently the only way to realize this is to bake the solution, keep the needed cameras and rerender.

I'm not sure if I fully get what you mean here, but as I understand it, you mean cases where you only need to redo a specific viewpoint(s), but keep the rest unchanged.

The thing I'm not sure about is why would you need to bake it to currently achieve it.

The Preserve Original Textures option in Refine Mode Parameters should already help with this scenario. It will refine / regenerate (depending on Denoise) using what could be a completely different set of cameras, and basically use the preexisting generation as the fallback color. It also includes a color ramp node, which you can use to fine tune the blending of the new refinement over the preexisting textures.

But I get that solution is probably not really meant for what you mean here. I could include some sort of tool which would regenerate specific viewpoint(s) using the very same inputs as the previous generation, only with new set of parameters. Only problem I currently see here is from the UX standpoint, as it would be hard to implement compatibility between the various modes (Grid, Sequential, etc.)

What if we would be a able to set up a black-white texture mask in which black represents 0% denoise and white 100% denoise. Then masked parts of a model could be rerendered in various strengths.

I already though about implementing this in some way. It could work, but there are some issues with inpainting in general, which I didn't quite solve yet (artifacting, smudging, color fringing), and need some kind of bypass at the moment (like mask blurring, expanding, lower mask strength overall). And this kind of complicates things as the result wouldn't be without additional problems and it certainly wouldn't be the same as just regenerating whole viewpoint(s).

I also had ideas like implementing regional prompting with this kind of user defined, painted on, mask / regions.

An automated solution idea to this approach:

Just a side note: This is already sort of what is happening during Sequential mode with Differential Diffusion enabled, as the mask there is already used to influence denoise. But its not based on distance, but rather the angle of the surface normal relative to the camera.

[entropy010]

I'm not sure if I fully get what you mean here, but as I understand it, you mean cases where you only need to redo a specific viewpoint(s), but keep the rest unchanged.

Exactly that yes. I explained it way to convoluted haha.

The Preserve Original Textures option in Refine Mode Parameters should already help with this scenario.

I use it to great success to fix certain areas. But when I have 6 cameras set up all 6 cameras run with lets say 0.5 denoise. There are many instances where just 1 camera projection result is bad and needs to be redone in "Generate Sequentially" or "Generate Seperately" mode.

A good example is the back of a head.

If cameras could quickly be excluded the process would look like this on a 6 camera setup:

• disable all but the cameras that have the back of the head in view
• rewrite the positive prompt just for this camera perspective
• regenerate in "Generate Sequentially" until the composition fits the back without a face
• enable all cameras
• refine with Img2Img the whole composition

I already though about implementing this in some way ... artifacting, smudging, color fringing ...

That sounds like a really tough nut to crack. From my experience touching up with one camera and adjusting the Viewpoint blending settings with inpainting options already does a good job.

Just a side note: This is already sort of what is happening during Sequential mode with Differential Diffusion enabled

https://i.imgur.com/RjgWZBk.png
I made a quick example of what I mean. The ability to cut off the cameras projection at a certain distance. So cutting off the projection distance is already possible with Differential Diffusion?

I adjusted Smooth Visibility Black/Whitepoint to various degrees and set the camera order but the projection distance is always quasi infinite.

[sakalond]

So cutting off the projection distance is already possible with Differential Diffusion?

No, it's not. I just meant that the mechanism is similar in the way that it effectively modifies the denoise levels based on the "visibility mask". Those weights are only based on the relative normal angle from all the previous cameras, so distance doesn't affect that.

I still feel like a strict distance based cutoff would probably be a bit too niche for its use-cases though. There would be some other uses for distance based blending / weighting:

One example of the current implementation's limitations is that now if you put two cameras facing the model at approximately the same angle, but one of those is further away and one is supposed to generate some details, they would still get about 50/50 split for the blending which would result in blur. Some sort of distance weight blending would help here a lot, but I feel like it would be finicky to get it to blend right, but I also didn't try it yet. For now, there is the Refine mode with the Preserve Original Textures option.

A good example is the back of a head.

I usually try to modify the camera specific prompts to fix such issues, but yeah, I see your point. Implementing the tool to regenerate a specific viewpoint is now definitely on the to-do list.

But when I have 6 cameras set up all 6 cameras run with lets say 0.5 denoise. There are many instances where just 1 camera projection result is bad

I'm not sure you are doing what I meant. What I meant is that you run the Refine mode with Preserve Original Textures only keeping the single "bad" camera, probably with a higher denoise as well. It should keep the other good viewpoints and only fix the single bad one.

But still, that doesn't change the fact that a tool for regenerating specific viewpoints would be nice.

disable all but the cameras that have the back of the head in view
rewrite the positive prompt just for this camera perspective
regenerate in "Generate Sequentially" until the composition fits the back without a face
enable all cameras
refine with Img2Img the whole composition

Ok, I guess I get it. That new tool I am thinking of should be capable of that.

[sakalond]

Also, it would have to be a separate tool, because currently, modes like Sequential and Separate overwrite the previous generations, and selecting only specific cameras would just mean they would only generate for those, keeping the rest blank / fallback.

[entropy010]

Also, it would have to be a separate tool, because currently, modes like Sequential and Separate overwrite the previous generations, and selecting only specific cameras would just mean they would only generate for those, keeping the rest blank / fallback.

Hmm makes sense. Would it be easier to just set the steps to 0 on excluded cameras? Because 0 steps auto skips the generation and keeps the previous generation intact.

So a possible setup could be:

• Camera_0 (hidden in viewport) > Set steps automatically to 0
• Camera_1 (not hidden in viewport) > Steps set to user value (eg 20)
• Camera_2 (hidden in viewport) > Set steps automatically to 0
• Camera_3 (hidden in viewport) > Set steps automatically to 0

[sakalond]

It skips it, but it is keeping the previous generation intact only because all of the cameras are currently being set to 0 steps. It basically also skips the whole image projecting step thus the previous generation is kept.