BlendLuxCore: Development and Release Workflow - 2nd proposal

[howetuft]

Problem Statement

Developing BlendLuxCore implies to be able to easily run and test in Blender a version of BlendLuxCore currently under development, that encompassing the Python code and the C++ extension (the wheel). This is a key feature.

A first working solution has been set in v2.10 (https://github.com/orgs/LuxCoreRender/discussions/984).

However, in my opinion, this solution presents several areas for improvements:

• Part of debug settings are not persistent, as they rely on exported environment variables (BLC_WHEEL_PATH, BLC_DEV_PATH). For instance, if we open a new console, we must reexport the variables, which can be tedious...
• Part of the debug configuration, including local paths, username etc., is stored directly in code tree (blendev-config.cfg). This creates a risk that this local data could be uploaded accidentally and be exposed worldwide, like there.
• The system relies on scripts, which must be adapted for each platform (bash, bat) - risk of desynchronization, maintenance overhead
• The system may be hard to set-up (my very personal experience).
• To update BlendLuxCore while Blender is running, we must close and reopen Blender, which may result in significant overhead when doing intensive code-and-test sessions.

Proposed improvements

There are actually 2 topics to address:

• The Python module (BlendLuxCore)
• The wheel

Python module

I merely propose to stick to Blender recommendations: https://developer.blender.org/docs/handbook/extensions/addon_dev_setup/#setting-up-project.
In short, we would ask developer to:

• create a new local repository in Blender (via GUI menu or command line)
• create a symbolic link (AKA symlink) to BlendLuxCore development directory inside the above local repository

That's all! From that point, reloading would just require to use Blender -> System -> Reload scripts menu.

It would require BLC to support reloading, as described here, thus some small adaptations would be needed.

Wheel

I propose to manage the configuration of wheel import from inside Blender.

The proposition consists in creating a dedicated submenu in BlendLuxCore settings (named "Advanced", for instance), where we could:

• set the source of the wheel: standard (PyPi), local
• trigger an installation of the wheel according to the above source and version
• optionally make the wheel be reinstalled each time BLC is reloaded
• reset everything if needed (factory reset)

Very important: this submenu should be available even if BlendLuxCore was not able to import pyluxcore.

The current cache mechanics could be kept.

Per use-case analysis

• Standard case: no change from today. pyluxcore version is hardcoded in BlendLuxCore, BLC automatically downloads the wheel from PyPi, without user interaction. Cache rules are active so that no redundant download/install occurs.

• Debug: in GUI, developer selectsLocal wheel and enter the path of the wheel to use. The wheel is typically located in LuxCore development repo (LuxCore/out/install/Debug/wheel/), but can be anywhere else as well. Then, developer can trigger installation via an Install button. Optionally, developer can ask for the install to occur each time BLC is reloaded by Blender, via a checkbox; this deactivates cache rules.

• Offline: the same as Debug. The wheel has typically been downloaded beforehand by the user in one of his/her folder. The user asks for install (Install button). Theoretically, the user should not ask for the wheel to be reinstalled each time BLC is reloaded, but the option remains open.

• Testing (advanced user). The same as above. The advanced user downloads a testing wheel from development. In the future, we could also provide a nightly build choice to download a nightly built wheel (but at the moment, we have no nightly build wheel ;-))

User is no more required to tamper with environment variable, local transfer directory (pyluxcore_custom) etc.

[CodeFHD]

Given that the v2.10-system originated from my personal way of working (at least most of what is being discussed here), I can fully agree that it can be updated to account for others preferences, programming commons, and importantly official Blender guidelines.

Responses

First, a few responses to your points. (Consider everything not commented on as fully accepted.)

• Part of debug settings are not persistent, ...
  I see the point. W.r.t. the proposed UI settings, I am wondering if we will run into a similar issue depending on the situation. While working on pyluxcore and only core BLC features, this should be fine. When BLC needs to be reinstalled a lot, for example because of changes to the installation mechanism, we would lose the settings each time. Hence, some persistent settings storage outside BLC itself would be nice. I expect the combination of reinstalling BLC and needing a custom pyluxcore version will be a rare event, however.
• Part of the debug configuration ... is stored directly in code tree ...
  Yes, that was a possible consequence if used directly from within the BLC directory. Though I would add that the script could be moved elsewhere, e.g. directly to ~/.local/bin so that custom settings never live in the git-repository. I concede that the readme does not mention this.
• ... which must be adapted for each platform (bash, bat) ...
  Here I would suggest to consider a line between "official" features and individual tools on top. At least from my perspective, this was the case with blendev: The primary mechanism by BLC are the environment variables. blendev is a utility on top to provide quick shell commands that combine them with local file paths.
• Offline [Installation]
  Small note to not forget in the implementation that there is a difference to regular local wheel usage : For the offline installation, additional dependency-wheels are needed, currently numpy and nvrtc.
• User is no more required to tamper with environment variable, local transfer directory (pyluxcore_custom) etc.
  Here I would add an additional use case: Enthusiast users/supporters testing new features. So far we did not change much in LuxCore's core rendering features. But when we do (and announce), users will want to test. The idea behind pyluxcore_custom/ was that average users don't know much about environment variables, but you can easily direct them to a folder and where to copy a file to.

An Idea

Spontaneous idea while writing the above: How about creating a separate BLC-development extension for Blender that holds local configuration?

• BLC checks for the presence of this addon (at installation and regular startup) and imports developer settings from it.
• This would ensure that the respective addon-preferences are accessible when import pyluxcore fails, without needing to rewrite BlendLuxCore (though that still may be useful, lets consider further)
• This would ensure persistent settings live in Blender-space without the need to link to persistent settings in the users local file structure.
• The regular addon preferences would not be populated with settings irrelevant to the average user.

Another note

... about previous work in the for_v2.11 branch:
In commits c6f3a9a to 56e825d I had started to use extensions_path_user() following support requests where users couldn't install in company networks.
The true reason (write permission vs pip blocked) remained unknown and we agreed then to not prioritize this because we should not actively try to work around IT regulations that a company may have made on purpose.
Still, in the scope of this we should revise the installation paths. This set of commits was on my personal list to verify again during the v2.11 development, i.e. I did not yet consider it fully tested for stable release.

[howetuft]

Hello,
Thank you for your feedback! Here are some answers:

When BLC needs to be reinstalled a lot, for example because of changes to the installation mechanism

I think this is a narrow case and, for such a special situation, we could fall back to scripts, as suggested in your third point - unless we consider the option of a 2nd extension. At any rate, to me, the main case remains that we are working on new features that do not in themselves require reinstalling BLC.

Offline installation

Yes, you're right, thank you for the reminder! It means we should have two variants under local mode: pyluxcore only (default) and pyluxcore + dependencies.

Enthusiast users/supporters testing new features

Yes, sure, that's what I called "testing/advanced user"! Indeed, copying a file in a folder is quite accessible to average user, but selecting a file via a GUI file browser may be easier yet.

creating a separate BLC-development extension

Yes, that's a good point! I had it in my mind as well...

The pros & cons, in my opinion:

Pros:

• It addresses the persistency topic in all cases, even in BLC reinstallation one.
• It releases the "never fail" constraint on BLC (which does not mean we should not seek for graceful failure handling in BLC!)
• It may involve less rework on BLC, although: 1. I'm not totally sure of that ; 2. Modifying BLC should not be an obstacle for such an important topic as dev & test process.
• It could be a good way to provide some additional debug features, like a debug toolbar, some additional diagnostics, a quick check of how well pyluxcore works, etc., without bloating BLC.
• It could even provide some debug features that can only be implemented from outside BLC, like installing BLC in debug mode (see my first post), or reinstalling BLC
• Without any setting in BLC, it would be a good protection against the "incompetent snooping user" - the kind of user who modifies settings "just to see" without understanding what he/she is doing, breaks his/her configuration and comes back to forum with a "BLENDLUXCORE NOT WORKING" post...

Cons:

• It's an additional extension to maintain
• Integration of the 2 extensions will be more complex than plain embedding. We will need to be rigorous and target a high quality level in the implementation of the interface, so that it remains maintainable in the future.
• For "enthusiastic users", it will be a less straightforward solution than direct access in BLC
• Integration in BLC could allow a per-version configuration (as the Blender recommended debugging practice allows different add-on versions to be installed side-by-side).

Considering all of that, I think that the pros outweigh the cons and we should opt for a second extension.

we should revise the installation paths.

Got it. Could you list the paths you've got in mind (in a separate thread)?

[CodeFHD]

Only few comments on the discussion:

but selecting a file via a GUI file browser may be easier yet.
Absolutely! I did not think of a GUI option when I wrote my last response.

For "enthusiastic users", it will be a less straightforward solution than direct access in BLC
One could argue that this is also a narrow case, if you think of enthusiasts more in terms of "want to install a nightly pyluxcore" rather than "need specific debug settings".
A few basic settings should remain in BLC anyways, like the switch to toggle LuxCore logs, because users need to be able to get us information for bug reports.
In contrast to this, just to provide an example, the second extension could have switches to toggle specific kinds of LuxCore logs (there are already internal separations).

Integration in BLC could allow a per-version configuration
If you write it like this it sounds more like a Pro rather than a Con.
If I imagine this correctly, you can either install this second addon directly and have independent configuration for Blender 4.2, 4.5, etc.,, or install it via symlink and have a single set of configuration.

Could you list the paths you've got in mind (in a separate thread)
I'll have to look into it (again) first, no specific paths on my mind right now.

About the procedure/project management kind of side to discussions like this:
I would propose to create an Issue as well. In that we could collect specific action items with checkboxes like we had in the v2.10 discussions.
That way we could separate long text (discussion) from short ToDo-lists.
What do you think?

[howetuft]

I would propose to create an Issue as well.

Sounds good to me!

On my side, I'm going to create also:

• a repository for the debug extension
• a dedicated branch in BlendLuxCore

[howetuft]

Hello,

I've created:

• an issue: [Feature request] Facilitate the development and debugging of BlendLuxCore #1031
• a branch: https://github.com/LuxCoreRender/BlendLuxCore/tree/blc-debug
• a repo: https://github.com/LuxCoreRender/BLC-Cockpit

The hardest part is over! 😄

[howetuft]

Update: repo has been renamed into BlendLuxHelper (hyphen was unwelcome in addon name...).
https://github.com/LuxCoreRender/BlendLuxHelper

[howetuft]

Wheel installation - process modification

Wheel installation is the process that takes a downloaded wheel and install it (unpack it) in the directory structure provided for this purpose by Blender. It takes place after the right wheel has been obtained (from PyPI, local path, offline directory etc.).

This process is entirely taken in charge by Blender, once the wheels entry of the blender_manifest.toml file has been correctly set.
In Blender standard, the wheels entry should be statically set at bundling time. In our case, however, the correct value for the wheels entry is only known at runtime, because it requires the platform and the Python interpreter version. As an alternative, Blender suggests to bundle all the wheels in the extension, but in our case, with 4 platforms and 5 Python versions supported, and an average wheel size of 100Mb, that's no serious answer.

Therefore, our present process consists in updating blender_manifest.toml at runtime, to point to the wheels we want to be installed, and then trigger standard Blender install process. However, there are at least 2 drawbacks:

• if the repository where BLC extension is installed is read-only, the update of blender_manifest.toml will fail - and consequently all the install.
• if BLC is installed in development mode (git repository symlinked to Blender extension repository), the update of blender_manifest.toml will directly modify the file in the dev repository, which is not acceptable.

To avoid that, I am going to modify the above process to not rely on blender_manifest.toml anymore. To do so, I will directly call the internal function that install wheels from a given list, after blender_manifest.toml has been processed (id est _bpy_internal.extensions.wheel_manager.apply_actions).
The only trouble is that it is an internal function, so no guarantee of stability, but my opinion is this should be quite stable.