Clarify usage of data store regarding reports

[clerie]

In Netbox v3.4.7 I could just place my report scripts in the specified directory on the host system and then they show up in the web interface for users to execute.

I installed a test system for Netbox v3.5.0 which introduces data stores. First thing I noticed was that scripts in the reports directory are not detected automatically anymore. I tried uploading a script and fetching one from a data store, both failed with missing write permissions in the reports directory. This changed, but is easy to fix.

When upgrading from Netbox v3.4.7 to Netbox v3.5.0 all existing reports get migrated, but safely deleting existing and adding new reports can only be done from the web interface.
If I just put a new report in the directory it does not show up, if I delete an existing one, the web interface returns a HTTP 500.

I expected placing new reports in the directory and removing them would have the same effect as in the previous Netbox version.

Is this changed behavior intended? I wonder because documentation and release notes do not tell me anything about that.

---

I changed reports and scripts directories in my Netbox configuration to a different path:

REPORTS_ROOT = '/opt/netbox-config/reports'
SCRIPTS_ROOT = '/opt/netbox-config/scripts'

[v0tti]

I think what you are supposed to do is to add a data source of type local pointing to the directory of your scripts. Those files will be imported to the corresponding root directory after you add them as a script or report.

[9er]

I have to say, I'm a bit confused, too. Does it then store the file path in the DB, or does it actually copy contents to DB or to SCRIPTS_ROOT?

We're currently using git to sync changes in scripts to the netbox servers, so uploading new scripts via the GUI is now a disappointing experience for users (directory is not writable). Is there an elegant solution? I feel like I'm missing something very obvious and am just holding it wrong.

There is custom-scripts.md in the docs and another file for reports. A few words about the local source stuff there would be appreciated.

[benley]

It appears that it wants to copy the file into SCRIPTS_ROOT, which might be okay if you check the "Auto sync enabled" option. What is unclear to me is if this can suffice in setups where there is no persistent storage for the scripts / reports directories, such as in Kubernetes deployments with multiple pod replicas (and no shared storage like a NFS server nearby).

[shepherdjay]

I am also trying to really understand how the new datastore and file structures work... since STORAGES_BACKEND doesn't seem to be reflected by the reports and scripts #13095 I'm not clear if the files are stored and served from the database or not

We're trying to set this up on GCP using container optimized os' and the reports/scripts have been a tough nut to crack as these have no persistent storage.

[candlerb]

I am still a bit confused by this too. As far as I can ascertain, the database stores references to the reports/scripts (i.e. just the filename, not the contents of the file) - because if you remove the file itself, you get an error - and optionally where it came from.

netbox=# select file_root,file_path,data_file_id,data_source_id,auto_sync_enabled from core_managedfile;
 file_root |             file_path             | data_file_id | data_source_id | auto_sync_enabled
-----------+-----------------------------------+--------------+----------------+-------------------
 scripts   | add_device_type_components.py     |              |                | f
 scripts   | bar.py                            |              |                | f
 scripts   | create_vm.py                      |              |                | f
...

Installing a report or script is no longer as simple as dropping it into a directory. But it does seem possible to update one of these simply by overwriting it on the filesystem. (If you use the UI, it seems you have to delete the old report and then upload it again).

[clerie]

I assume now, the whole situation with scripts and reports is based on the unified job scheduler introduced in v3.5.0.
The job scheduler requires an object in the database to reference the source which triggered the job and to reference all the logs correctly.

So when creating a new script, an object is created in the database and the script is placed in the database. Everything else still works as before. (Like loading available classes on demand from the file etc)

I think a possible solution to this can be the automatic creation of script and report objects on demand. So scripts get discovered in the script directory and all available classes will be displayed. You can open the form for the specific script class. As soon as you execute the script, it checks if a database entry exist and creates one if none is found. Then it can submit a job for this script with a matching object.