HMI - Device Bulk Import - 504 Gateway Time-out after the second import

[Djo91]

NetBox version

v3.4.0

Python version

3.8

Steps to Reproduce

1. Make a Device Bulk Import (Racked Devices, Format Auto detect), example :
   
   This import works (note that I can't delete this device, error 504 too and since my import the https://demo.netbox.dev/ site is very slow)

2. Try to import a similar Device (juste change the name and use an unused name of course)
   
   This import doesn't work, error 504 displays after few seconds

Expected Behavior

The second import should work.

Observed Behavior

[jeremystretch]

There are myriad reasons you may experience temporary outages on the public demo. It looks like it's being hammered pretty hard at the moment, likely due to an increase of people testing various features of the recent v3.4 release. Unfortunately there's not much we can do about this short of provisioning more resources.

[Djo91]

Health of my VM (upgrade to v3.4 near 10:44) :

Each disk peak corresponds to a HMI Device Bulk Import test

[kkthxbye-code]

You created 13208 objects in one bulk import: https://demo.netbox.dev/extras/changelog/?request_id=b75ac2ac-3682-4a2d-a28b-fe938034c22c

With the search indexing that's a little much for the demo instance.

[Djo91]

What king of Netbox debug can I activate to figure out what is appening ? Thanks for your reply :).

[kkthxbye-code]

There's nothing really going on. Importing 13000 objects takes a lot of time and the 504 was displayed because it didn't finish before nginx hit the configured timeout. Having multiple processes service netbox and configuring a very large timeout and there should really be no issues.

Not sure why you would get a 503 on your own instance, maybe all web server processes were busy.

[Djo91]

ith the search indexing that's a little much for the demo inst

Oh I'm sorry, I'll delete all of thoses objects.

So I suppose than my problem in my own Netbox intance it's not the same problem.

[Djo91]

Hello,

I've made some new tests this morning and I noted that CPU (to much CPU contention) seems to be less well managed in v3.4 than in v3.3.8 (it's just a guess).

My architecture (I'm the only user of my Netbox VM) :

• Netbox v3.4.0 (in docker mode) installed on a dedicated VM
• BDD installed on a dedicated VM

History :

• 15/12/2022 from 10h02 to 10h19 import of 6k devices, ~ 200k objects (devices with 32 interfaces each) => so 17min to import 200k objects (good performance). Note that in the v3.3.8 the change log doesn't include components (maybe it's important about performance)
• 15/12/2022 at 10h45 upgrade from v3.3.8 (docker mode) to v3.4.0 (docker mode)
• 15/12/2022 from 11h00 to 11h05 import of 284 devices (same type of device as at 10h02) ~ 9k objects => so 5min to import 9k objects (not a good performance if I compare with the import made at 10h02)
• 15/12/2022 from 11h10 to 16h00, multiple tests of import (6k, 4k, 2k or 1k device) => all of it had returned HTTP 503 error
• 15/12/2022 from 17h40 to 17h41 import of 21 devices (same type of device as at 10h02) => import OK without error
• 16/12/2022 from 09h16 to 09h19 import of 100 devices => import OK
• 16/12/2022 from 09h21 to 09h29 import of 200 devices => import OK
• 16/12/2022 from 09h37 to 09h54 import of 400 devices => import OK (so 17min to import 12k objects, poor performance if I compare with the import made the 15/12/2022 at 10h02)

Grafs :
As you can see, my CPU contention on my Netbox VM is very bad since I upgraded Netbox from v3.3.8 to v3.4.0.

• Netbox VM (15/12/2022)
  

• Netbox VM (16/12/2022). Note that I've tried to add 2 vCPU and 4Go of RAM
  

• BDD VM (from 15/12/2022 to 16/12/2022)
  

What king of debug can I enable on my Netbox VM to try to troubleshoot this problem ?

Regards,

[kkthxbye-code]

It's related to both what you said, that components created during devicetype initialization didn't get changelogs before and the added search indexing.

Moving indexing and changelogging to the background is being investigated here: #11202

What king of debug can I enable on my Netbox VM to try to troubleshoot this problem ?

There's nothing really to debug here, it's a simple issue of the fact that importing your 284 devices before included the following database operations:

284 devices
284 changelog entries for devices
284 * 32 = 9088 device components

Total: 9656 objects created

Now:

284 devices
284 changelog entries for devices
284 * 32 = 9088 device components
284 * 32 = 9088 changelog entries for components
284 * 32 * 7 = up to 63616 search index entries (name, label, mac_address, wwn, description, mtu, speed) - most likely we would be at less as not all fields are populated.

Total: Minimum 27832 -> 82360 objects created.

Note that even if we move these to the background, the CPU usage will still be present, but the HTTP requests will not be impacted.

[Djo91]

Moving indexing and changelogging to the background is being investigated here: #11202

Thank you for this information.

Note that even if we move these to the background, the CPU usage will still be present, but the HTTP requests will not be impacted.

Okay, perfect, thx.

[ZPrimed]

@Djo91
I'm not sure what virtualization platform you are using, or what your physical host is, but I'm going to pass along some wisdom that I've learned over the years with VMs... If you already know all of this, I apologize, but maybe it can benefit someone else at least.

Imagine that you have a server with 8 physical CPU cores. You create two VMs, Alex and Bob, each of them with 4 vCPU (or one vCPU with 4 virtual cores... the behavior should be about the same AFAIK).

A given virtual machine can only be scheduled to run (execute code) in a timeslice if there are enough available physical cores to meet its requirements.

So, with two 4-core VMs and an 8-core host, most of the time both of the VMs will be able to run simultaneously. (The exception is when/if the physical host needs CPU time for something, like interrupts, networking, disk, etc.)

Now, imagine that you change the Alex VM to have 6 vCPU/vCores. In this scenario, Alex and Bob cannot be scheduled simultaneously anymore. Alex always requires 6 cores in order to run; Bob only requires 4, but your host server only has 8 cores. So instead of both VMs running simultaneously, you now have constant switching in and out between Alex and Bob, with each VM having to "take turns" before there are enough cores available. When Bob runs, 4 of the cores are basically sitting idle, and when Alex runs, two of them are (unless the physical host needs to use them for interrupts/networking/etc).

This is what is normally represented as "CPU contention," or in some hypervisors, "CPU ready" or "ready %" or something along those lines. When you see this percentage increasing, it means that the physical host doesn't have enough cpu cores available to schedule the VM, so the VM is sitting "ready" but not actually executing any code.

Lesson to be learned here: run your VMs with as few (virtual) cores/CPUs as possible. In many cases, a VM only needs a single vCPU/vCore. Unless a VM is constantly sitting at or near 100% usage within the VM, you don't necessarily gain much from giving it more vCPUs. I place a lot of blame for this trend on 3rd party software vendors who have "requirements" of "4 core CPUs" and throw fits that their VMs are not configured this way... In my experience, much of the time the vendor is dead wrong. ;)

As @kkthxbye-code said, there are a lot of database operations happening in your scenario, so there are likely I/O waits happening (unless your storage is all-flash, and even then there may still be some delays). Plus, go back to what I said above about how VM scheduling works -- if your physical host only has 8 cores, you give 6 of them to your Netbox VM... that leaves 2 cores free to handle I/O at the hypervisor level, unless you also have other VMs on the same host, in which case the other VMs will be wanting cores + having their own disk i/o as well.