AGX Orin Industrial failing to flash because of the Ecc Scrub taking too long?

[conradmenchine]

Hi all, I'm trying to bing up the AGX Orin Industrial on a ConnectTech Forge board (I previously have gotten the non-industrial module working so I'm hopeful this should be minimal changes) ... After chaning the device tree and flashvars a bit, I'm able to get tegraflash to run, but tegrarcm_v2 seems to be failing due to a timeout. I know that usually these "timeouts" are really the bootloader crashing, but in this case, I believe that it really is a timeout caused by the ecc scrub taking too long (over 5 seconds!)

The full output is attached below, but here's the part that I'm looking at:

[0025.114] I> Task: Dram Ecc scrub
[0025.117] I> DRAM ECC Scrub Mode: full
[0031.026] I> SDRAM scrub successful
[0031.030] I> Task: DRAM alias check

The doflash.sh script seems to drop out and report an error about one second before the ECC scrub completes. Although this could possibly be the result of un-flushed output because from my manual console-watching, there is about a 4 second delay between the "Sending blob" and "Error" lines being printed... But if this is actually 0.1 seconds, then it's failing about 5 seconds before the ecc scrub is complete:

[   5.2691 ] tegrarcm_v2 --chip 0x23 0 --pollbl --download bct_mem mem_rcm_sigheader.bct.encrypt --download blob blob.bin
[   5.2694 ] BL: version 1.4.0.4-t234-54845784-e89ea9bc last_boot_error: 0
[   5.2700 ] Sending bct_mem
[   5.2701 ] Sending blob
[   5.3465 ] ERROR: might be timeout in USB write.
Error: Return value 3

Anyway, I may be entirely barking up the wrong tree, but I could definitely use another set of eyes. Specifically, has anyone else flashed an Orin Industrial recently, and if so do you know how long the ECC scrub nominally takes? Thanks!
Here it the output from the uart on the tegra:

ýý
[0010.482] I> MB1 (version: 1.4.0.4-t234-54845784-e89ea9bc)
[0010.488] I> t234-A01-1-Silicon (0x12347) Prod
[0010.492] I> Boot-mode : BPMP Diagnostic
[0010.496] I> Entry timestamp: 0x00000000
[0010.500] I> last_boot_error: 0x0
[0010.503] I> BR-BCT: preprod_dev_sign: 0
[0010.506] I> rst_source: 0x0, rst_level: 0x0
[0010.511] I> Task: Initialize MB2 params
[0010.515] I> MB2-params @ 0x40060000
[0010.518] I> Task: Crypto init
[0010.521] I> Task: Perform MB1 KAT tests
[0010.525] I> Task: NVRNG health check
[0010.529] I> NVRNG: Health check success
[0010.532] I> Task: Secure debug controls
[0010.536] I> Task: strap war set
[0010.539] I> Task: Program NV master stream id
[0010.544] I> Task: Verify boot mode
[0010.549] I> Task: Alias fuses
[0010.553] W> FUSE_ALIAS: Fuse alias on production fused part is not supported.
[0010.560] I> Task: Print SKU type
[0010.563] I> FUSE_OPT_CCPLEX_CLUSTER_DISABLE = 0x00000000
[0010.568] I> FUSE_OPT_GPC_DISABLE = 0x00000000
[0010.573] I> FUSE_OPT_TPC_DISABLE = 0x00000000
[0010.577] I> FUSE_OPT_DLA_DISABLE = 0x00000000
[0010.581] I> FUSE_OPT_PVA_DISABLE = 0x00000000
[0010.586] I> FUSE_OPT_NVENC_DISABLE = 0x00000000
[0010.590] I> FUSE_OPT_NVDEC_DISABLE = 0x00000000
[0010.595] I> FUSE_OPT_FSI_DISABLE = 0x00000000
[0010.599] I> FUSE_OPT_EMC_DISABLE = 0x00000000
[0010.603] I> FUSE_BOOTROM_PATCH_VERSION = 0x7
[0010.607] I> FUSE_PSCROM_PATCH_VERSION = 0x7
[0010.611] I> FUSE_OPT_ADC_CAL_FUSE_REV = 0x2
[0010.615] I> FUSE_SKU_INFO_0 = 0x90
[0010.619] I> FUSE_OPT_SAMPLE_TYPE_0 = 0x3 PS
[0010.623] I> FUSE_PACKAGE_INFO_0 = 0x2
[0010.627] I> SKU: Prod
[0010.629] I> Task: Boost clocks
[0010.632] I> Initializing PLLC2 for AXI_CBB.
[0010.636] I> AXI_CBB : src = 35, divisor = 0
[0010.640] I> Task: Voltage monitor
[0010.643] I> VMON: Vmon re-calibration and fine tuning done
[0010.649] I> Task: UPHY init
[0010.652] W> UPHY: UPHY lane info table is empty in MB1 BCT.
[0010.657] I> Task: Boot device init
[0010.661] I> Boot_device: RCM
[0010.663] I> USB configuration success
[0010.667] I> Task: TSC init
[0010.670] I> Task: Enable WDT 5th expiry
[0010.674] I> Task: I2C register
[0010.677] I> Task: Set I2C bus freq
[0010.680] I> Task: Reset FSI
[0010.683] I> Task: Enable clock-mon
[0010.697] I> FMON: Fmon re-programming done
[0010.701] I> Task: Mapper init
[0010.704] I> Task: MB1 fixed firewalls
[0010.710] W> Firewall readback mismatch
[0010.715] I> Task: Load MB2/Applet/FSKP
[0010.719] I> Loading MB2 Applet
[0010.722] I> Slot: 0
[0010.724] I> Binary[21] block-352321536 (partition size: 0x50000)
[0010.730] I> Binary name: MB2-Applet
[0010.734] I> Size of crypto header is 8192
[0010.738] I> Size of crypto header is 8192
[0010.742] I> BCH of MB2-Applet read from storage
[0010.746] I> BCH address is : 0x4004e000
[0010.750] I> MB2-Applet header integrity check is success
[0010.755] I> Binary magic in BCH component 0 is MB2A
[0010.760] I> component binary type is 21
[0010.764] I> Size of crypto header is 8192
[0010.776] I> MB2-Applet binary is read from storage
[0010.781] I> MB2-Applet binary integrity check is success
[0010.786] I> Binary MB2-Applet loaded successfully at 0x40000000 (0x41b40)
[0010.793] I> Task: Prepare MB2 params
[0010.797] I> Task: Misc NV security settings
[0010.801] I> NVDEC sticky bits programming done
[0010.806] I> Successfully powergated NVDEC
[0010.810] I> Task: Disable/Reload WDT
[0010.813] I> Task: Program misc carveouts
[0010.817] I> Task: Disable SCPM/POD reset
[0010.821] I> SLCG Global override status := 0x0
[0010.825] I> MB1: MSS reconfig completed
[0010.829] W> Firewall readback mismatch
[0010.833] W> Firewall readback mismatch
[0010.837] W> Firewall readback mismatch
[0010.840] W> Firewall readback mismatch
[0010.844] W> Firewall readback mismatch
[0010.848] W> Firewall readback mismatch
[0010.851] W> Firewall readback mismatch
[0010.855] W> Firewall readback mismatch
[0010.859] W> Firewall readback mismatch
[0010.862] W> Firewall readback mismatch
I> Applet (version: 0.0.0.0-t234-54845784-d30edb11)
I> t234-A01-1-Silicon (0x12347)
I> Emulation:
I> Entry timestamp: 0xffffffff
I> Task: Pinmux init (0x4000062d)
I> Task: Boot device init (0x40001321)
I> Applet do storage init
I> Boot_device: QSPI_FLASH instance: 0
I> Qspi clock source : clk_m
I> QSPI-0l initialized successfully
I> sdmmc DDR50 mode
I> sdmmc bdev is already initialized
E> Unknown device 7
E> Unknown device 8
I> Task: Partition Manager Init (0x40002d61)
W> Cannot find any partition table for 00030000
 > PARTITION_MANAGER: Failed to publish partition.
W> Cannot find any partition table for 00000003
 > PARTITION_MANAGER: Failed to publish partition.
W> Cannot find any partition table for 00010003
 > PARTITION_MANAGER: Failed to publish partition.
I> Task: I2C register (0x400006a5)
I> Task: Register exit handlers (0x400005f5)
I> Task: Enter 3p server (0x40001925)
I> USB configuration success
I> Populate chip info
I> RAM_CODE 0x4000041
I> Populate eeprom info
I> Populate eeprom info for module cvm
I> dump bct
I> Rebooting : reboot-recovery


ýý
[0023.768] I> MB1 (version: 1.4.0.4-t234-54845784-e89ea9bc)
[0023.773] I> t234-A01-1-Silicon (0x12347) Prod
[0023.778] I> Boot-mode : Coldboot
[0023.781] I> Entry timestamp: 0x00000000
[0023.785] I> last_boot_error: 0x0
[0023.788] I> BR-BCT: preprod_dev_sign: 0
[0023.792] I> rst_source: 0xb, rst_level: 0x1
[0023.796] I> Task: SE error check
[0023.799] I> Task: Bootchain select WAR set
[0023.803] I> Task: Enable SLCG
[0023.806] I> Task: CRC check
[0023.809] I> Skip FUSE records CRC check as records_integrity fuse is not burned
[0023.816] I> Task: Initialize MB2 params
[0023.820] I> MB2-params @ 0x40060000
[0023.823] I> Task: Crypto init
[0023.826] I> Task: Perform MB1 KAT tests
[0023.830] I> Task: NVRNG health check
[0023.834] I> NVRNG: Health check success
[0023.838] I> Task: MSS Bandwidth limiter settings for iGPU clients
[0023.844] I> Task: Enabling and initialization of Bandwidth limiter
[0023.850] I> No request to configure MBWT settings for any PC!
[0023.855] I> Task: Secure debug controls
[0023.859] I> Task: strap war set
[0023.862] I> Task: Initialize SOC Therm
[0023.866] I> Task: Program NV master stream id
[0023.870] I> Task: Verify boot mode
[0023.876] I> Task: Alias fuses
[0023.879] W> FUSE_ALIAS: Fuse alias on production fused part is not supported.
[0023.887] I> Task: Print SKU type
[0023.890] I> FUSE_OPT_CCPLEX_CLUSTER_DISABLE = 0x00000000
[0023.895] I> FUSE_OPT_GPC_DISABLE = 0x00000000
[0023.899] I> FUSE_OPT_TPC_DISABLE = 0x00000000
[0023.903] I> FUSE_OPT_DLA_DISABLE = 0x00000000
[0023.908] I> FUSE_OPT_PVA_DISABLE = 0x00000000
[0023.912] I> FUSE_OPT_NVENC_DISABLE = 0x00000000
[0023.916] I> FUSE_OPT_NVDEC_DISABLE = 0x00000000
[0023.921] I> FUSE_OPT_FSI_DISABLE = 0x00000000
[0023.925] I> FUSE_OPT_EMC_DISABLE = 0x00000000
[0023.930] I> FUSE_BOOTROM_PATCH_VERSION = 0x7
[0023.934] I> FUSE_PSCROM_PATCH_VERSION = 0x7
[0023.938] I> FUSE_OPT_ADC_CAL_FUSE_REV = 0x2
[0023.942] I> FUSE_SKU_INFO_0 = 0x90
[0023.945] I> FUSE_OPT_SAMPLE_TYPE_0 = 0x3 PS
[0023.949] I> FUSE_PACKAGE_INFO_0 = 0x2
[0023.953] I> SKU: Prod
[0023.955] I> Task: Boost clocks
[0023.958] I> Initializing PLLC2 for AXI_CBB.
[0023.962] I> AXI_CBB : src = 35, divisor = 0
[0023.967] I> Task: Voltage monitor
[0023.970] I> VMON: Vmon re-calibration and fine tuning done
[0023.975] I> Task: UPHY init
[0023.980] I> HSIO UPHY init done
[0023.983] W> Skipping GBE UPHY config
[0023.987] I> Task: Boot device init
[0023.990] I> Boot_device: RCM
[0023.993] I> USB configuration success
[0024.532] I> Task: TSC init
[0024.535] I> Task: Load membct
[0024.537] I> RAM_CODE 0x4000041
[0024.541] I> Loading MEMBCT
[0024.543] I> Slot: 0
[0024.545] I> Binary[1] block-0 (partition size: 0x40000)
[0024.550] I> Binary name: MEM-BCT-1
[0024.554] I> Size of crypto header is 8192
[0024.558] I> Size of crypto header is 8192
[0024.562] I> BCH of MEM-BCT-1 read from storage
[0024.566] I> BCH address is : 0x40050000
[0024.570] I> MEM-BCT-1 header integrity check is success
[0024.575] I> Binary magic in BCH component 0 is MEM1
[0024.580] I> component binary type is 1
[0024.586] I> MEM-BCT-1 binary is read from storage
[0024.591] I> MEM-BCT-1 binary integrity check is success
[0024.596] I> Binary MEM-BCT-1 loaded successfully at 0x40040000 (0xe580)
[0024.603] I> RAM_CODE 0x4000041
[0024.608] I> RAM_CODE 0x4000041
[0024.612] I> Task: Load Page retirement list
[0024.616] I> Task: SDRAM params override
[0024.620] I> Task: Save mem-bct info
[0024.623] I> Task: Carveout allocate
[0024.627] I> Update CCPLEX IST carveout from MB1-BCT
[0024.631] I> ECC region[0]: Start:0x80000000, End:0xe80000000
[0024.637] I> ECC region[1]: Start:0x0, End:0x0
[0024.641] I> ECC region[2]: Start:0x0, End:0x0
[0024.646] I> ECC region[3]: Start:0x0, End:0x0
[0024.650] I> ECC region[4]: Start:0x0, End:0x0
[0024.654] I> Non-ECC region[0]: Start:0x0, End:0x0
[0024.659] I> Non-ECC region[1]: Start:0x0, End:0x0
[0024.663] I> Non-ECC region[2]: Start:0x0, End:0x0
[0024.668] I> Non-ECC region[3]: Start:0x0, End:0x0
[0024.673] I> Non-ECC region[4]: Start:0x0, End:0x0
[0024.683] I> allocated(CO:44) base:0xe49800000 size:0x36800000 align: 0x100000
[0024.690] I> allocated(CO:31) base:0xe40000000 size:0x8000000 align: 0x8000000
[0024.698] I> allocated(CO:43) base:0xe3c000000 size:0x4000000 align: 0x200000
[0024.705] I> allocated(CO:39) base:0xe39e00000 size:0x2200000 align: 0x10000
[0024.712] I> allocated(CO:20) base:0xe36000000 size:0x2000000 align: 0x2000000
[0024.719] I> allocated(CO:24) base:0xe34000000 size:0x2000000 align: 0x2000000
[0024.726] I> allocated(CO:28) base:0xe32000000 size:0x2000000 align: 0x2000000
[0024.733] I> allocated(CO:29) base:0xe30000000 size:0x2000000 align: 0x2000000
[0024.740] I> allocated(CO:22) base:0xe48000000 size:0x1000000 align: 0x1000000
[0024.747] I> allocated(CO:41) base:0xe38e00000 size:0x1000000 align: 0x100000
[0024.754] I> allocated(CO:35) base:0xe38000000 size:0xe00000 align: 0x10000
[0024.761] I> allocated(CO:02) base:0xe49000000 size:0x800000 align: 0x800000
[0024.768] I> allocated(CO:03) base:0xe2f800000 size:0x800000 align: 0x800000
[0024.775] I> allocated(CO:06) base:0xe2f000000 size:0x800000 align: 0x800000
[0024.782] I> allocated(CO:56) base:0xe2e800000 size:0x800000 align: 0x200000
[0024.789] I> allocated(CO:07) base:0xe2e400000 size:0x400000 align: 0x400000
[0024.796] I> allocated(CO:33) base:0xe2e000000 size:0x400000 align: 0x200000
[0024.803] I> allocated(CO:23) base:0xe2de00000 size:0x200000 align: 0x200000
[0024.810] I> allocated(CO:01) base:0xe2dd00000 size:0x100000 align: 0x100000
[0024.817] I> allocated(CO:05) base:0xe2dc00000 size:0x100000 align: 0x100000
[0024.824] I> allocated(CO:08) base:0xe2db00000 size:0x100000 align: 0x100000
[0024.831] I> allocated(CO:09) base:0xe2da00000 size:0x100000 align: 0x100000
[0024.838] I> allocated(CO:15) base:0xe2d900000 size:0x100000 align: 0x100000
[0024.844] I> allocated(CO:17) base:0xe2d800000 size:0x100000 align: 0x100000
[0024.851] I> allocated(CO:27) base:0xe2d700000 size:0x100000 align: 0x100000
[0024.858] I> allocated(CO:42) base:0xe2d600000 size:0x100000 align: 0x100000
[0024.865] I> allocated(CO:54) base:0xe2d580000 size:0x80000 align: 0x80000
[0024.872] I> allocated(CO:34) base:0xe2d570000 size:0x10000 align: 0x10000
[0024.879] I> allocated(CO:72) base:0xe2d370000 size:0x200000 align: 0x10000
[0024.886] I> allocated(CO:46) base:0xe00000000 size:0x20000000 align: 0x20000000
[0024.893] I> allocated(CO:47) base:0xe2ce00000 size:0x400000 align: 0x200000
[0024.900] I> allocated(CO:48) base:0xe2d350000 size:0x20000 align: 0x10000
[0024.907] I> allocated(CO:69) base:0xe2d330000 size:0x20000 align: 0x10000
[0024.913] I> allocated(CO:49) base:0xe2d320000 size:0x10000 align: 0x10000
[0024.920] I> allocated(CO:50) base:0xe2d310000 size:0x10000 align: 0x10000
[0024.927] I> allocated(CO:52) base:0xe2d300000 size:0x10000 align: 0x10000
[0024.934] I> NSDRAM base: 0x80000000, end: 0xe2d370000, size: 0xdad370000
[0024.940] I> Task: Thermal check
[0024.944] I> Using min_chip_limit as min_tmon_limit
[0024.948] I> Using max_chip_limit as max_tmon_limit
[0024.953] I> BCT max_tmon_limit = 118
[0024.956] I> BCT min_tmon_limit = -43
[0024.960] I> BCT max_tmon_limit = 118
[0024.963] I> BCT min_tmon_limit = -43
[0024.967] I> SKU specific max_chip_limit = 125
[0024.971] I> SKU specific min_chip_limit = -43
[0024.976] I> BCT max_chip_limit = 118
[0024.979] I> BCT min_chip_limit = -43
[0024.982] I> enable_soctherm_polling = 0
[0024.986] I> max temp read = 32
[0024.989] I> min temp read = 32
[0024.992] I> Enabling thermtrip
[0024.995] I> Task: Update FSI SCR with thermal fuse data
[0025.000] I> Task: Enable WDT 5th expiry
[0025.004] I> Task: I2C register
[0025.007] I> Task: Set I2C bus freq
[0025.011] I> Task: Reset FSI
[0025.013] I> Task: Pinmux init
[0025.017] I> Task: Prod config init
[0025.020] I> Task: Pad voltage init
[0025.023] I> Task: Prod init
[0025.026] I> Task: Program rst req config reg
[0025.030] I> Task: Common rail init
[0025.034] I> DONE: Thermal config
[0025.037] W> DEVICE_PROD: module = 13, instance = 4 not found in device prod.
[0025.046] I> DONE: SOC rail config
[0025.049] W> PMIC_CONFIG: Rail: MEMIO rail config not found in MB1 BCT.
[0025.056] I> DONE: MEMIO rail config
[0025.059] I> DONE: GPU rail info
[0025.062] I> DONE: CV rail info
[0025.065] I> Task: Mem clock src
[0025.068] I> Task: Misc. board config
[0025.072] I> PMIC_CONFIG: Platform config not found in MB1 BCT.
[0025.078] I> Task: SDRAM init
[0025.081] I> MemoryType: 4 MemBctRevision: 8
[0025.088] I> MSS CAR: PLLM/HUB programming for MemoryType: 4 and MemBctRevision: 8
[0025.096] I> Encryption:   MTS: en, TX: en, VPR: en, GSC: en
[0025.107] I> SDRAM initialized!
[0025.109] I> SDRAM Size in Total 0xe00000000
[0025.114] I> Task: Dram Ecc scrub
[0025.117] I> DRAM ECC Scrub Mode: full
[0031.026] I> SDRAM scrub successful
[0031.030] I> Task: DRAM alias check
[0031.051] I> Task: Program NSDRAM carveout
[0031.055] I> NSDRAM carveout encryption is enabled
[0031.060] I> Program NSDRAM carveout
[0031.063] I> Task: Register checker
[0031.067] I> Task: Enable clock-mon
[0031.081] I> FMON: Fmon re-programming done
[0031.085] I> Task: Mapper init
[0031.088] I> Task: SC7 Context Init
[0031.092] W> SC7_CONTEXT_SAVE: SC7 ctx save is not supported.
[0031.098] I> Task: CCPLEX IST init
[0031.101] I> Task: CPU WP0
[0031.104] I> Loading MCE
[0031.106] I> Slot: 0
[0031.108] I> Binary[8] block-134217728 (partition size: 0x80000)
[0031.114] I> Binary name: MCE
[0031.117] I> Size of crypto header is 8192
[0031.121] I> Size of crypto header is 8192

Here is the output of doflash:

[dwalkes]

Specifically, has anyone else flashed an Orin Industrial recently, and if so do you know how long the ECC scrub nominally takes?

Not sure if someone will chime in on this one here but this might also be something I'd try to post to the Nvidia developer forum to see if someone could at least answer this part about whether a 5 second delay is normal. Would be interesting to know if nvidia's flashing tools work any differently, not sure whether that's possible/easy to do with this configuration though.

[conradmenchine]

Well, I figured out the issue- I was basing my MACHINE on the non-industrial jetson rather than the industrial one. After that change, it's no longer getting stuck in that part of the flash! My guess is that EMMC_BCT ?= "tegra234-p3701-0008-sdram-l4t.dts" is what fixed it.