🐛 [Bug] Lots of ops are failing due to fast refit

[zewenli98]

Bug Description

The PR #3167 changed the workflow of building engines to: 1) build a weight-stripped engine 2) refit the engine with gm weights.

So refitting is now used by default. While testing, I found fast refit is problematic in the CI tests of #3167.

I tried setting weight_name_map to None in the function _refit_single_trt_engine_with_gm(), i.e., fast refit -> slow refit. For some ops that works well, but for others that still fails due to AssertionError: XXX is not found in weight mapping like the CI tests

[zewenli98]

Close as NAB