Answers checklist.
General issue report
It seems that esp_modem/esp_modem_usb_dte examples initialize HS PHY by default.
I tried to disable auto phy setup by usb_host_install() with host_config.skip_phy_setup = true and initialize PHY manually before calling usb_host_install():
usb_phy_handle_t phy_hdl = NULL;
usb_phy_config_t phy_conf = {
.controller = USB_PHY_CTRL_OTG,
.target = USB_PHY_TARGET_INT,
.otg_mode = USB_OTG_MODE_HOST,
.otg_speed = USB_PHY_SPEED_FULL
};
esp_err_t ret = usb_new_phy(&phy_conf, &phy_hdl);
but this did not work as expected.