Asus GL552VW

[rob-mur]

Hi there

First - thanks so much for this repo, it's super useful to have all this information in one place! Below is a description of what happened as I tried to get this running, hopefully it will help! Secondly, apologies in advance for the long post, I’ve tried to include everything I ran into in order to find areas where the documentation could be improved.
I've had a go at using it on a clean install of Fedora 35 with an Asus GL552VW. That is an intel i7-6700HQ and an Nvidia 960M.
I'd recommend either pulling the windows-unattended-install branch into main OR adding a big flag on the master readme that this is the branch to currently use, as the master branch currently doesn't work (as referenced in other issues).

In order to get my dGPU to appear in the check script I had to set kernel parameter intel_iommu=on. It would be handy to add this to the readme (both that it needs to be done, and how to set a kernel parameter for newcomers).

I then ran into an issue where spice would give a connection refused, leading to a system hang as you don't start the vm quick enough because the default port in the config script had changed to 5904, but the docs in master still say 5900. I then realised that this is again just due to docs, as I had the git page on master still (my fault).
I then ran "auto" as is referenced in the unattended-win-install branch readme. This failed because SERVICE_NAME was ambiguous and so the service to run on next boot wasn't set - this error occured in /scripts/utils/manager-specific/service/systemd. I tracked this down to line 187 of setup.sh where the command line arguments were in the wrong order at least for my setup. I switched them around and reran but the service is just set to run auto again so nothing happened.
From here I then attempted to run the stages individually to try and work out what was happening. In this case the main issue was that the install was failing because the default vm-file names (such as the windows 10 iso), don't match the file names as downloaded by default in the setup script. This means that the install script doesn't work, but still unloads your dGPU leading to a crash. It was very helpful to have the dry-run parameter to work this out so thanks for that! It would be helpful if the script detected a failed install and didn't unload the gpu however.
The first reason I could see for a failure of the install script was qemu-img not found. I then installed that package separately, perhaps an early step failed to grab it.
The next error became “failed to connect to the hypervisor” from qemu. What seemed to work is installing libvirt from Discover and then importantly rebooting.
This is where I’ve stopped for now as I’m on the edges of what I know without significant further reading. The bottom line is I can’t create a vGPU due to a linux error. The remaining errors that cause the install to fail can be found in the output of my install dry run below. Any pointers of next steps to troubleshoot would be appreciated! I will try and investigate this further when I get more time.

Action: install
> Start mode: qemu
> Using network mode bridged...
> Using MAC address: 52:54:BE:EF:C0:0E...
> Using 7 CPU cores...
> Using 5G of RAM...
> Using a virtual OS drive...
> Removing old virtual disk...
> Creating a virtual disk for the VM...
> Virtual OS drive has 40G of storage.
> Bumblebee is not available...
> Not using SMB share...
> Using dGPU passthrough... 
> dGPU is: 3D controller: NVIDIA Corporation GM107M [GeForce GTX 960M] (rev a2)
> Retrieving and parsing DGPU IDs...
> Not using DGPU vBIOS override...
modprobe: ERROR: could not insert 'kvmgt': No such device
modprobe: FATAL: Module vfio-mdev not found in directory /lib/modules/5.14.18-300.fc35.x86_64
> Creating a vGPU for mediated iGPU passthrough...
bash: line 1: /sys/bus/pci/devices/0000:00:02.0/mdev_supported_types/*/create: No such file or directory
> [Error] Failed creating a vGPU. (You can try again. If you still get this error, you have to reboot. This seems to be a bug in Linux.)
> Continuing without mediated iGPU passthrough...
> Loading display-mode-4 plugin...
> Using spice on port 5904...
> Not using QXL...
> Not using Looking Glass...
> Using fake battery...
> Creating fresh OVMF_VARS copy for this VM...
> Not using patched OVMF...
> Not using USB passthrough...
> Using virtual input method 'virtio' for keyboard/mouse input...
> Using RDP...
> Deleting VM if it already exists...
> [Background task] Starting RDP autoconnect...
> Generating qemu-system-x86_64 command (dry-run)...

sudo qemu-system-x86_64 \
  -name MBPT_WindowsVM \
  -machine type=q35,accel=kvm \
  -global ICH9-LPC.disable_s3=1 \
  -global ICH9-LPC.disable_s4=1 \
  -enable-kvm \
  -cpu host,kvm=off,hv_vapic,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vendor_id=12alphanum \
  -mem-prealloc \
  -rtc clock=host,base=localtime \
  -nographic \
  -serial none \
  -parallel none \
  -boot menu=on \
  -boot once=d \
  -k en-us \
  -device ich9-intel-hda \
  -device hda-output \
  -device pci-bridge,addr=12.0,chassis_nr=2,id=head.2 \
  -net nic,model=e1000,macaddr=52:54:BE:EF:C0:0E \
  -net bridge,br=virbr0 \
  -smp 7 \
  -m 5G \
  -drive file=/home/rmurphy/MobilePassThrough/vm-files/windows10.iso,index=1,media=cdrom \
  -drive file=/home/rmurphy/MobilePassThrough/vm-files/mobile-passthrough-helper.iso,index=2,media=cdrom \
  -drive id=disk0,if=virtio,cache.direct=on,if=virtio,aio=native,format=raw,file=/home/rmurphy/MobilePassThrough/vm-files/MBPT_WindowsVM.img \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=pci.1 \
  -device vfio-pci,host=01:00.0,bus=pci.1,addr=00.0,x-pci-sub-device-id=0x1c5d,x-pci-sub-vendor-id=0x1043,multifunction=on,rombar=0 \
  -spice port=5904,addr=127.0.0.1,disable-ticketing \
  -acpitable file=/home/rmurphy/MobilePassThrough/vm-files/fake-battery.aml \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.fd \
  -drive if=pflash,format=raw,file=/home/rmurphy/MobilePassThrough/vm-files/OVMF_VARS_VM.fd \
  -usb \
  -device virtio-keyboard-pci,bus=head.2,addr=03.0,display=video.2 \
  -device virtio-mouse-pci,bus=head.2,addr=04.0,display=video.2

Cleaning up...
> Unbinding device ' from the vfio-pci driver, then bind it back to its original driver...
driverctl: no such device: 0000:
> [Error] Seems like the installation failed...

[rob-mur]

Just to add to this, the first error here (kvmgt) was down to the kernel parameters not being persisted. My gut feeling without digging into it is that update-grub has been used in the script which doesn't work on fedora, you need

grub2-mkconfig -o "$(readlink -e /etc/grub2.conf)"

This still leaves the fatal vfio-mdev which I'll keep looking into.

[rob-mur]

After doing the above grub update with a reboot I then get a different set of compatibility information, see below:

[OK] The IOMMU kernel parameters are set.
[OK] VT-X / AMD-V virtualization is enabled in the UEFI.
[OK] VT-D / IOMMU is enabled in the UEFI.
[Success] GPU with ID '01:00.0' could be passed through to a virtual machine!
[Problem] Other devices have been found in the IOMMU group of the GPU with the ID '00:02.0'. Depending on the devices, this could make it impossible to pass this GPU through to a virtual machine!
The devices found in this GPU's IOMMU Group are:
IOMMU Group 10 00:1f.2 Memory controller [0580]: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller [8086:a121] (rev 31)
IOMMU Group 12 03:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTL8411B PCI Express Card Reader [10ec:5287] (rev 01)
IOMMU Group 12 03:00.1 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 12)
IOMMU Group 4 00:14.2 Signal processing controller [1180]: Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem [8086:a131] (rev 31)
[Info] It might be possible to get it to work by putting the devices in different slots on the motherboard and/or by using the ACS override patch. Otherwise you'll probably have to get a different motherboard. If you're on a laptop, there is nothing you can do as far as I'm aware. Although it would theoretically be possible for ACS support for laptops to exist. TODO: Find a way to check if the current machine has support for that.
[Success] There are 1 GPU(s) in this system that could be passed through to a VM!

Is Compatible?  Name                       IOMMU_GROUP  PCI Address
No              HD Graphics 530            13
2               pci@0000:00:02.0
Yes             GM107M [GeForce GTX 960M]  1            pci@0000:01:00.0

[OK] You have GPUs that are not in the same IOMMU group. At least one of these could be passed through to a VM and at least one of the remaining ones could be used for the host system.
[Info] This system is probably MUX-less. (The connection between the GPU(s) and the [internal display]/[display outputs] is not multiplexed.)
If you found a notebook that appears to be GPU passthrough compatible, please open an issue on Github and let me know.
You may now proceed and run './mbpt.sh configure' if you haven't already.

I'm reading this as "only the dGPU can be passed to a VM, not the integrated". So I changed to config to put false for passing through the iGPU but unfortunately this doesn't improve the situation.

[rob-mur]

After some further digging, I think it's because the helper iso hasn't been generated. Looking at the documentation this should be with the iso argument but then looking at the actual mbpt.sh file I think you need to write helper-iso. Running with helper-iso generates the required file and clears this specific error.

[rob-mur]

A further test I've perfomed is removing the vfio relevant commands from qemu and checking windows would boot if given the chance, which it does (I can see it with spice).

This indicates that the issue is more than disabling the dGPU in the host bricks fedora rather than it not actually being able to passthrough. Will investigate later.

[rob-mur]

After narrowing down the hang it's on driverctl set-override. Running driverctl list-overrides returns

driverctl: No overridable devices found. Kernel too old?

Unsure if this is just a fedora 35 issue, but will keep investigating when I can.

[rob-mur]

Got a little further by installing the proprietary Nvidia drivers but that just opens an extra can of worms with the system not booting, and bumblebee setup being dodgy (as you well mention in the guide!)

I think I'm done trying to get this to work on Fedora and chalk this up as a loss unless anyone has a good idea for a next troubleshooting step.

[T-vK]

Hey, thanks for the feedback. I agree that I really need to merge the windows-unattended-install into the master. Considering that the master branch has so many issues for Ubuntu users there isn't really a good reason not to do it anymore.
intel_iommu=on should automatically be set by mbpt.sh when running during the setup (i.e. when running mbpt.sh setup). It does require a reboot though.

It's possible that some adjustments have to be made in order to support Fedora 35. I have only tested that branch on Fedora 34 so far. If that still doesn't work for you and you have a 64GB USB stick, it might be worth it to try the Live CD image which you can build by running ./mbpt.sh live build.

The helper ISO should be generated automatically whenever necessary. This happens during in the setup (mbpt.sh setup) as well.

MobilePassThrough/scripts/main/setup.sh

Line 164 in 4a1627a

generateHelperIso

It shouldn't be necessary to install a specific Nvidia driver. If you have either Nouveau or the proprietary one installed you should be fine. Bumblebee is completely untested in regards to the new branch, I advise against setting it up before getting it to work without it.

In regards to driverctl I'm not sure. Often times the only way to get a GPU (driver) to behave normally again is a reboot.
There still are a lot of bugs in Linux or in the GPU drivers (this includes Intel, AMD, Nvidia and Nouveau) causing all sorts of weird behavior when binding or unbinding a driver.

[rob-mur]

Thanks for the response!

I have a feeling unbinding the driver just isn't happy on Fedora 35 yet. Under the nouveau driver I got a full hang, and under the proprietary driver I got a shell hang. Other people have experienced something similar (although not Fedora 35 specifically) https://forum.level1techs.com/t/problem-cant-use-driverctl-overrides-on-nvidia-driver/176777

I think what I will do is use the live environment and see if I can get that working. If so I think it would probably be a good idea for those to be the main focus of what's provided? That way it's always a clean install and all variables are controlled.

[T-vK]

The live version isn't really meant to be installed though. Its main purpose is to provide a fully automated fool-proof way of testing if GPU passthrough works on a given device or not.
Theoretically it shouldn't make a difference if you use the live image or a fresh install of Fedora 34.
The goal is to eventually support most distros or at least to make it easy to add support for new distros.

[rob-mur]

That makes sense. Assuming I get it working on Fedora 34 (both live and fresh), do you have a priority list in mind for which distros you'd like to support? I'm happy to just throw stuff at this laptop for testing.

[T-vK]

I think it would be Fedora 34, Fedora 35 and then Ubuntu 21. I can't give you a date on when all this will happen though.

[rob-mur]

Sounds good. Currently running into issues just building the live iso as there are python modules not installed that are required. Namely it seems to assume you have pip and then also the errors package. Potentially need to look into probing for this ahead of time/ensuring there's a venv setup for python, but in the first instance I'm just seeing if I can install the missing packages.

Full trace if you like

 [Skipped] Executable dependencies are already installed.
> Downlaoding and installing livecd-tools...
Cloning into 'livecd-tools'...
remote: Enumerating objects: 35, done.
remote: Counting objects: 100% (35/35), done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 35 (delta 2), reused 17 (delta 1), pack-reused 0
Receiving objects: 100% (35/35), 170.84 KiB | 3.22 MiB/s, done.
Resolving deltas: 100% (2/2), done.
<string>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
<string>:1: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
pod2man --section=8 --release="livecd-tools 28.3" --center "LiveCD Tools" docs/livecd-creator.pod > docs/livecd-creator.8
pod2man --section=8 --release="livecd-tools 28.3" --center "LiveCD Tools" docs/livecd-iso-to-disk.pod > docs/livecd-iso-to-disk.8
/usr/bin/install -c -D tools/livecd-creator /usr/bin/livecd-creator
ln -sf livecd-creator /usr/bin/image-creator
/usr/bin/install -c -D tools/liveimage-mount /usr/bin/liveimage-mount
/usr/bin/install -c -D tools/livecd-iso-to-disk.sh /usr/bin/livecd-iso-to-disk
/usr/bin/install -c -D tools/livecd-iso-to-pxeboot.sh /usr/bin/livecd-iso-to-pxeboot
/usr/bin/install -c -D tools/editliveos /usr/bin/editliveos
/usr/bin/install -c -D tools/mkbiarch /usr/bin/mkbiarch
/usr/bin/install -c -m 644 -D AUTHORS /usr/share/doc/livecd-tools/AUTHORS
/usr/bin/install -c -m 644 -D COPYING /usr/share/doc/livecd-tools/COPYING
/usr/bin/install -c -m 644 -D README /usr/share/doc/livecd-tools/README
/usr/bin/install -c -m 644 -D HACKING /usr/share/doc/livecd-tools/HACKING
mkdir -p /usr/share/livecd-tools/
mkdir -p //usr/lib/python3.10/site-packages/imgcreate
/usr/bin/install -c -m 644 -D imgcreate/*.py //usr/lib/python3.10/site-packages/imgcreate/
/usr/bin/python -c "import compileall as c; c.compile_dir('//usr/lib/python3.10/site-packages/imgcreate', force=1)"
Listing '//usr/lib/python3.10/site-packages/imgcreate'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/__init__.py'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/creator.py'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/debug.py'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/dnfinst.py'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/errors.py'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/fs.py'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/kickstart.py'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/live.py'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/util.py'...
/usr/bin/python -O -c "import compileall as c; c.compile_dir('//usr/lib/python3.10/site-packages/imgcreate', force=1)"
Listing '//usr/lib/python3.10/site-packages/imgcreate'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/__init__.py'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/creator.py'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/debug.py'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/dnfinst.py'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/errors.py'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/fs.py'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/kickstart.py'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/live.py'...
Compiling '//usr/lib/python3.10/site-packages/imgcreate/util.py'...
mkdir -p /usr/share/man/man8
/usr/bin/install -c -m 644 -D docs/*.8 /usr/share/man/man8
/usr/bin/sed -i "s:#!/usr/bin/python:#!/usr/bin/python:g" /usr/bin/livecd-creator
/usr/bin/sed -i "s:#!/usr/bin/python:#!/usr/bin/python:g" /usr/bin/liveimage-mount
/usr/bin/sed -i "s:#!/usr/bin/python:#!/usr/bin/python:g" /usr/bin/editliveos
/usr/bin/sed -i "s:#!/usr/bin/python:#!/usr/bin/python:g" /usr/bin/mkbiarch
sudo: pip3: command not found
> Downloading Fedora ISO...
--2021-11-24 14:48:41--  https://download.fedoraproject.org/pub/fedora/linux/releases/34/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-34-1.2.iso
Resolving download.fedoraproject.org (download.fedoraproject.org)... 2a05:d01c:c6a:cc01:269:da52:9ae1:43e6, 2a05:d014:10:7803:f774:4d7c:e277:a457, 2001:4178:2:1269::fed2, ...
Connecting to download.fedoraproject.org (download.fedoraproject.org)|2a05:d01c:c6a:cc01:269:da52:9ae1:43e6|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/fedora/linux/releases/34/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-34-1.2.iso [following]
--2021-11-24 14:48:41--  https://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/fedora/linux/releases/34/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-34-1.2.iso
Resolving www.mirrorservice.org (www.mirrorservice.org)... 2001:630:341:12::184, 212.219.56.184
Connecting to www.mirrorservice.org (www.mirrorservice.org)|2001:630:341:12::184|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2007367680 (1.9G) [application/x-iso9660-image]
Saving to: ‘/home/rmurphy/MobilePassThrough/live-iso-files/Fedora-Workstation-Live-x86_64-34-1.2.iso.part’

/home/rmurphy/MobilePassThrough/live-iso-files/ 100%[=======================================================================================================>]   1.87G  8.14MB/s    in 4m 23s  

2021-11-24 14:53:04 (7.29 MB/s) - ‘/home/rmurphy/MobilePassThrough/live-iso-files/Fedora-Workstation-Live-x86_64-34-1.2.iso.part’ saved [2007367680/2007367680]

> Rebuilding the ISO adding kernel parameters and some files...
Traceback (most recent call last):
  File "/usr/bin/editliveos", line 74, in <module>
    from imgcreate.fs import *
  File "/usr/lib/python3.10/site-packages/imgcreate/__init__.py", line 19, in <module>
    from imgcreate.live import *
  File "/usr/lib/python3.10/site-packages/imgcreate/live.py", line 35, in <module>
    from imgcreate.fs import *
  File "/usr/lib/python3.10/site-packages/imgcreate/fs.py", line 35, in <module>
    from imgcreate.util import *
  File "/usr/lib/python3.10/site-packages/imgcreate/util.py", line 23, in <module>
    from errors import *
ModuleNotFoundError: No module named 'errors'
mv: cannot stat '/home/rmurphy/MobilePassThrough/live-iso-files/mbpt-*.iso': No such file or directory

(btw I'm doing this on a fresh install of 35 as I had the live usb ready to go, so my previous work isn't causing these errors)

[rob-mur]

If I install python-imgcreate https://yum-info.contradodigital.com/view-package/epel/python-imgcreate/ then the error simplifies a bit, but still unclear what to do.

As a next troubleshooting step I'm going to fresh install Fedora 34 to isolate 35 as a source of error. Final trace below:

> [Skipped] Executable dependencies are already installed.
> [Skipped] livecd-tools already installed.
> [Skipped] Fedora ISO already downloaded.
> Rebuilding the ISO adding kernel parameters and some files...

Source image at '/home/rmurphy/MobilePassThrough/live-iso-files/Fedora-Workstation-Live-x86_64-34-1.2.iso'

LiveOS edit has ended.
Process duration: 00:00:02
Traceback (most recent call last):
  File "/usr/bin/editliveos", line 2729, in <module>
    sys.exit(main())
  File "/usr/bin/editliveos", line 2601, in main
    editor._pre_mount(args.liveos, args.rootfsimg, args.overlay)
  File "/usr/bin/editliveos", line 745, in _pre_mount
    self._LoopImageCreator__fstype = losm.imgloop.fstype
AttributeError: 'LoopbackDisk' object has no attribute 'fstype'
mv: cannot stat '/home/rmurphy/MobilePassThrough/live-iso-files/mbpt-*.iso': No such file or directory

[T-vK]

Seems like pip3 is missing in the dependencies.sh.

MobilePassThrough/requirements.sh

Line 64 in 1527df2

EXEC_DEPS_LIVE_ISO+=("wget" "xorriso" "mksquashfs" "unsquashfs") # wget xorriso squashfs-tools

I'm not exactly sure about that AttributeError. Seems like an issue with editliveos. Not sure why you're getting it.
It might be worth trying to delete the livecd-tools directory from the thirdparty directory, so that the generate-live-iso.sh will try to reinstall it. Maybe the installation wasn't completed because of the missing pip3 and now it thinks the installation is complete just because it sees the livecd-tools directory.

[rob-mur]

So I'm now on a fresh Fedora 34 install, which runs into the same issue.

I then added pip3 as you instructed and deleted the livecd-tools directory. After re-running this does clear up the pip3 bug, but the module not found still remains. I haven't explicitly installed python-imgcreate this time but I imagine it would just do the same thing. Is that the correct dependency to use? If so I'll try it/dig again later on.