Kernel 6.5-rc - vmmon fails..

[ayles2014]

Tested with kernel.org Kernel 6.5-rc1 to rc3.. reproduced on different systems, and with standard Fedora39 version of 6.5-rc2..
.............................
LD [M] /tmp/modconfig-Dj3fJq/vmmon-only/vmmon.o
MODPOST /tmp/modconfig-Dj3fJq/vmmon-only/Module.symvers
ERROR: modpost: "__pte_offset_map" [/tmp/modconfig-Dj3fJq/vmmon-only/vmmon.ko] undefined!
make[3]: *** [scripts/Makefile.modpost:144: /tmp/modconfig-Dj3fJq/vmmon-only/Module.symvers] Error 1
make[2]: *** [/usr/src/linux-6.5-rc3/Makefile:1984: modpost] Error 2
make[1]: *** [Makefile:234: __sub-make] Error 2
make[1]: Leaving directory '/usr/src/linux-6.5-rc3'
make: *** [Makefile:117: vmmon.ko] Error 2

[xevilstar]

confirmed

[mkubecek]

Yes, I'm aware of this. My research so far indicates the whole function using pte_offset_map() should be likely replaced by a call to get_user_pages() or pin_user_pages() or one of their variants (which is what kvm code seems to do for similar purpose). But there are two problems: (1) I'm not very familiar with this API and (2) without the source of the userspace part of the VMware hypervisor, it's unclear if and for what does the userspace code use these PFNs so it's hard to say which variant and which flags would be most appropriate.

[xevilstar]

there's a discussion I found http://rglinuxtech.com/?p=3157 but the proposed patch seems not complete ....

[mkubecek]

Well, pte_offset_kernel() might be a quick workaround, I didn't notice it is still available for modules. But I'll have to check what exactly is the difference and it would need at least a replacement of *pte in the following call to pte_present() the way upstream does in mainline commit c33c794828f2 ("mm: ptep_get() conversion").

[xevilstar]

What I am humbly proposing is ... can you prettypleasewithacherryontop temporarily patch the code the way that the discussion is suggesting while searching for a better way ?

Thank you for your time and effort :)

[xevilstar]

vmmon.patch.txt
`— vmware/vmmon-only/include/pgtbl.h 2023-06-30 08:04:13.247729900 +0800
+++ vmware/vmmon-only/include/pgtbl.h 2023-06-30 08:05:41.717728599 +0800
@@ -91,7 +91,7 @@
} else {
pte_t *pte;

– pte = pte_offset_map(pmd, addr);

• pte = pte_offset_kernel(pmd, addr);
  if (pte_present(*pte) == 0) {
  pte_unmap(pte);
  return INVALID_MPN;
  — vmware/vmnet-only/bridge.c 2023-06-30 09:13:50.917668443 +0800
  +++ vmware/vmnet-only/bridge.c 2023-06-30 09:14:05.747668224 +0800
  @@ -44,6 +44,7 @@
  #include
  #include
  #include
  +#include <net/gso.h>

#ifdef CONFIG_NET_RADIO

include

`

[xevilstar]

vmmon.tar.zip
vmnet.tar.zip
Hello,

I have applied the patch manually and it works

root@ghost:~# uname -a
Linux ghost 6.5.0-rc5 #1 SMP PREEMPT_DYNAMIC Wed Aug  9 14:43:13 CEST 2023 x86_64 GNU/Linux
root@ghost:~# lsmod|grep vm
vmw_vsock_vmci_transport    45056  0
vsock                  61440  1 vmw_vsock_vmci_transport
vmnet                  77824  17
vmmon                 163840  1
iwlmvm                589824  0
videobuf2_vmalloc      20480  1 uvcvideo
mac80211             1396736  1 iwlmvm
videobuf2_memops       16384  1 videobuf2_vmalloc
iwlwifi               544768  1 iwlmvm
cfg80211             1339392  3 iwlmvm,iwlwifi,mac80211
vmwgfx                442368  0
videobuf2_common       77824  4 videobuf2_vmalloc,videobuf2_v4l2,uvcvideo,videobuf2_memops
rfkill                 40960  10 iwlmvm,asus_wmi,bluetooth,cfg80211
drm_ttm_helper         12288  1 vmwgfx
ttm                   106496  2 vmwgfx,drm_ttm_helper
drm_kms_helper        266240  3 vmwgfx,nvidia_drm,gud
nvme                   57344  5
nvme_core             196608  6 nvme
t10_pi                 20480  2 sd_mod,nvme_core
drm                   765952  19 vmwgfx,drm_kms_helper,drm_shmem_helper,nvidia,drm_ttm_helper,nvidia_drm,ttm,gud

[mkubecek]

First crude attempt: commit b049eda. Note that I still have only vague idea what the returned PFNs are used for so the get_user_pages flags are quite likely wrong. On the hand, I was able to build and load the module and run a quick test with a VM on top of a 6.5-rc7 kernel so I may be on the right track after all.

TL;DR: use on your risk. :-)

[playervalley]

can confirm this on kernel 6.5.0-1.

[xevilstar]

I got it working with your latest b049eda git no risk involved.

latest mainline 6.5.0 no rc kernel compiled with make -j16 bindeb-pkg
Linux ghost 6.5.0 #2 SMP PREEMPT_DYNAMIC Mon Aug 28 12:19:05 CEST 2023 x86_64 GNU/Linux

It was a bit of a hassle to get the latest vmmon-only and vmnet-only workstation-17.0.2-k6.5
If I humbly may it would be easier for me to just have the whole thing with just a git clone (but I am a nobody heheheh)

[JoeSalmeri]

@mkubecek , @xevilstar

FWIW, I just updated one of my opensuse TW test machines to TW build 20230911 and it is using kernel 6.5.2.1 and I'm seeing the same issue.

Here's the compile log in case it has other info that will help:

AppLoader] GLib does not have GSettings support.
make: Entering directory '/tmp/modconfig-iPjRpu/vmmon-only'
Using kernel build system.
/usr/bin/make -C /lib/modules/6.5.2-1-default/build/include/.. M=$PWD SRCROOT=$PWD/.
MODULEBUILDDIR= modules
make[1]: Entering directory '/usr/src/linux-6.5.2-1-obj/x86_64/default'
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/linux/driver.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/linux/driverLog.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/linux/hostif.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/common/apic.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/common/comport.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/common/cpuid.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/common/crosspage.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/common/memtrack.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/common/moduleloop.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/common/phystrack.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/common/sharedAreaVmmon.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/common/statVarsVmmon.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/common/task.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/common/vmx86.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/bootstrap/bootstrap.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/bootstrap/monLoader.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/bootstrap/monLoaderVmmon.o
CC [M] /tmp/modconfig-iPjRpu/vmmon-only/bootstrap/vmmblob.o
LD [M] /tmp/modconfig-iPjRpu/vmmon-only/vmmon.o
MODPOST /tmp/modconfig-iPjRpu/vmmon-only/Module.symvers
ERROR: modpost: "__pte_offset_map" [/tmp/modconfig-iPjRpu/vmmon-only/vmmon.ko] undefined!
make[3]: *** [/usr/src/linux-6.5.2-1/scripts/Makefile.modpost:150: /tmp/modconfig-iPjRpu/vmmon-only/Module.symvers] Error 1
make[2]: *** [/usr/src/linux-6.5.2-1/Makefile:2001: modpost] Error 2
make[1]: *** [../../../linux-6.5.2-1/Makefile:234: __sub-make] Error 2
make[1]: Leaving directory '/usr/src/linux-6.5.2-1-obj/x86_64/default'
make: *** [Makefile:117: vmmon.ko] Error 2
make: Leaving directory '/tmp/modconfig-iPjRpu/vmmon-only'
make: Entering directory '/tmp/modconfig-iPjRpu/vmnet-only'
Using kernel build system.
/usr/bin/make -C /lib/modules/6.5.2-1-default/build/include/.. M=$PWD SRCROOT=$PWD/.
MODULEBUILDDIR= modules
make[1]: Entering directory '/usr/src/linux-6.5.2-1-obj/x86_64/default'
CC [M] /tmp/modconfig-iPjRpu/vmnet-only/driver.o
CC [M] /tmp/modconfig-iPjRpu/vmnet-only/hub.o
CC [M] /tmp/modconfig-iPjRpu/vmnet-only/userif.o
CC [M] /tmp/modconfig-iPjRpu/vmnet-only/netif.o
CC [M] /tmp/modconfig-iPjRpu/vmnet-only/bridge.o
CC [M] /tmp/modconfig-iPjRpu/vmnet-only/procfs.o
CC [M] /tmp/modconfig-iPjRpu/vmnet-only/smac_compat.o
CC [M] /tmp/modconfig-iPjRpu/vmnet-only/smac.o
CC [M] /tmp/modconfig-iPjRpu/vmnet-only/vnetEvent.o
CC [M] /tmp/modconfig-iPjRpu/vmnet-only/vnetUserListener.o
LD [M] /tmp/modconfig-iPjRpu/vmnet-only/vmnet.o
MODPOST /tmp/modconfig-iPjRpu/vmnet-only/Module.symvers
CC [M] /tmp/modconfig-iPjRpu/vmnet-only/vmnet.mod.o
LD [M] /tmp/modconfig-iPjRpu/vmnet-only/vmnet.ko
BTF [M] /tmp/modconfig-iPjRpu/vmnet-only/vmnet.ko
Skipping BTF generation for /tmp/modconfig-iPjRpu/vmnet-only/vmnet.ko due to unavailability of vmlinux
make[1]: Leaving directory '/usr/src/linux-6.5.2-1-obj/x86_64/default'
/usr/bin/make -C $PWD SRCROOT=$PWD/.
MODULEBUILDDIR= postbuild
make[1]: Entering directory '/tmp/modconfig-iPjRpu/vmnet-only'
make[1]: 'postbuild' is up to date.
make[1]: Leaving directory '/tmp/modconfig-iPjRpu/vmnet-only'
cp -f vmnet.ko ./../vmnet.o
make: Leaving directory '/tmp/modconfig-iPjRpu/vmnet-only'
Unable to install all modules. See log for details.

[rgadsdon]

I had also posted this issue on the VMware Forum for the Tech Preview (W/S 18), and just received the following suggested fix, from comphilip:

In vmmon-only/include/pgtbl.h, replace pte_offset_map with pte_offset_kernel

I have tested this briefly with kernel 6.5 and the Tech Preview, and also with 6.5.3 and 6.6-rc1 and the standard 17.0.2 patchset (not the tmp version) and this compiles OK and seems to work..

[mkubecek]

I really don't like the trick with pte_offset_kernel(), even if I wouldn't be too surprised if it ended up being what VMware comes with later. I guess there is no point waiting any more, I'll add the patch using get_user_pages() to regular branches later today, we can always change it later if any issues are found.