Failed to run resnet example

[Minamoto25]

I have built PhoenixOS in the container provided in the document, and then I want to test C/R of ResNet program in the same container, but there seems to be a problem with the API forwarding.
I first executed the following command in one terminal, and the outputs are as follow:

~# pos_cli --start --target daemon
 POS Log  >>>>>>>>>> PhOS Workspace <<<<<<<<<<
 _____  _                      _       ____   _____
|  __ \| |                    (_)     / __ \ / ____|
| |__) | |__   ___   ___ _ __  ___  _| |  | | (___
|  ___/| '_ \ / _ \ / _ \ '_ \| \ \/ / |  | |\___ \
| |    | | | | (_) |  __/ | | | |>  <| |__| |____) |
|_|    |_| |_|\___/ \___|_| |_|_/_/\_\\____/|_____/

 POS Log  PhoenixOS workspace created, welcome!
+00:00:00.420194 INFO:  waiting for RPC requests...
Cache Optimization: Enabled!
Async Optimization: Enabled!
Handler Optimization: Enabled!
xpu remote address: localhost
create shm buffer

And then executed the following command in another terminal:

~/examples/resnet# env $phos python3 train.py
+00:00:00.629521 ERROR: error registering fatbin: 32621 in cpu-client.c:478
+00:00:00.629555 ERROR: error registering function: 496557680   in cpu-client.c:437
+00:00:00.681239 INFO:  api-call-cnt: 0
+00:00:00.681268 INFO:  memcpy-cnt: 0
----client_total_infos----
api 1: count 1, client_total_time 65336.000000
api 2: count 1, client_total_time 34162.000000
free(): invalid pointer
Aborted (core dumped)

Then nvidia-smi outputs (executed in container):

~/examples/resnet# nvidia-smi
Sat Nov  9 14:00:19 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07              Driver Version: 550.90.07      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:B8:00.0 Off |                  Off |
| 30%   30C    P8              4W /  450W |     782MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off |   00000000:D8:00.0 Off |                  Off |
| 30%   32C    P8              9W /  450W |     782MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A     83346      C   cricket-rpc-server                              0MiB |
|    1   N/A  N/A     83346      C   cricket-rpc-server                              0MiB |
+-----------------------------------------------------------------------------------------+

Some details about my environment (executed in container):

~/examples/resnet# nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_May__3_19:15:13_PDT_2021
Cuda compilation tools, release 11.3, V11.3.109
Build cuda_11.3.r11.3/compiler.29920130_0



~/examples/resnet# python3
Python 3.8.10 (default, Sep 11 2024, 16:02:53)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.13.0+cu117'

[zobinHuang]

Hi Minamoto25

Could you please check whether there's a file named client_exist.txt on client-side's working directory?
If there's, try delete this file and run again.

[Minamoto25]

Yes, this file exists. I deleted it, and run again, but still came across an error.
On the server side:

~# pos_cli --start --target daemon
 POS Log  >>>>>>>>>> PhOS Workspace <<<<<<<<<<
 _____  _                      _       ____   _____
|  __ \| |                    (_)     / __ \ / ____|
| |__) | |__   ___   ___ _ __  ___  _| |  | | (___
|  ___/| '_ \ / _ \ / _ \ '_ \| \ \/ / |  | |\___ \
| |    | | | | (_) |  __/ | | | |>  <| |__| |____) |
|_|    |_| |_|\___/ \___|_| |_|_/_/\_\\____/|_____/

 POS Log  PhoenixOS workspace created, welcome!
+00:00:00.439605 INFO:  waiting for RPC requests...
Cache Optimization: Enabled!
Async Optimization: Enabled!
Handler Optimization: Enabled!
xpu remote address: localhost
create shm buffer
file:       ../pos/src/worker.cpp;
function:   static void POSWorker::__restore(POSWorkspace*, POSAPIContext_QE*);
line:       92;
 POS Log  [POSParser @ 0x7fb558001040]  parser started
 POS Log  [POSWorker @ 0x7fb558001cd0]  worker started
 POS Log  parsing cached kernel metas from file /var/log/phos/daemon/resnet152_kernel_metas.txt...
 POS Log  reserved virtual memory space: device_id(0), base(0x7facd0000000), size(21908946944)
 POS Log  reserved virtual memory space: device_id(1), base(0x7fb630000000), size(21908946944)
+00:00:40.522713 INFO:  string: "hello"

 POS Log  [POSParser @ 0x7fb5589d7660]  parser started
 POS Log  [POSWorker @ 0x7fb558715c90]  worker started
 POS Log  parsing cached kernel metas from file /var/log/phos/daemon/resnet152_kernel_metas.txt...
 POS Log  finish dump kernel metadata to /var/log/phos/daemon/resnet152_kernel_metas.txt
 POS Log  [POSParser @ 0x7fb5589d7660]  parser daemon thread shutdown
 POS Log  [POSWorker @ 0x7fb558715c90]  worker daemon thread shutdown
 POS Warn  failed to cuModuleLoadData normal module
 POS Error  execute failed, restore mechanism to be implemented: api_id(1019), retcode(209), pc(19392)
terminate called after throwing an instance of 'char const*'
Aborted (core dumped)
 POS Warn  failed execution of command cricket-rpc-server 2>&1: exit_code(134)
 POS Warn  failed to start posd

On the client side, I got lots of outputs like this, and then the program stalled:

+00:02:56.138921 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:02:56.138969 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:02:56.139067 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:02:56.139114 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:02:56.139162 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:02:56.139210 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:02:56.139258 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:02:56.139307 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:02:56.139354 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:02:56.139455 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:02:56.139503 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:02:56.139550 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:02:56.139598 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:02:56.139645 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:02:56.139692 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:02:56.139740 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:03:13.914566 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:03:13.914756 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:03:13.914816 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:03:13.914873 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:03:13.914927 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:03:13.914978 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922
+00:03:13.915029 WARNING: could not find .nv.info section. This means this binary does not contain any kernels. in cpu-elf2.c:922

[zobinHuang]

Hi Minamoto25,

I think it's due to the image phoenixos/pytorch:11.3-ubuntu20.04 we provided is based on CUDA 11.3, which is too old to run on your platform -- RTX 4090 (i.e., sm_89 which should be supported since CUDA 11.8 and later).

We will fix this issue by providing support for CUDA 12.1 as soon as possible, please hold this issue. Thanks for your patience!

[Minamoto25]

I was using container version nvidia/cuda:11.3.1-cudnn8-devel-ubuntu20.04 before, which provided in README for building the project. It seems that I used another version of torch ('1.13.0+cu117'), which used some unimplemented cuda api. And when I switched to version phoenixos/pytorch:11.3-ubuntu20.04, I successfully ran the ResNet migration sample.But the client side still output some error log:

+00:00:11.309114 ERROR: dlopen failed: /opt/conda/lib/python3.8/site-packages/torchvision/_C.so in cpu-client.c:344
+00:00:11.322186 ERROR: dlopen failed: /opt/conda/lib/python3.8/site-packages/torchvision/image.so      in cpu-client.c:344

Is that normal?
Then I tried the migration sample for Llama2, and it seems that some package versions are still incorrect. When downloading the model, I encountered the following error:

/opt/conda/lib/python3.8/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
Traceback (most recent call last):
  File "download.py", line 30, in <module>
    model = AutoModelForCausalLM.from_pretrained(model_id, device_map='auto')
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 484, in from_pretrained
    return model_class.from_pretrained(
  File "/opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py", line 2801, in from_pretrained
    max_memory = get_balanced_memory(
  File "/opt/conda/lib/python3.8/site-packages/accelerate/utils/modeling.py", line 1020, in get_balanced_memory
    module_sizes = compute_module_sizes(model, dtype=dtype, special_dtypes=special_dtypes)
  File "/opt/conda/lib/python3.8/site-packages/accelerate/utils/modeling.py", line 732, in compute_module_sizes
    size = tensor.numel() * dtype_byte_size(tensor.dtype)
  File "/opt/conda/lib/python3.8/site-packages/accelerate/utils/modeling.py", line 166, in dtype_byte_size
    elif is_torch_version(">=", "2.1.0") and dtype == torch.float8_e4m3fn:
AttributeError: module 'torch' has no attribute 'float8_e4m3fn'

It seems that the version of the accelerate package is not matching, what version of the package did you use for testing?
My environment is as follows:

Package                   Version
 ------------------------- -------------------
accelerate                1.0.1
torch                     1.13.0a0+git2263262
transformers              4.30.0

[zobinHuang]

Hi Minamoto,

For your first question:
Glad to see you're able to run the example. Yes this is normal, simply ignore this error message. The reason for this message is torchvision would attempt to open some .sos even though they are not exist. We have hijacked the dlopen API for hooking CUDA API calls, and print an error message everytime we failed to open a .so.

For your second question:
I think it's due to the accelerate package's version is too new for torch 11.3.
Could you avoid using accelerate to download the model? You can try our download script provided under examples/llama2-13b-chat-hf/download.py