[compiler] Support Ubuntu24.04 with latest packages

[seanshpark]

Check is done and rough patches are now ready.
Let's support Ubuntu 24.04 and with (almost) latest python packages too.

• check: [compiler] Check build on Ubuntu 24.04 #15220

[seanshpark]

py versions

package	U20.04	U22.04	U24.04	note
python	3.8	3.10	3.12
TensorFlow	2.12.1 (~ 2.13)	2.19	(2.16.1 ~) 2.19	2.19 is latest
onnx	1.17	1.17	1.17	1.17 is latest
onnxruntime	1.18.1	1.21.1	1.21.1	1.21.1 is latest
onnx-tf	1.10.0	1.10.0	N/A	not maintained anymore
numpy	1.24.3	1.26.4	1.26.4	not compatible with 2.x
flatbuffers	23.5.26	24.3.25	24.3.25
torch	2.1.2+cpu	2.7.0+cpu	2.7.0+cpu	2.7.0 is latest

Note

• for U20.04, use TF 2.12.1 as before, last for U20.04 is 2.13
  • tensorflow-cpu 2.12.1 depends on numpy<=1.24.3 and >=1.22
  • tensorflow-cpu 2.19.0 depends on flatbuffers>=24.3.25
• onnxruntime fails with value check in 1.19, 1.20
  • [circle-mlir] Build failure on Ubuntu-20.04 focal distribution #15181 (comment)
• onnx-tf depends on tensorflow-addon which is not available on python 3.12

---

Three U2x.04 doesn't seems to give single package version.
Even, grouping to two doesn't seem to be possible.

[seanshpark]

U20.04 build fails with all modules

externals/CAFFE/src/caffe/util/io.cpp: In function 'bool caffe::ReadProtoFromBinaryFile(const char*, google::protobuf::Message*)':
externals/CAFFE/src/caffe/util/io.cpp:57:66: error: no matching function for call to 'google::protobuf::io::CodedInputStream::SetTotalBytesLimit(const int&, int)'
   57 |   coded_input->SetTotalBytesLimit(kProtoReadBytesLimit, 536870912);
      |                                                                  ^
In file included from externals/CAFFE/src/caffe/util/io.cpp:2:
build/debug/overlay/include/google/protobuf/io/coded_stream.h:407:8: note: candidate: 'void google::protobuf::io::CodedInputStream::SetTotalBytesLimit(int)'
  407 |   void SetTotalBytesLimit(int total_bytes_limit);
      |        ^~~~~~~~~~~~~~~~~~

-> #14639 isn't seem to be applied
-> try with delete externsla/CAFFE cache

[seanshpark]

On U24.04, one-import-onnx tests will fail, as there is no onnx-tf.
we need to install onnx2circle tool built from circle-mlir.

-> onecc_016.test FAILED in https://github.com/Samsung/ONE/actions/runs/14656456915/job/41132304396?pr=15227

[seanshpark]

package version with Ubuntu platform

It looks hard to set to single set of packages to all Ubuntu releases.
I'd like to gather opinions about this situation.

What I think;

• select versions for each packages that will be the best per Ubuntu version,
• resulting there can be no common set of package with same version
• current available set
  • U18.04 : drop
  • U20.04 : python3.8, TF 2.12.1 (stay), onnxrt 1.18.1 (stay), onnx-tf 1.10.0 (stay)
  • U22.04 : python3.10, TF 2.19.0, onnxrt 1.21.1, onnx-tf 1.10.0 (stay)
  • U24.04 : python3.12, TF 2.19.0, onnxrt 1.21.1, onnx-tf N/A, onnx2circle

@Samsung/one_compiler , please share your thoughts.

[shs-park]

please share your thoughts.

It seems difficult to maintain a single common set of package versions across all Ubuntu releases now.

I would like to suggest the following approach:

• Define a Recommended Ubuntu Version: 22.04
  We officially recommend using a specific Ubuntu version (e.g., Ubuntu 22.04) with a verified set of package versions.

• Provide a guide for using other Ubuntu versions
  If users choose to use different Ubuntu versions (e.g., Ubuntu 20.04 or 24.04), provide setup guidelines, including compatible Python, TensorFlow, ONNX Runtime, and ONNX-TF (or alternatives like onnx2circle) versions.

• Benefits of this approach

  • Ensures stability for the recommended environment.
  • Offers flexibility for users who need or prefer other Ubuntu versions.
  • Simplifies maintenance by focusing on a primary environment while still supporting alternatives.

Example guideline structure:

Ubuntu Version	Python	TensorFlow	ONNX Runtime	ONNX-TF / Alternative
20.04	3.8	2.12.1	1.18.1	1.10.0
22.04 (Recommended)	3.10	2.19.0	1.21.1	1.10.0
24.04	3.12	2.19.0	1.21.1	onnx2circle

[jinevening]

I think onnx2circle can be installed on 20.04 and 22.04 as well.

[seanshpark]

I think onnx2circle can be installed on 20.04 and 22.04 as well.

Yes, currently, onnx2circle is installed. (well... in our inhouse tool)
But executed as fallback when onnx-tf fails to convert :)
For U24.04, onnx2circle should become the primary ONNX to circle converter.

as onnx2circle is now opened, I need to update things with Debian packages...

[seanshpark]

to support multiple platforms with different python packages sets...
I need to revise "requirements.txt" generation codes in CMakeLists.txt

--> I need to split

  add_custom_command(
    OUTPUT ${REQUIREMENTS_OVERLAY_PATH_TF_2_12_1}
    ...
  )

to 4 parts.
1/ prepare a list of package==version per platform
2/ remove existing requirements.txt file
3/ generate requirements.txt from the list of 1/
4/ install python packages

--> as these are executed with add_custom_command(), maybe in parallel, we need to care with setting DEPENDS argument.

---

done

[seanshpark]

one-prepare-venv has package version numbers inside the code that depends on python(or platform) version.
and already exists multiple files, that is not good for maintenace.

let's refactor one-prepare-venv with separating package versions numbers.