Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
8 changes: 8 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,7 @@ ompi/tools/ompi_info/ompi_info

ompi/tools/wrappers/mpic++-wrapper-data.txt
ompi/tools/wrappers/mpicc-wrapper-data.txt
ompi/tools/wrappers/mpicc_abi-wrapper-data.txt
ompi/tools/wrappers/mpifort-wrapper-data.txt
ompi/tools/wrappers/ompi_wrapper_script
ompi/tools/wrappers/ompi.pc
Expand Down Expand Up @@ -534,6 +535,13 @@ docs/man

# Generated C Bindings
ompi/mpi/c/*_generated*.c
ompi/mpi/c/standard_*.c
ompi/mpi/c/abi.h
ompi/mpi/c/abi_get_info.c
ompi/mpi/c/abi_converters.h
ompi/mpi/c/abi_converters.c
ompi/mpi/c/standard_abi
ompi/mpi/tool/*_generated*.c

# Generated Fortran Bindings
ompi/mpi/fortran/use-mpi-f08/*_generated.F90
Expand Down
13 changes: 11 additions & 2 deletions VERSION
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@ minor=1
release=0

# MPI Standard Compliance Level
mpi_standard_version=3
mpi_standard_subversion=1
mpi_standard_version=5
mpi_standard_subversion=0

# OMPI required dependency versions.
# List in x.y.z format.
Expand Down Expand Up @@ -92,6 +92,14 @@ date="Unreleased developer copy"
# but they are "public" within the OMPI code base and select 3rd party
# software packages.

# The current and age values of the libmpi_abi shared library are
# implicitly determined by the MPI ABI definition of the
# version of the MPI specification supported by this Open MPI release.
# The MPI ABI is intended not to break backward compatibility,
# so presumably the current and age values will move in lock step.
# We assume we have flexibility with the revision number as
# bugs are fixed, etc.

# Version numbers are described in the Libtool current:revision:age
# format.

Expand All @@ -100,6 +108,7 @@ libmpi_mpifh_so_version=0:0:0
libmpi_usempi_tkr_so_version=0:0:0
libmpi_usempi_ignore_tkr_so_version=0:0:0
libmpi_usempif08_so_version=0:0:0
libmpi_abi_so_version=0:0:0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I built MPICH 5.0b1 and see that it installs libmpi_abi.so.0[.0.0].

Per our conventions, will we be shifting this CRA value to something other than 0:0:0 on release branches (e.g., 0:1:0, so that C-A is still 0, and we still produce libmpi_abi.so.0 to match MPICH)? I ask this only because Libtool recommends not shipping 0:0:0.

Thinking a little further down this rabbit hole: does having both ABI-enabled Open MPI and MPICH allow installing Open MPI and MPICH into the same $prefix (with all default sub directories, like $includedir being $prefix/include)? I'm thinking "no" for at least the following reasons:

  1. mpi.h will have all the ABI things being identical, but we'll have other differences from MPICH's mpi.h (right?).
    • Bottom line: package managers would conflict on $prefix/include/mpi.h.
    • That being said, perhaps creative use of ./configure --includedir=... could workaround that.
  2. There's nothing to distinguish between libmpi_abi.so.* -- you couldn't tell if it was from Open MPI or MPICH. From a user perspective, that might be ok, but from a package manager and/or system administrator point of view -- that might get a little weird. For example, what if we both ship libmpi_abi.so.0.a.b with the same a and b values?
    • Bottom line: package managers would conflict if we both ship -- for example -- libmpi_abi.so.0.0.0.
    • Put differently, should we ensure that our libmpi_abi shared library a and b values are different than MPICH's somehow? (I really haven't thought this through to know if this is even possible in a sustainable way over time -- nor what the consequences are outside of Linux)

I guess I'm wondering if it's useful to build Open MPI and MPICH with something like:

# Open MPI
$ ./configure --prefix=/opt/mpi --includedir=/opt/mpi/openmpi ...
# MPICH
$ ./configure --prefix=/opt/mpi --includedir=/opt/mpi/mpich --enable-mpi-abi ...

This would keep a single libdir so that we don't introduce (more) LD_LIBRARY_PATH complexity, but still allow unique mpi.h.

But then again, there's still problems with mpirun and mpiexec filename clashes in $bindir (not to mention CLI flag differences). Maybe something like Linux-style alternatives could be useful here...? Shrug.

I understand how ABI between MPI libraries solves some perceived problems for users, but trying to go the next steps to actually hide the differences between MPI implementations gets pretty tricky pretty quickly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking a little further down this rabbit hole: does having both ABI-enabled Open MPI and MPICH allow installing Open MPI and MPICH into the same $prefix (with all default sub directories, like $includedir being $prefix/include)? I'm thinking "no" for at least the following reasons:

The point of having a unique ABI is to be able to switch between different backends, which effectively requires same sonames (and filenames), so that's pretty much by design. You'd either have a single MPI library at a time in a single-prefix scenario, or as many as you want in a multiple-prefix scenario (think of Spack).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

another use case I could see is the usual "modules" setup on an HPC cluster. the user could just switch between the mpich module and the openmpi module without needing to recompile/relink. that's basically how one would use spack modules system.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You cannot just blindly switch from one module to another UNLESS the ABI's are an exact match - in other words, you have to ensure that the version of the ABI you are switching to is the same as the one you currently are using. An ABI is a rigid specification - there is no such thing as a minor revision to it. Any change - be it an addition, subtraction, or (heaven forbid) a modification - results in a new ABI, and the version number (in libtool parlance, the .so number) must change.

That's the entire point of the libtool .so number - to guide the linker to picking the library that matches the signature required by the executable. In this case, that's the ABI.

People who have been building ABIs learned this the hard way. As @jsquyres pointed out, there are a ton of other problems - but setting the .so number to the ABI version is a basic necessity. Having an unchanging .so number even when the ABI changes is a disaster. The linker will basically be playing russian roulette, and users will rapidly find it...let's politely say, less than useful.

In this case, you want the ABI library to have a .so that matches the ABI it supports. You benefit from having a second library - the actual backend implementation - that can change as it is modified. Key is to tie the ABI library to the matching backend, and then change that connection as you update backends. In other words, you update the ABI-backend combination when the backend gets updated.

So the "module" is an ABI-backend combination, and the user picks the ABI they want supported along with the underlying implementation that supports that ABI. In other words, "give me MPI v2 ABI and the OMPI v6.2.1 backend". If you don't care about ABI, then just pick the implementation library. If you don't care about backend, then select the ABI module and let it pick the default backend.

Copy link
Contributor

@dalcinl dalcinl Nov 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per our conventions, will we be shifting this CRA value to something other than 0:0:0 on release branches (e.g., 0:1:0, so that C-A is still 0, and we still produce libmpi_abi.so.0 to match MPICH)?

@jsquyres IMHO, the way this should be handled is the following: The CURRENT and AGE number are implicitly defined from the MPI_ABI_VERSION/MPI_ABI_SUBVERSION numbers, that is, by the set of backward-compatible additions or the backward-incompatible changes. I am assuming that MPI_ABI_VERSION will stay at 1 as long as there are no backward-incompatible changes, and MPI_ABI_SUBVERSION will bump on backward-compatible additions/updates.
The REVISION number could be left for use by the MPI implementation, this way multiple revisions can be installed in the same prefix location, with ldconfig going through its usual cache update rules.

I tried to layout the rules here mpi-forum/mpi-abi-stubs#28. The "formulas" there would produce a soname libmpi_abi.so.1, but if we want a .0 suffix, that's trivial to fix by subtracting 1 from the formula for current.

I ask this only because Libtool recommends not shipping 0:0:0.

Can you point me to such recommendation?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to be clear: I know what the use cases are for ABI (e.g., env modules-style or spack-style environments). My question about whether two installs could share a common $prefix was probably more of a hypothetical musing more than anything else. I even answered my own question -- the answer is probably "no", for the reasons already discussed.


@jsquyres IMHO, the way this should be handled is the following: The CURRENT and AGE number are implicitly defined from the MPI_ABI_VERSION/MPI_ABI_SUBVERSION numbers,

I admit that I don't quite understand the purpose of MPI_ABI_VERSION and MPI_ABI_SUBVERSION. As you pointed out earlier in this thread, there's essentially 2 common styles of maintaining binary compatibility these days:

  1. The Libtool CRA triple
  2. MacOS linker-style single-integer versioning

How exactly do a pair of compile-time constants fit into either of those schemes? It's not described in MPI-5.0, nor is an alternate (i.e., 3rd) scheme described. MPI-5.0:20.2 loosely implies that MPI_ABI_SUBVERSION can be used as a proxy for MPI_VERSION and MPI_SUBVERSION (i.e., be used for conditional compilation of various MPI symbols / types / functions / etc.). But that seems odd -- why have new constants for a mechanism that has worked for decades?

Sure, MPI_ABI_VERSION could be a proxy for a Linux SONAME. But then what's the point of MPI_ABI_SUBVERSION at compile time (or even run time, via MPI_ABI_GET_VERSION())?

If we intend MPI_ABI_VERSION to be a proxy for Linux SONAME, that seems fine. Is there a scheme for how MPI_ABI_SUBVERSION should factor in here? The way that MPI_ABI_SUBVERSION is (loosely) defined in MPI-5.0 does not seem like a hypothetical scheme such as -- for example -- (MPI_ABI_VERSION * 100 + MPI_ABI_SUBVERSION) would be a good candidate as a proxy for the Linux SONAME. So what do implementations and/or users use MPI_ABI_SUBVERSION for?

I'm digressing from the main question here, and I don't mean to open a whole debate about these 2 compile-time constants here in OMPI -- such issues can be discussed at the Forum level.

For an ABI to satisfy the use cases described above (e.g., swapping out the back end), the questions of how to create linker-compatible shared library versions should really be resolved into some kind of scheme that both Open MPI and MPICH -- and our various derivative implementations -- follow. This doesn't necessarily have to be in the MPI spec itself; it's probably better as an agreement between the Open MPI and MPICH communities. My point: we need to have the discussion and then publicly document the scheme so that anyone can follow it (e.g., even outside of Open MPI and MPICH).

I ask this only because Libtool recommends not shipping 0:0:0.

Can you point me to such recommendation?

Doh! I just re-read the LT docs and I cannot find such a recommendation. So I guess I'm wrong here. Perhaps I'm remembering some super-old conversation about how we (OMPI?) didn't want to release with 0:0:0 because that's what's on main and we don't release off main (similar to the project version number) -- i.e., more of a release philosophy kind of thing than a strict technical requirement.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am assuming that MPI_ABI_VERSION will stay at 1 as long as there are no backward-incompatible changes, and MPI_ABI_SUBVERSION will bump on backward-compatible additions/updates.

My assumption at that time is now part of the standard, MPI 5.0 says (sec 20.2 pp 844):

Backwards-compatible changes, such as the addition of new handle types, will incre-
ment the minor version. Backwards-incompatible changes will increment the major version.
The addition of new functions to the MPI API does not change the ABI version.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backwards-compatible changes, such as the addition of new handle types, will incre-
ment the minor version. Backwards-incompatible changes will increment the major version.
The addition of new functions to the MPI API does not change the ABI version.

Yes, I saw that. But can you provide an example of how it would be useful / used?

I.e., how exactly is it different than MPI_VERSION an MPI_SUBVERSION?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I saw that. But can you provide an example of how it would be useful / used?

Long ago, somewhere, I claimed that MPI_ABI_VERSION/SUBVERSION was not that useful. As usual, I was ignored ;'-), and now we have these version numbers in the standard. Anyway, now I believe it is still good to have the version defined (though not necessary the C macros). The MPI ABI version/subversion values are to be updated following similar rules as libtool, therefore we can use them to define the C/R/A tuple the following way:

C := MPI_ABI_VERSION + MPI_ABI_SUBVERSION - 1
R := <free for MPI implementations to update at will>
A := MPI_ABI_SUBVERSION

and then under these rules we get a SONAME libmpi_abi.so.0 and then all the planets are aligned.

Could you please give a bit of though to this claim of mine? Think again about the rules for updating the MPI ABI version/subversion, the libtool c/r/a update rules, my formulas above, my claim about the SONAME, and confirm whether am I right?

The other obvious uses if conditional compilation with the macros. I use the presence of MPI_ABI_VERSION in mpi4py to conditionally-compile if building against the MPI standard ABI. Regarding the use of the values of MPI_ABI_VERSION/SUBVERSION, there is definitely some overlap with MPI_VERSION/SUBVERSION.

I.e., how exactly is it different than MPI_VERSION an MPI_SUBVERSION?

MPI_VERSION/SUBVERSION follow the version of the MPI standard, this is unrelated to ABI or even API. The MPI standard version is not only about API but also about runtime behavior changes. MPI_VERSION/SUBVERSION evolve in ways that are totally unrelated to whether the API/ABI changes are backward compatible or not, while MPI_ABI_VERSION will stay at 1 for as long as the MPI Forum does not introduce backward incompatible changes.

Copy link
Member

@jsquyres jsquyres Nov 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting CRA proposal. I don't think it's quite right, though -- I think you have to give the MPI_ABI_[SUB]VERSION their own distinct digits (somewhat akin to bit mapping). E.g.:

// Give subversion its own 2 digits.  I.e., the Forum should never allow SUBVERSION>99
C := (MPI_ABI_VERSION * 100) + MPI_ABI_SUBVERSION
R := <free for MPI implementations to update at will>
A := MPI_ABI_SUBVERSION

This gives you unique values. Otherwise, you could end up with

  • Version X, with:
    • MPI_ABI_VERSION=1, MPI_ABI_SUBVERSION=2
    • Original scheme yields C=2, A=2 --> Linux SONAME == 0
    • New scheme yields C=102, A=2 --> Linux SONAME == 100
  • Version Y with:
    • MPI_ABI_VERSION=2, MPI_ABI_SUBVERSION=0
    • Original scheme yields C=1, A=0
      • C went backwards compared to version X, which seems bad --> Linux SONAME == 1
    • New scheme yields C=200, A=0 --> Linux SONAME == 200
      • This seems appropriate because changing MPI_ABI_VERSION to 2 indicates that a backwards-incompatible change was made.

Put differently: the original scheme only works if MPI_ABI_VERSION never increases (which would be morally equivalent to hard-coding C=MPI_ABI_SUBVERSION-1, A=MPI_ABI_SUBVERSION). Otherwise, we can get repeat C and A values for different values of MPI_ABI_[SUB]VERSION.

MPI_VERSION/SUBVERSION follow the version of the MPI standard, this is unrelated to ABI or even API.

I guess what I'm asking for: can you give an example of something you'd need to #if on that is based on ABI and not API. This might be a failure of imagination on my part to come up with a useful example here...

And FWIW, I tend to prefer always defining preprocessor macros (as opposed to undefining them vs. defining them). If you always define them, you protect against typos:

// Both of these result in true
#if !defined(MPI_ABI_VERSION)
#if !defined(MPI_ABI_VERSIONBUT_I_HAVE_A_TYPO_HERE)

whereas this will result in a compilation error:

#if MPI_ABI_VERSIONBUT_I_HAVE_A_TYPO_HERE > 0

libopen_pal_so_version=0:0:0
libmpi_java_so_version=0:0:0
liboshmem_so_version=0:0:0
Expand Down
1 change: 1 addition & 0 deletions config/ompi_config_files.m4
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ AC_DEFUN([OMPI_CONFIG_FILES],[
ompi/tools/ompi_info/Makefile
ompi/tools/wrappers/Makefile
ompi/tools/wrappers/mpicc-wrapper-data.txt
ompi/tools/wrappers/mpicc_abi-wrapper-data.txt
ompi/tools/wrappers/mpic++-wrapper-data.txt
ompi/tools/wrappers/mpifort-wrapper-data.txt
ompi/tools/wrappers/ompi.pc
Expand Down
26 changes: 25 additions & 1 deletion config/ompi_configure_options.m4
Original file line number Diff line number Diff line change
Expand Up @@ -256,5 +256,29 @@ AM_CONDITIONAL(OMPI_OMPIO_SUPPORT, test "$ompi_want_ompio" = "1")
AC_ARG_ENABLE([deprecate-mpif-h],
[AS_HELP_STRING([--enable-deprecate-mpif-h],
[Mark the mpif.h bindings as deprecated (default: enabled)])])

# If the binding source files don't exist, then we need Python to generate them
AM_PATH_PYTHON([3.6],,[:])
binding_file="${srcdir}/ompi/mpi/c/ompi_send_generated.c"
AS_IF([! test -e "$binding_file" && test "$PYTHON" = ":"],
[AC_MSG_ERROR([Open MPI requires Python >=3.6 for generating the bindings. Aborting])])
AM_CONDITIONAL(OMPI_GENERATE_BINDINGS,[test "$PYTHON" != ":"])

AC_MSG_CHECKING([if want to enable standard ABI library])
AC_ARG_ENABLE([standard-abi],
[AS_HELP_STRING([--enable-standard-abi],
[Enable building the standard ABI library (default: enabled)])])
if test "$enable_standard_abi" = "no"; then
AC_MSG_RESULT([no])
ompi_standard_abi=0
else
AC_MSG_RESULT([yes])
ompi_standard_abi=1
fi
AC_DEFINE_UNQUOTED([OMPI_STANDARD_ABI],[$ompi_standard_abi],
[Whether we want to build the standard ABI library])
AM_CONDITIONAL(OMPI_STANDARD_ABI,[test $ompi_standard_abi = 1])
AS_IF([test $ompi_standard_abi -eq 1],
[gen_abi="yes"],
[gen_abi="no"])
OPAL_SUMMARY_ADD([Miscellaneous], [MPI Standard ABI support], [], [$gen_abi])
])dnl
4 changes: 3 additions & 1 deletion config/ompi_fortran_check_logical_array.m4
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ dnl All rights reserved.
dnl Copyright (c) 2011-2012 Cisco Systems, Inc. All rights reserved.
dnl Copyright (c) 2015 Research Organization for Information Science
dnl and Technology (RIST). All rights reserved.
dnl Copyright (c) 2025 Triad National Security, LLC. All rights
dnl reserved.
dnl $COPYRIGHT$
dnl
dnl Additional copyrights may follow
Expand Down Expand Up @@ -64,7 +66,7 @@ void ompi_check_f(ompi_fortran_logical_t * logical)
FILE *f=fopen("conftestval", "w");
if (!f) exit(1);

if (logical[[0]] == 0 &&
if (logical[[0]] == $ompi_cv_fortran_false_value &&
logical[[1]] == $ompi_cv_fortran_true_value)
result = 1;
fprintf(f, "%d\n", result);
Expand Down
60 changes: 56 additions & 4 deletions config/ompi_fortran_get_value_true.m4
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ dnl All rights reserved.
dnl Copyright (c) 2011-2012 Cisco Systems, Inc. All rights reserved.
dnl Copyright (c) 2015 Research Organization for Information Science
dnl and Technology (RIST). All rights reserved.
dnl Copyright (c) 2025 Triad National Security, LLC. All rights
dnl reserved.
dnl $COPYRIGHT$
dnl
dnl Additional copyrights may follow
Expand All @@ -27,16 +29,22 @@ AC_DEFUN([OMPI_FORTRAN_GET_VALUE_TRUE],[
if test "$ompi_cv_fortran_true_value" = "0" ; then
unset ompi_cv_fortran_true_value
fi
if test "$ompi_cv_fortran_false_value" = "0" ; then
unset ompi_cv_fortran_false_value
fi

AS_VAR_PUSHDEF([fortran_true_var],
[ompi_cv_fortran_true_value])
AS_VAR_PUSHDEF([fortran_false_var],
[ompi_cv_fortran_false_value])

AC_CACHE_CHECK([Fortran value for .TRUE. logical type],
fortran_true_var,
[if test "$1" = "none" || \
test $OMPI_TRY_FORTRAN_BINDINGS -eq $OMPI_FORTRAN_NO_BINDINGS || \
test $ompi_fortran_happy -eq 0 ; then
value=77
tvalue=77
fvalue=77
else
#
# C module
Expand Down Expand Up @@ -98,6 +106,14 @@ EOF
CALL ompi_print(value)
end
EOF
cat > conftestf2.f <<EOF
program main
logical value
value=.FALSE.
CALL ompi_print(value)
end
EOF


#
# Try the compilation and run.
Expand All @@ -114,11 +130,40 @@ EOF
AS_IF([test "$cross_compiling" = "yes"],
[AC_MSG_ERROR([Can not determine value of .TRUE. when cross-compiling])],
[OPAL_LOG_COMMAND([./conftest],
[value=`sed 's/ *//' conftestval`],
[tvalue=`sed 's/ *//' conftestval`],
[AC_MSG_ERROR([Could not determine value of Fotran .TRUE.. Aborting.])])])

cat > conftestf.f <<EOF
program main
logical value
value=.FALSE.
CALL ompi_print(value)
end
EOF

#
# Try the compilation and run.
#
OPAL_LOG_COMMAND([$CC $CFLAGS -I. -c conftest.c],
[OPAL_LOG_COMMAND([$FC $FCFLAGS -o conftest conftest.o conftestf.f $LDFLAGS $LIBS],
[happy=1], [happy=0])],
[happy=0])

AS_IF([test $happy -eq 0 && test $ompi_fortran_happy -eq 1],
[AC_MSG_ERROR([Could not compile Fortran .FALSE. test. Aborting.])
])

AS_IF([test "$cross_compiling" = "yes"],
[AC_MSG_ERROR([Can not determine value of .FALSE. when cross-compiling])],
[OPAL_LOG_COMMAND([./conftest],
[fvalue=`sed 's/ *//' conftestval`],
[AC_MSG_ERROR([Could not determine value of Fotran .FALSE.. Aborting.])])])
fi
AS_VAR_SET(fortran_true_var, [$value])
unset value
AS_VAR_SET(fortran_true_var, [$tvalue])
unset tvalue
AS_VAR_SET(fortran_false_var, [$fvalue])
unset fvalue

])

AS_VAR_COPY([ompi_fortran_true_value], [fortran_true_var])
Expand All @@ -127,6 +172,13 @@ EOF
[Fortran value for LOGICAL .TRUE. value])
AS_VAR_POPDEF([fortran_true_var])

AS_VAR_COPY([ompi_fortran_false_value], [fortran_false_var])
AC_DEFINE_UNQUOTED([OMPI_FORTRAN_VALUE_FALSE],
[$ompi_fortran_false_value],
[Fortran value for LOGICAL .FALSE. value])
AS_VAR_POPDEF([fortran_false_var])


unset happy ompi_print_logical_fn
rm -rf conftest*
])dnl
1 change: 1 addition & 0 deletions configure.ac
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@ OPAL_SAVE_VERSION([OPAL], [Open Portable Access Layer], [$srcdir/VERSION],

m4_ifdef([project_ompi],
[AC_SUBST(libmpi_so_version)
AC_SUBST(libmpi_abi_so_version)
AC_SUBST(libmpi_mpifh_so_version)
AC_SUBST(libmpi_usempi_tkr_so_version)
AC_SUBST(libmpi_usempi_ignore_tkr_so_version)
Expand Down
Loading