Conflicts of blas/lapack setting when using openfast with simulink in Ubuntu18.04

[isablanca]

Dear all,

I have encountered some error when running openfast with simulink. My system os is Ubuntu18.04, matlab version is R2021b (It can't work in R2020b either). I have tried two ways to install the openfast.

Method1:

I follow the document, compile openfast use

cmake .. -DBLAS_LIBRARIES=/usr/lib/x86_64-linux-gnu/openblas/libblas.so -DLAPACK_LIBRARIES=/usr/lib/x86_64-linux-gnu/openblas/liblapack.so -DBUILD_SHARED_LIBS=ON -DBUILD_OPENFAST_SIMULINK_API=ON
make -j
make install

Then, create_FAST_SFunc.m runs successfully.

But, when I run Run_OpenLoop.m, the matlab crashed, with following message shown in the terminal

Intel MKL ERROR: Parameter 4 was incorrect on entry to DGETRF.

It looks like the problem mentioned in https://github.com/OpenFAST/openfast/issues/730#issuecomment-826921225,
but the solution didn't work for me as shown in the method 2.

Method 2

I try to use the method shown in the post shown in #730

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<path to openfast/install/lib>
export BLAS_VERSION=libopenblas.so
export LAPACK_VERSION=liblapacke.so
matlab

The both two openfast examples can run without error, but the above modification make some builtin function provided by Matlab not working, like c2d command with out of memory error, looks like an error posted in https://se.mathworks.com/matlabcentral/answers/522196-matlab-lapack-blas-configuration.

I am not sure but for me it looks like when compile the openfast, I didn't give a correct setting about blas and lapack packages, but I don't know how to do.

[Seager1989]

You may refer to #556 for a possible solution?

[isablanca]

The script

export BLAS_VERSION=libopenblas.so 
export LAPACK_VERSION=libopenblas.so 
matlab

not working, though it can make the example model run without error, it also causes errors when matlab runs its builtin function. Thus, it is not a good solution when some builtin matlab functions are involved in the simulink model.

So do you have any suggestion that we can run the openfast simulink model while keeping the default blas/lapack setting of matlab so that the builtin function of matlab is not affected? @bjonkman @rafmudaf

[reos-rcrozier]

Our pull request #1010 would address your issue

[AaronTUB]

Dear all,

I'm experiencing the same problem as described above. For my case, the solution

export BLAS_VERSION=libopenblas.so 
export LAPACK_VERSION=libopenblas.so 
matlab

only works, if I call simple matlab functions that do not involve matlab's built-in BLAS/LAPACK. If I try to used functions like lqr I get an Out of memory Error.

Unfortunately @reos-rcrozier, the pull request does not fix the issue.

@isablanca could you find a solution for the problem?

[reos-rcrozier]

Hmmm, yes, I've seen this, but not isolated the reason yet. Can you try make -j1 and see if it helps?

[AaronTUB]

Before building with make -j1 I deleted everything inside the build folder and ran CMAKE again:

cmake .. -DBUILD_SHARED_LIBS=ON -DBUILD_OPENFAST_SIMULINK_API=ON -DUSE_LOCAL_STATIC_LAPACK=ON

Then the compiling works much better, but almost at the end, I still get the following error:

...
Scanning dependencies of target openfastlib
[ 97%] Building Fortran object modules/openfast-library/CMakeFiles/openfastlib.dir/src/FAST_Library.f90.o
[ 97%] Linking CXX shared library libopenfastlib.so
/usr/bin/ld: ../map/CMakeFiles/maplib_obj.dir/src/bstring/bstrlib.c.o: relocation R_X86_64_PC32 against symbol `binstr' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
make[2]: *** [modules/openfast-library/CMakeFiles/openfastlib.dir/build.make:485: modules/openfast-library/libopenfastlib.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:2281: modules/openfast-library/CMakeFiles/openfastlib.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

[AaronTUB]

After looking around a little bit, it seems the option -fPIC is required. So I add the following three lines to the CMakeLists.txt at line 90:

set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fPIC")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fPIC")
set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -fPIC")

(I have absolutely no experience with the creation of CMakeLists.txt files, so I added the option to all three flags. I'm sure there is a simpler and better way to do it.)
Deleting everything from the build directory and running CMAKE with:

cmake .. -DBUILD_SHARED_LIBS=ON -DBUILD_OPENFAST_SIMULINK_API=ON -DUSE_LOCAL_STATIC_LAPACK=ON

again, then compiling with:

make -j1

and installing with:

make install

ran through without any issues. Then I went on to building the mexa64 file by navigating into the right directory:
cd ../glue-codes/simulink/src

and double-checking the paths in create_FAST_SFUNC.m, but everything seemed fine. Running the script with:
matlab -r "create_FAST_SFunc"

resulted in the following output:

MATLAB is selecting SOFTWARE OPENGL rendering.

                                                    < M A T L A B (R) >
                                          Copyright 1984-2021 The MathWorks, Inc.
                                     R2021b Update 1 (9.11.0.1809720) 64-bit (glnxa64)
                                                      November 2, 2021

 
To get started, type doc.
For product information, visit www.mathworks.com.
 

----------------------------
Creating ./FAST_SFunc.mexa64

Verbose mode is on.
No MEX options file identified; looking for an implicit selection.
... Looking for compiler 'gcc' ...
... Executing command 'which gcc' ...Yes ('/usr/bin/gcc').
... Executing command 'gcc -dumpversion' ...Yes ('9').
... Executing command 'gcc -print-file-name=libstdc++.so' ...Yes ('/usr/lib/gcc/x86_64-linux-gnu/9/libstdc++.so').
Found installed compiler 'gcc'.
Options file details
-------------------------------------------------------------------
	Compiler location: /usr/bin/gcc
	Options file: /home/aaron/MATLAB/R2021b/bin/glnxa64/mexopts/gcc_glnxa64.xml
	CMDLINE2 : /usr/bin/gcc -pthread -Wl,--no-undefined -Wl,-rpath-link,/home/aaron/MATLAB/R2021b/bin/glnxa64 -shared  -O -Wl,--version-script,"/home/aaron/MATLAB/R2021b/extern/lib/glnxa64/c_exportsmexfileversion.map" /tmp/mex_483637799316_1796/FAST_SFunc.o /tmp/mex_483637799316_1796/c_mexapi_version.o   -lopenfastlib  -lgfortran  -lquadmath  -llapack  -lblas  -ldl   -L../../../install/lib   -L"/home/aaron/MATLAB/R2021b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -o /home/aaron/Documents/OpenFAST/glue-codes/simulink/src/./FAST_SFunc.mexa64
	CC : /usr/bin/gcc
	DEFINES : -DMX_COMPAT_64  -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE 
	MATLABMEX : -DMATLAB_MEX_FILE 
	CFLAGS : -fexceptions -fPIC -fno-omit-frame-pointer -pthread -DS_FUNCTION_NAME=FAST_SFunc
	INCLUDE : -I"../../../install/include"  -I"/home/aaron/MATLAB/R2021b/extern/include" -I"/home/aaron/MATLAB/R2021b/simulink/include"
	COPTIMFLAGS : -O2 -fwrapv -DNDEBUG
	CDEBUGFLAGS : -g
	LD : /usr/bin/gcc
	LDFLAGS : -pthread -Wl,--no-undefined -Wl,-rpath-link,/home/aaron/MATLAB/R2021b/bin/glnxa64
	LDTYPE : -shared 
	FUNCTIONMAP : "/home/aaron/MATLAB/R2021b/extern/lib/glnxa64/mexFunction.map"
	VERSIONMAP : "/home/aaron/MATLAB/R2021b/extern/lib/glnxa64/c_exportsmexfileversion.map"
	LINKEXPORT : -Wl,--version-script,"/home/aaron/MATLAB/R2021b/extern/lib/glnxa64/mexFunction.map"
	LINKEXPORTVER : -Wl,--version-script,"/home/aaron/MATLAB/R2021b/extern/lib/glnxa64/c_exportsmexfileversion.map"
	LINKLIBS : -lopenfastlib  -lgfortran  -lquadmath  -llapack  -lblas  -ldl   -L../../../install/lib   -L"/home/aaron/MATLAB/R2021b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++
	LDOPTIMFLAGS : -O
	LDDEBUGFLAGS : -g
	MWCPPLIB : "/home/aaron/MATLAB/R2021b/sys/os/glnxa64/orig/libstdc++.so.6"
	OBJEXT : .o
	LDEXT : .mexa64
	SETENV : CC="/usr/bin/gcc"
                CXX="g++"
                CFLAGS="-fexceptions -fPIC -fno-omit-frame-pointer -pthread -DS_FUNCTION_NAME=FAST_SFunc -DMX_COMPAT_64  -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE "
                CXXFLAGS="-fexceptions -fPIC -fno-omit-frame-pointer -pthread -DS_FUNCTION_NAME=FAST_SFunc -std=c++11 -DMX_COMPAT_64  -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE "
                COPTIMFLAGS="-O2 -fwrapv -DNDEBUG"
                CXXOPTIMFLAGS="-O2 -fwrapv -DNDEBUG"
                CDEBUGFLAGS="-g"
                CXXDEBUGFLAGS="-g"
                MW_GLIBC_SHIM="$MW_GLIBC_SHIM"
                LD="/usr/bin/gcc"
                LDXX="g++"
                LDFLAGS="-pthread -Wl,--no-undefined -Wl,-rpath-link,/home/aaron/MATLAB/R2021b/bin/glnxa64 -shared  -lopenfastlib  -lgfortran  -lquadmath  -llapack  -lblas  -ldl   -L../../../install/lib   -L"/home/aaron/MATLAB/R2021b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -Wl,--version-script,"/home/aaron/MATLAB/R2021b/extern/lib/glnxa64/mexFunction.map""
                LDDEBUGFLAGS="-g"
	GCC : /usr/bin/gcc
	VER : 9
	CPPLIBS : /usr/lib/gcc/x86_64-linux-gnu/9/libstdc++.so
	MATLABROOT : /home/aaron/MATLAB/R2021b
	ARCH : glnxa64
	SRC : "/home/aaron/Documents/OpenFAST/glue-codes/simulink/src/FAST_SFunc.c";"/home/aaron/MATLAB/R2021b/extern/version/c_mexapi_version.c"
	OBJ : /tmp/mex_483637799316_1796/FAST_SFunc.o;/tmp/mex_483637799316_1796/c_mexapi_version.o
	OBJS : /tmp/mex_483637799316_1796/FAST_SFunc.o /tmp/mex_483637799316_1796/c_mexapi_version.o 
	SRCROOT : /home/aaron/Documents/OpenFAST/glue-codes/simulink/src/FAST_SFunc
	DEF : /tmp/mex_483637799316_1796/FAST_SFunc.def
	EXP : "/home/aaron/Documents/OpenFAST/glue-codes/simulink/src/./FAST_SFunc.exp"
	LIB : "/home/aaron/Documents/OpenFAST/glue-codes/simulink/src/./FAST_SFunc.lib"
	EXE : /home/aaron/Documents/OpenFAST/glue-codes/simulink/src/./FAST_SFunc.mexa64
	ILK : "/home/aaron/Documents/OpenFAST/glue-codes/simulink/src/./FAST_SFunc.ilk"
	MANIFEST : "/home/aaron/Documents/OpenFAST/glue-codes/simulink/src/./FAST_SFunc.mexa64.manifest"
	TEMPNAME : /home/aaron/Documents/OpenFAST/glue-codes/simulink/src/./FAST_SFunc
	EXEDIR : /home/aaron/Documents/OpenFAST/glue-codes/simulink/src/./
	EXENAME : FAST_SFunc
	OPTIM : -O2 -fwrapv -DNDEBUG
	LINKOPTIM : -O
	CMDLINE1_0 : /usr/bin/gcc -c -DMX_COMPAT_64  -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE  -I"../../../install/include"  -I"/home/aaron/MATLAB/R2021b/extern/include" -I"/home/aaron/MATLAB/R2021b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -DS_FUNCTION_NAME=FAST_SFunc -O2 -fwrapv -DNDEBUG "/home/aaron/Documents/OpenFAST/glue-codes/simulink/src/FAST_SFunc.c" -o /tmp/mex_483637799316_1796/FAST_SFunc.o
	CMDLINE1_1 : /usr/bin/gcc -c -DMX_COMPAT_64  -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE  -I"../../../install/include"  -I"/home/aaron/MATLAB/R2021b/extern/include" -I"/home/aaron/MATLAB/R2021b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -DS_FUNCTION_NAME=FAST_SFunc -O2 -fwrapv -DNDEBUG "/home/aaron/MATLAB/R2021b/extern/version/c_mexapi_version.c" -o /tmp/mex_483637799316_1796/c_mexapi_version.o
-------------------------------------------------------------------
Building with 'gcc'.
/usr/bin/gcc -c -DMX_COMPAT_64  -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE  -I"../../../install/include"  -I"/home/aaron/MATLAB/R2021b/extern/include" -I"/home/aaron/MATLAB/R2021b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -DS_FUNCTION_NAME=FAST_SFunc -O2 -fwrapv -DNDEBUG "/home/aaron/Documents/OpenFAST/glue-codes/simulink/src/FAST_SFunc.c" -o /tmp/mex_483637799316_1796/FAST_SFunc.o
/usr/bin/gcc -c -DMX_COMPAT_64  -DMATLAB_DEFAULT_RELEASE=R2017b  -DUSE_MEX_CMD   -D_GNU_SOURCE -DMATLAB_MEX_FILE  -I"../../../install/include"  -I"/home/aaron/MATLAB/R2021b/extern/include" -I"/home/aaron/MATLAB/R2021b/simulink/include" -fexceptions -fPIC -fno-omit-frame-pointer -pthread -DS_FUNCTION_NAME=FAST_SFunc -O2 -fwrapv -DNDEBUG "/home/aaron/MATLAB/R2021b/extern/version/c_mexapi_version.c" -o /tmp/mex_483637799316_1796/c_mexapi_version.o
/usr/bin/gcc -pthread -Wl,--no-undefined -Wl,-rpath-link,/home/aaron/MATLAB/R2021b/bin/glnxa64 -shared  -O -Wl,--version-script,"/home/aaron/MATLAB/R2021b/extern/lib/glnxa64/c_exportsmexfileversion.map" /tmp/mex_483637799316_1796/FAST_SFunc.o /tmp/mex_483637799316_1796/c_mexapi_version.o   -lopenfastlib  -lgfortran  -lquadmath  -llapack  -lblas  -ldl   -L../../../install/lib   -L"/home/aaron/MATLAB/R2021b/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -o /home/aaron/Documents/OpenFAST/glue-codes/simulink/src/./FAST_SFunc.mexa64
MEX completed successfully.

Then moving the mexa64 file into the examples folder and running the model with:
matlab -r "Run_OpenLoop;exit"

Results in the following segmentation violation:

MATLAB is selecting SOFTWARE OPENGL rendering.

                                                    < M A T L A B (R) >
                                          Copyright 1984-2021 The MathWorks, Inc.
                                     R2021b Update 1 (9.11.0.1809720) 64-bit (glnxa64)
                                                      November 2, 2021

 
To get started, type doc.
For product information, visit www.mathworks.com.
 

 **************************************************************************************************
 OpenFAST

 Copyright (C) 2022 National Renewable Energy Laboratory
 Copyright (C) 2022 Envision Energy USA LTD

 This program is licensed under Apache License Version 2.0 and comes with ABSOLUTELY NO WARRANTY.
 See the "LICENSE" file distributed with this software for details.
 **************************************************************************************************

 OpenFAST-v3.1.0-18-g73614b03-dirty
 Compile Info:
  - Compiler: GCC version 9.4.0
  - Architecture: 64 bit
  - Precision: double
  - OpenMP: No
  - Date: Apr 19 2022
  - Time: 14:56:27
 Execution Info:
  - Date: 04/19/2022
  - Time: 18:31:01+0200

 OpenFAST input file heading:
     FAST Certification Test #06: AOC 15/50 with many DOFs with gen start, loss of grid, and
     tip-brake shutdown. Many parameters are pure fiction.

 Running ElastoDyn.
 Nodal outputs section of ElastoDyn input file not found or improperly formatted.
 Running AeroDyn14.
   WARNING:   AeroDyn 14 is deprecated and will be removed in a future release.
 Running InflowWind.
 Running ServoDyn.

 FAST_InitializeAll: IfW_Init:IfW_UniformWind_Init: Could not read upflow column in uniform wind
 files. Assuming upflow is 0.
 FAST_InitializeAll:SrvD_Init:ValidatePrimaryData:Yaw angle and rate are not commanded from
 Simulink model.
 ValidatePrimaryData:Pitch angles are not commanded from Simulink model.
 ValidatePrimaryData:Generator torque and power are not commanded from Simulink model.
 ValidatePrimaryData:HSS brake is not commanded from Simulink model.
 ValidatePrimaryData:Airfoil control is not commanded from Simulink model.
 ValidatePrimaryData:Cable control is not commanded from Simulink model.

  Time: 0 of 60 seconds.

Intel MKL ERROR: Parameter 4 was incorrect on entry to DGETRF.


  OpenFAST completed.

Error using Run_OpenLoop (line 12)
Error reported by S-function 'FAST_SFunc' in 'OpenLoop/FAST Nonlinear Wind Turbine/S-Function':
FAST_Solution0:CalcOutputs_And_SolveForInputs:SolveOption2:SolveOption2a_Inp2BD:ED_CalcContStateDeriv:LAPACK_DGETRF: illegal
value in argument           -4.

As before, this error could be fixed by setting the environment variables BLAS_VERSION and LAPACK_VERSION but that leads to issues with some builtin MATLAB functions. Do you have any ideas/suggestions?

[AaronTUB]

Hi everyone,
I apologize for my impatience but I really need to solve this problem.

@isablanca @rafmudaf @reos-rcrozier do you have any idea where I'm making a mistake or how it could be fixed?

[reos-rcrozier]

try -DBUILD_SHARED_LIBS=OFF with my PR I mean.