Recursive initialization of repos

[RemonComputer]

📚 The doc issue

I was following the tutorial from Building Executorch from source

I used this command to clone the submodules

# Update and pull submodules
git submodule sync
git submodule update --init

Then following this tutorial Qualcomm AI Engine backend, I tried to build the qualcomm backend using

./backends/qualcomm/scripts/build.sh

but there were some missing modules from executorch/extension/llm/tokenizers/third-party/

Suggest a potential alternative/fix

I suggest you add the --recursive to the documentation so that the documentation could be

# Update and pull submodules
git submodule sync
git submodule update --init --recursive

cc @larryliu0820 @jathu

[tarbaig]

Looks like a good point to start contributing, can I take this ?

[GregoryComer]

Apologies for the slow response. @jathu do you have any thoughts on recursively cloning? It may pull more than needed - I don't want to regress the clone time much. But it sounds like the tokenizers need it, at least. Also CC @larryliu0820

[GregoryComer]

Tentatively assigning to @jathu but feel free to let me know if you're not the right owner.

[jathu]

but there were some missing modules from executorch/extension/llm/tokenizers/third-party/

@RemonComputer can you paste the error you're seeing?

[RemonComputer]

@jathu
I cannot remember the error, it was a while ago,
but I think it was an issue with CMake compilation,
it was complaining that while adding some directories it didn't contain a CMakeLists.txt files,
and when I checked those, those were empty directories (it hasn't been cloned due to no recursive cloning),
I think the directories were in the third-party directory inside the tokenizer directory

[jathu]

@RemonComputer I'm unable to repro this issue, so it would be helpful if you can repro it and paste the exact steps and/or the error logs

[RemonComputer]

@jathu

Here are the steps:

mkdir reproduce_error
cd reproduce_error/
# Clone the ExecuTorch repo from GitHub
git clone -b viable/strict https://github.com/pytorch/executorch.git && cd executorch
conda create -yn rep_error python=3.10.0 && conda activate rep_error
git submodule sync
git submodule update --init
./install_executorch.sh --clean
 (mkdir cmake-out && cd cmake-out && cmake ..)
cmake --build cmake-out -j16
./install_executorch.sh --clean
./backends/qualcomm/scripts/build.sh

Here is the error message:

...
optimized_kernels library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
optimized_portable_kernels library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
cpublas library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
eigen_blas library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
optimized_ops_lib library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
optimized_native_cpu_ops_lib library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
quantized_ops_aot_lib library is not found.
            If needed rebuild with the proper options in CMakeLists.txt
-- executorch: Using source file list /mnt/Data/reproduce_error/executorch/build-android/examples/qualcomm/../../executorch_srcs.cmake
-- Generating operator lib:
--   LIB_NAME: full_portable_ops_lib
--   OPS_SCHEMA_YAML: 
--   ROOT_OPS: 
--   INCLUDE_ALL_OPS: ON
Command - python;-m;codegen.tools.gen_oplist;--output_path=/mnt/Data/reproduce_error/executorch/build-android/examples/qualcomm/full_portable_ops_lib/selected_operators.yaml;--include_all_operators
-- Generating kernel bindings:
--   LIB_NAME: full_portable_ops_lib
--   FUNCTIONS_YAML: /mnt/Data/reproduce_error/executorch/examples/qualcomm/../../kernels/portable/functions.yaml
--   CUSTOM_OPS_YAML: 
--   ADD_EXCEPTION_BOUNDARY: FALSE
-- Generating operator lib:
--   LIB_NAME: full_portable_ops_lib
--   KERNEL_LIBS: portable_kernels
--   DEPS: executorch
CMake Error at CMakeLists.txt:77 (add_subdirectory):
  The source directory

    /mnt/Data/reproduce_error/executorch/extension/llm/tokenizers/third-party/abseil-cpp

  does not contain a CMakeLists.txt file.


CMake Error at CMakeLists.txt:81 (add_subdirectory):
  The source directory

    /mnt/Data/reproduce_error/executorch/extension/llm/tokenizers/third-party/re2

  does not contain a CMakeLists.txt file.


-- Configuring incomplete, errors occurred!

Note that latest install_executorch.sh clones the missing repos, so If you used it you wouldn't get the error

[jathu]

@RemonComputer I noticed two things:

1. It seems like you are not running ./install_requirements.sh (note this is required for ./install_executorch.sh)
2. When you run ./install_executorch.sh with --clean, it does not install executorch. Instead if cleans build artifacts and returns early:

   executorch/install_executorch.py

   Lines 214 to 216 in 7d497f1

   	if args.clean:
   	clean()
   	return

Can you try again by running install_requirements.sh first, then running install_executorch.sh without the --clean flag?

[RemonComputer]

@jathu
Using

./install_requirements.sh 
./install_executorch.sh
./backends/qualcomm/scripts/build.sh

produces the same error, not sure why