Remove remaining git submodules (JeanPhilippeKernel#494)

- Remove fmt and spdlog submodules that we're not removed in the
  previous commit

Co-authored-by: Mathew Benson <mathew@benson.co.ke>

Author: MathewBensonCode