Lightning docker image's s3 support is broken: 'AioClientCreator' object has no attribute '_register_lazy_block_unknown_fips_pseudo_regions'

[akyrola]

Hello,

I was previously using docker image pytorchlightning/pytorch_lightning happily and submitting s3:// urls for tensorboard and checkpoint directories. However, this week this is failing due to apparent backward or forward incompatibility issue with boto (?).

I do run "pip install boto s3fs" on the docker image, as those are not present on the image (I think they should be).

I wonder if you have any advice?

Traceback (most recent call last):
   File "/usr/local/lib/python3.9/dist-packages/pytorch_lightning/loggers/tensorboard.py", line 229, in log_metrics
     self.experiment.add_scalar(k, v, step)
   File "/usr/local/lib/python3.9/dist-packages/pytorch_lightning/loggers/base.py", line 43, in experiment
     return get_experiment() or DummyExperiment()
   File "/usr/local/lib/python3.9/dist-packages/pytorch_lightning/utilities/distributed.py", line 49, in wrapped_fn
      return fn(args, kwargs)
    File "/usr/local/lib/python3.9/dist-packages/pytorch_lightning/loggers/base.py", line 41, in get_experiment
      return fn(self)
    File "/usr/local/lib/python3.9/dist-packages/pytorch_lightning/loggers/tensorboard.py", line 172, in experiment
      self._fs.makedirs(self.root_dir, exist_ok=True)
    File "/usr/local/lib/python3.9/dist-packages/fsspec/asyn.py", line 91, in wrapper
      return sync(self.loop, func, args, kwargs)
    File "/usr/local/lib/python3.9/dist-packages/fsspec/asyn.py", line 71, in sync
      raise return_result
    File "/usr/local/lib/python3.9/dist-packages/fsspec/asyn.py", line 25, in _runner
      result[0] = await coro
    File "/usr/local/lib/python3.9/dist-packages/s3fs/core.py", line 733, in _makedirs
      await self._mkdir(path, create_parents=True)
    File "/usr/local/lib/python3.9/dist-packages/s3fs/core.py", line 718, in _mkdir
      await self._call_s3("create_bucket", params)
    File "/usr/local/lib/python3.9/dist-packages/s3fs/core.py", line 241, in _call_s3
      await self.set_session()
    File "/usr/local/lib/python3.9/dist-packages/s3fs/core.py", line 409, in set_session
      self._s3 = await s3creator.__aenter__()
    File "/usr/local/lib/python3.9/dist-packages/aiobotocore/session.py", line 37, in __aenter__
      self._client = await self._coro
    File "/usr/local/lib/python3.9/dist-packages/aiobotocore/session.py", line 121, in _create_client
      client = await client_creator.create_client(
    File "/usr/local/lib/python3.9/dist-packages/aiobotocore/client.py", line 46, in create_client
      self._register_lazy_block_unknown_fips_pseudo_regions(service_client)
  AttributeError: 'AioClientCreator' object has no attribute '_register_lazy_block_unknown_fips_pseudo_regions'

[akyrola]

Found out that this works:

pip install s3fs botocore==1.22.5

Found it from here:
https://stackoverflow.com/questions/69994834/attributeerror-aioclientcreator-object-has-no-attribute-register-lazy-block/70028682

Might be good idea to include the boto and s3sf on the docker image?