alembic 1.8.1 can't create initial migration on MacOS due to missing folder

[staticdev]

Describe the bug
I have a new project and followed the documentation doing alembic init alembic, edited the alembic.ini and env.py. When I try to run my initial migration I get a FileNotFoundError for alembic/versions.

Expected behavior
Created migration.

To Reproduce
Create a new project.

alembic init alembic
alembic revision --autogenerate -m "Initial migration"

Error

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
Traceback (most recent call last):
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/bin/alembic", line 8, in <module>
    sys.exit(main())
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/config.py", line 590, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/config.py", line 584, in main
    self.run_cmd(cfg, options)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/config.py", line 561, in run_cmd
    fn(
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/command.py", line 229, in revision
    script_directory.run_env()
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
    module = load_module_py(module_id, path)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/user/workspace/project/alembic/env.py", line 91, in <module>
    run_migrations_online()
  File "/Users/user/workspace/project/alembic/env.py", line 85, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/runtime/environment.py", line 853, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/runtime/migration.py", line 611, in run_migrations
    for step in self._migrations_fn(heads, self):
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/command.py", line 205, in retrieve_migrations
    revision_context.run_autogenerate(rev, context)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/autogenerate/api.py", line 526, in run_autogenerate
    self._run_environment(rev, migration_context, True)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/autogenerate/api.py", line 545, in _run_environment
    self.script_directory.get_revisions("heads")
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/script/base.py", line 296, in get_revisions
    self.revision_map.get_revisions(id_),
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/script/revision.py", line 529, in get_revisions
    resolved_id, branch_label = self._resolve_revision_number(
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/script/revision.py", line 753, in _resolve_revision_number
    self._revision_map
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 1113, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/script/revision.py", line 191, in _revision_map
    for revision in self._generator():
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/script/base.py", line 131, in _load_revisions
    for file_ in Script._list_py_dir(self, vers):
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/script/base.py", line 981, in _list_py_dir
    return os.listdir(path)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/user/workspace/project/alembic/versions'
(project-Y6OSQIHv-py3.10) user@UM01035 project % alembic revision -m "Create tenant table" --autogenerate
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
Traceback (most recent call last):
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/bin/alembic", line 8, in <module>
    sys.exit(main())
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/config.py", line 590, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/config.py", line 584, in main
    self.run_cmd(cfg, options)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/config.py", line 561, in run_cmd
    fn(
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/command.py", line 229, in revision
    script_directory.run_env()
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
    module = load_module_py(module_id, path)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/user/workspace/project/alembic/env.py", line 91, in <module>
    run_migrations_online()
  File "/Users/user/workspace/project/alembic/env.py", line 85, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/runtime/environment.py", line 853, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/runtime/migration.py", line 611, in run_migrations
    for step in self._migrations_fn(heads, self):
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/command.py", line 205, in retrieve_migrations
    revision_context.run_autogenerate(rev, context)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/autogenerate/api.py", line 526, in run_autogenerate
    self._run_environment(rev, migration_context, True)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/autogenerate/api.py", line 545, in _run_environment
    self.script_directory.get_revisions("heads")
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/script/base.py", line 296, in get_revisions
    self.revision_map.get_revisions(id_),
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/script/revision.py", line 529, in get_revisions
    resolved_id, branch_label = self._resolve_revision_number(
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/script/revision.py", line 753, in _resolve_revision_number
    self._revision_map
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 1113, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/script/revision.py", line 191, in _revision_map
    for revision in self._generator():
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/script/base.py", line 131, in _load_revisions
    for file_ in Script._list_py_dir(self, vers):
  File "/Users/user/Library/Caches/pypoetry/virtualenvs/project-Y6OSQIHv-py3.10/lib/python3.10/site-packages/alembic/script/base.py", line 981, in _list_py_dir
    return os.listdir(path)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/user/workspace/project/alembic/versions'

Versions.

• OS: MacOS 12.4
• Python: 3.10.5
• Alembic: 1.8.1
• SQLAlchemy: 1.4.39
• Database: Postgres 14
• DBAPI: psycopg2

Have a nice day!

[zzzeek]

hi -

there's likely some change you've made in your alembic.ini or env.py that's affecting things. since it can't find 'versions', likely something with "script_location" or "version_locations".

try using a fresh start, dont edit any files whatsoever. then make sure "alembic revision -m 'some rev'" creates a file.

here's what it looks like:

(.venv) [classic@photon3 tmp]$ alembic init alembic
  Creating directory /home/classic/tmp/alembic ...  done
  Creating directory /home/classic/tmp/alembic/versions ...  done
  Generating /home/classic/tmp/alembic/script.py.mako ...  done
  Generating /home/classic/tmp/alembic/README ...  done
  Generating /home/classic/tmp/alembic.ini ...  done
  Generating /home/classic/tmp/alembic/env.py ...  done
  Please edit configuration/connection/logging settings in '/home/classic/tmp/alembic.ini' before proceeding.
(.venv) [classic@photon3 tmp]$ alembic revision -m "Initial migration"
  Generating /home/classic/tmp/alembic/versions/ae8efe518f87_initial_migration.py ...  done

then run alembic from that same directory and make sure script_location points to "alembic".

[staticdev]

@zzzeek thanks for your response. Regarding your observations:

1. in alembic.ini only change I did was remove the line sqlalchemy.sql. So script_location = alembic and version_locations is commented.
2. in env.py I only added the import for os and application metadata, and a method get_url called on run_migrations_offline and .._online.

I just tried again in a new test project and reproduced the same error result with alembic revision --autogenerate -m "Initial migration".
If I create the folder alembic/versions manually and run: alembic revision --autogenerate -m "Initial migration" the error goes away.

[zzzeek]

can you run the command and then show me the output? that is, we should see:

 Creating directory /home/classic/tmp/alembic/versions ...  done

then when the output completes, stop. is that directory then present? is the "alembic" parent directory present ? it would be extremely strange that "versions" simply didnt exist, after it successfully built the parent directory and then ran the exact same command to create the subdirectory?

[staticdev]

@zzzeek after some investigation I discovered what was happening.. alembic init was correctly creating this empty folder alembic/versions, and I had a git commit in between but git simply ignores empty folders. If I do a migration in a step after where I cloned the repo initialized but not with a migration the alembic revision will fail because the cloned repo will not have the folder. This is very edge case but it is related to how git behaves, not really related to MacOS. To make the behavior of first migration more bullet-proof it could just create versions folder in case of FileNotFoundError. Do you think that would make sense?

Other options:

1. ignore it, but this can happen in the future with another user working maybe with a team or in more complex CI/CD environment (low probability)
2. on alembic init create also in an empty versions/__init__.py. This option is not so idiot-proof because some developer can always delete a folder with just an empty __init__.py thinking it is cleaning up.

If you agree with my proposal I am happy to make a PR for it.

[zzzeek]

2. on `alembic init` create also in an empty `versions/__init__.py`. This option is not so idiot-proof because some developer can always delete a folder with just an empty `__init__.py` thinking it is cleaning up.

If you agree with my proposal I am happy to make a PR for it.

no need, call alembic init --package, will add the empty __init__.py file.

[staticdev]

This is why I like using mature software. Congratz, great lib @zzzeek !

[CaselIT]

would creating a .gitkeep file inside the versions folder be something appropriate when not called with --package?