Fix: Newly created GitLab instance fails to mount data volume (#4890)#7757
Open
achave11-ucsc wants to merge 5 commits intodevelopfrom
Open
Fix: Newly created GitLab instance fails to mount data volume (#4890)#7757achave11-ucsc wants to merge 5 commits intodevelopfrom
achave11-ucsc wants to merge 5 commits intodevelopfrom
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #7757 +/- ##
===========================================
- Coverage 84.82% 84.81% -0.01%
===========================================
Files 157 157
Lines 23145 23148 +3
===========================================
+ Hits 19632 19634 +2
- Misses 3513 3514 +1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
achave11-ucsc
force-pushed
the
issues/achave11-ucsc/4890-fix-gitlab-mounting
branch
from
06d2758 to
e8db5a7
Compare
achave11-ucsc
added
the
deploy:gitlab
achave11-ucsc
force-pushed
the
issues/achave11-ucsc/4890-fix-gitlab-mounting
branch
4 times, most recently
from
998fffd to
9cf5005
Compare
achave11-ucsc
force-pushed
the
issues/achave11-ucsc/4890-fix-gitlab-mounting
branch
from
e417fc7 to
15ea497
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Member
Author
|
Tested the recommended solution which appears to work as intended. Correctly mounting the GitLab data volume. SSH into the instance after initial reboot: |
achave11-ucsc
force-pushed
the
issues/achave11-ucsc/4890-fix-gitlab-mounting
branch
3 times, most recently
from
e539175 to
9fe2481
Compare
nadove-ucsc
requested changes
Feb 12, 2026
Contributor
nadove-ucsc
left a comment
nadove-ucsc
left a comment
There was a problem hiding this comment.
Changes look good but I think there are issues with the history.
achave11-ucsc
force-pushed
the
issues/achave11-ucsc/4890-fix-gitlab-mounting
branch
2 times, most recently
from
8cd5e39 to
0045615
Compare
nadove-ucsc
previously approved these changes
Feb 12, 2026
hannes-ucsc
requested changes
Feb 12, 2026
environment.py
Outdated
| # instance. This volume is mounted at `/mnt/gitlab` and contains all of | ||
| # GitLab's data and configuration. The volume is mounted by UUID rather | ||
| # than device path (e.g., `/dev/nvme1n1`) to ensure reliable mounting | ||
| # across instance types and reboots, as device names can vary. This |
Member
There was a problem hiding this comment.
Suggested change
| # across instance types and reboots, as device names can vary. This | |
| # across instance types and reboots, as device names can vary and may be attached in a non-deterministic order. This |
Member
There was a problem hiding this comment.
Also don't duplicate this. I think you currently mention it three times. Consolidate.
environment.py
Outdated
| # GitLab's data and configuration. The volume is mounted by UUID rather | ||
| # than device path (e.g., `/dev/nvme1n1`) to ensure reliable mounting | ||
| # across instance types and reboots, as device names can vary. This | ||
| # variable is only used in GitLab component deployments. See the |
Member
There was a problem hiding this comment.
Suggested change
| # variable is only used in GitLab component deployments. See the | |
| # variable is only used in the `.gitlab` component. See the |
OPERATOR.rst
Outdated
| df # Verify device /dev/nvme1n1 is mounted on /mnt/gitlab, note available size | ||
| sudo resize2fs /dev/nvme1n1 | ||
| df | grep /mnt/gitlab # Verify the device mount (/dev/nvme1n1) and note available size | ||
| sudo resize2fs /dev/nvme1n1 # This should match the filesystem of the previous output |
Member
There was a problem hiding this comment.
Suggested change
| sudo resize2fs /dev/nvme1n1 # This should match the filesystem of the previous output | |
| sudo resize2fs /dev/nvme1n1 # Match the device name emitted by the previous command |
| # sudo mkdir /mnt/gitlab | ||
| # sudo mkfs.ext4 /dev/nvme1n1 | ||
| # | ||
| # After formatting the volume, obtain its filesystem UUID and set it as value |
Member
There was a problem hiding this comment.
Suggested change
| # After formatting the volume, obtain its filesystem UUID and set it as value | |
| # After formatting the volume, obtain its filesystem UUID and set it as the value |
| # sudo mkfs.ext4 /dev/nvme1n1 | ||
| # | ||
| # After formatting the volume, obtain its filesystem UUID and set it as value | ||
| # of `azul_gitlab_data_volume_id` in the GitLab deployment's environment.py. |
Member
There was a problem hiding this comment.
Suggested change
| # of `azul_gitlab_data_volume_id` in the GitLab deployment's environment.py. | |
| # of `azul_gitlab_data_volume_id` in the `.gitlab` component's `environment.py`. |
Comment on lines
155
to
156
Member
There was a problem hiding this comment.
Suggested change
| # into the Gitlab instance, format `/dev/nvme1n1`. Occasionally, the device name may be different, `/dev/nvme0n1`. The invariant is that there should be two controller devices, one for the root volume and one for the data volume. Determine the controller device name to which the root volume is attached and then simply use the other controller device name. Reboot the instance. For example: |
Comment on lines
274
to
276
| # Cloud-init's cc_mounts module does not support the UUID=<uuid> device | ||
| # specification format, even though Amazon Linux /etc/fstab and the mount | ||
| # command do support it. We use /dev/disk/by-uuid/ symlink path as a |
Member
There was a problem hiding this comment.
Suggested change
| # Cloud-init's cc_mounts module does not support the UUID=<uuid> device | |
| # specification format, even though Amazon Linux /etc/fstab and the mount | |
| # command do support it. We use /dev/disk/by-uuid/ symlink path as a | |
| # Cloud-init's cc_mounts module does not support the UUID=<uuid> device | |
| # specification format. We use /dev/disk/by-uuid/ symlink path as a |
| # Cloud-init's cc_mounts module does not support the UUID=<uuid> device | ||
| # specification format, even though Amazon Linux /etc/fstab and the mount | ||
| # command do support it. We use /dev/disk/by-uuid/ symlink path as a | ||
| # workaround, relying on udev to create the symbolic link once the device |
Member
There was a problem hiding this comment.
Suggested change
| # workaround, relying on udev to create the symbolic link once the device | |
| # workaround, relying on udev to create the symlink once the device |
| # workaround, relying on udev to create the symbolic link once the device | ||
| # is attached. | ||
| # | ||
| gitlab_storage_device_uuid = f'/dev/disk/by-uuid/{config.gitlab_data_volume_id}' |
Member
There was a problem hiding this comment.
Suggested change
| gitlab_storage_device_uuid = f'/dev/disk/by-uuid/{config.gitlab_data_volume_id}' | |
| data_volume_device = f'/dev/disk/by-uuid/{config.gitlab_data_volume_id}' |
GitLab is implied in this context. This is a path, or device name, or device path, not a UUID. It contains a UUID, but it is not a UUID.
| f'do echo "{gitlab_storage_device_uuid} does not exist, sleeping 1s"', | ||
| 'sleep 1', | ||
| 'done' | ||
| f'done && echo "GL Volume Device: $(readlink -f {gitlab_storage_device_uuid})"' |
Member
There was a problem hiding this comment.
Suggested change
| f'done && echo "GL Volume Device: $(readlink -f {gitlab_storage_device_uuid})"' | |
| f'done && echo "Data volume attached to $(readlink -f {gitlab_storage_device_uuid})"' |
achave11-ucsc
force-pushed
the
issues/achave11-ucsc/4890-fix-gitlab-mounting
branch
from
ea117eb to
0b394e8
Compare
achave11-ucsc
force-pushed
the
issues/achave11-ucsc/4890-fix-gitlab-mounting
branch
from
0b394e8 to
6153024
Compare
hannes-ucsc
requested changes
Feb 14, 2026
| # into the Gitlab instance, format `/dev/xvdf` (`/dev/nvme1n1` on newer | ||
| # instance types) and reboot the instance. For example: | ||
| # into the Gitlab instance, format `/dev/nvme1n1`. Occasionally, the device name | ||
| # may be different, `/dev/nvme0n1`. The invariant is that there should be two |
Member
There was a problem hiding this comment.
Suggested change
| # may be different, `/dev/nvme0n1`. The invariant is that there should be two | |
| # may be different, like `/dev/nvme0n1`. The invariant is that there should be two |
| # into the Gitlab instance, format `/dev/xvdf` (`/dev/nvme1n1` on newer | ||
| # instance types) and reboot the instance. For example: | ||
| # into the Gitlab instance, format `/dev/nvme1n1`. Occasionally, the device name | ||
| # may be different, `/dev/nvme0n1`. The invariant is that there should be two |
Member
There was a problem hiding this comment.
Suggested change
| # may be different, `/dev/nvme0n1`. The invariant is that there should be two | |
| # may be different, like `/dev/nvme0n1`. The invariant is that there should be two |
| # may be different, `/dev/nvme0n1`. The invariant is that there should be two | ||
| # controller devices, one for the root volume and one for the data volume. | ||
| # Determine the controller device name to which the root volume is attached and | ||
| # then simply use the other controller device name. Reboot the instance. For |
Member
There was a problem hiding this comment.
Suggested change
| # then simply use the other controller device name. Reboot the instance. For | |
| # then simply use the other controller device name. For |
Comment on lines
+169
to
188
| # After formatting the volume, obtain its filesystem UUID and set it as the | ||
| # value of `azul_gitlab_data_volume_id` in the `.gitlab` component's | ||
| # `environment.py`. | ||
| # | ||
| # To get the UUID (or see below if device is already mounted), run: | ||
| # | ||
| # sudo lsblk -f | ||
| # | ||
| # Identify the UUID for the device named `nvme1n1` (which should match the UUID | ||
| # for the `/mnt/gitlab` mountpoint). | ||
| # | ||
| # Or if the device is already mounted, run: | ||
| # | ||
| # sudo blkid /dev/nvme1n1 | ||
| # | ||
| # Set this UUID to the `azul_gitlab_data_volume_id` variable. Once the UUID is | ||
| # configured, you can reboot the instance: | ||
| # | ||
| # sudo reboot | ||
| # sudo rm -rf /mnt/gitlab.deleteme |
Member
There was a problem hiding this comment.
Suggested change
| # After formatting the volume, obtain its filesystem UUID and set it as the | |
| # value of `azul_gitlab_data_volume_id` in the `.gitlab` component's | |
| # `environment.py`. | |
| # | |
| # To get the UUID (or see below if device is already mounted), run: | |
| # | |
| # sudo lsblk -f | |
| # | |
| # Identify the UUID for the device named `nvme1n1` (which should match the UUID | |
| # for the `/mnt/gitlab` mountpoint). | |
| # | |
| # Or if the device is already mounted, run: | |
| # | |
| # sudo blkid /dev/nvme1n1 | |
| # | |
| # Set this UUID to the `azul_gitlab_data_volume_id` variable. Once the UUID is | |
| # configured, you can reboot the instance: | |
| # | |
| # sudo reboot | |
| # sudo rm -rf /mnt/gitlab.deleteme | |
| # After formatting the volume, obtain its UUID and set it as the | |
| # value of `azul_gitlab_data_volume_id` in the `.gitlab` component's | |
| # `environment.py`. | |
| # | |
| # To get the UUID of an unmounted volume, run | |
| # | |
| # sudo lsblk -f | |
| # | |
| # For a mounted volume, use | |
| # | |
| # sudo blkid /dev/nvme1n1 | |
| # | |
| # After setting `azul_gitlab_data_volume_id` to the obtained volume UUID, terminate the instance and (re)deploy the `.gitlab` component. |
| [gitlab.tf.json.template.py]. The volume is mounted at `/mnt/gitlab`. Also, the | ||
| volume is mounted by its filesystem UUID rather than by device path, ensuring | ||
| reliable mounting across different instance types, as device names can vary. See | ||
| [gitlab.tf.json.template.py] for details on obtaining the UUID. The |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linked issues: #4890
Checklist
Author
developissues/<GitHub handle of author>/<issue#>-<slug>1 when the issue title describes a problem, the corresponding PR
title is
Fix:followed by the issue titleAuthor (partiality)
ptag to titles of partial commitspartialor completely resolves all linked issuespartiallabelAuthor (reindex)
rtag to commit title or the changes introduced by this PR will not require reindexing of any deploymentreindex:devor the changes introduced by it will not require reindexing ofdevreindex:anvildevor the changes introduced by it will not require reindexing ofanvildevreindex:anvilprodor the changes introduced by it will not require reindexing ofanvilprodreindex:prodor the changes introduced by it will not require reindexing ofprodreindex:partialand its description documents the specific reindexing procedure fordev,anvildev,anvilprodandprodor requires a full reindex or carries none of the labelsreindex:dev,reindex:anvildev,reindex:anvilprodandreindex:prodAuthor (API changes)
APIor this PR does not modify a REST APIa(A) tag to commit title for backwards (in)compatible changes or this PR does not modify a REST APIapp.pyor this PR does not modify a REST APIAuthor (upgrading deployments)
make docker_images.jsonand committed the resulting changes or this PR does not modifyazul_docker_images, or any other variables referenced in the definition of that variableutag to commit title or this PR does not require upgrading deploymentsupgradeor does not require upgrading deploymentsdeploy:sharedor does not modifydocker_images.json, and does not require deploying thesharedcomponent for any other reasondeploy:gitlabor does not require deploying thegitlabcomponentdeploy:runneror does not require deploying therunnerimageAuthor (hotfixes)
Ftag to main commit title or this PR does not include permanent fix for a temporary hotfixanvilprodandprod) have temporary hotfixes for any of the issues linked to this PRAuthor (before every review)
develop, squashed fixups from prior reviewsmake requirements_updateor this PR does not modifyDockerfile,environment,requirements*.txt,common.mk,Makefileorenvironment.bootRtag to commit title or this PR does not modifyrequirements*.txtreqsor does not modifyrequirements*.txtmake integration_testpasses in personal deployment or this PR does not modify functionality that could affect the IT outcomePeer reviewer (after approval)
Note that after requesting changes, the PR must be assigned to only the author.
System administrator (after approval)
demoorno demono demono sandboxN reviewslabel is accurateOperator
reindex:…labels andrcommit title tagno demodevelopOperator (deploy
.sharedand.gitlabcomponents)_select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unusedor this PR is not labeleddeploy:shared_select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab applyor this PR is not labeleddeploy:gitlab_select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unusedor this PR is not labeleddeploy:shared_select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab applyor this PR is not labeleddeploy:gitlabdeploy:gitlabdeploy:gitlabSystem administrator (post-deploy of
.gitlabcomponent)dev.gitlabare complete or this PR is not labeleddeploy:gitlabanvildev.gitlabare complete or this PR is not labeleddeploy:gitlabOperator (deploy runner image)
_select dev.gitlab && make -C terraform/gitlab/runneror this PR is not labeleddeploy:runner_select anvildev.gitlab && make -C terraform/gitlab/runneror this PR is not labeleddeploy:runnerOperator (sandbox build)
sandboxlabel or PR is labeledno sandboxdevor PR is labeledno sandboxanvildevor PR is labeledno sandboxsandboxdeployment or PR is labeledno sandboxanvilboxdeployment or PR is labeledno sandboxsandboxdeployment or PR is labeledno sandboxanvilboxdeployment or PR is labeledno sandboxsandboxor this PR does not remove catalogs or otherwise causes unreferenced indices insandboxanvilboxor this PR does not remove catalogs or otherwise causes unreferenced indices inanvilboxsandboxor this PR is not labeledreindex:devanvilboxor this PR is not labeledreindex:anvildevsandboxor this PR is not labeledreindex:devanvilboxor this PR is not labeledreindex:anvildevOperator (merge the branch)
pif the PR is also labeledpartialOperator (main build)
devanvildevdevdevanvildevanvildev_select dev.shared && make -C terraform/shared applyor this PR is not labeleddeploy:shared_select anvildev.shared && make -C terraform/shared applyor this PR is not labeleddeploy:shareddevanvildevOperator (reindex)
devor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR is neither labeledreindex:partialnorreindex:devanvildevor this PR is neither labeledreindex:partialnorreindex:anvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdevor this PR does not require reindexingdevdevor this PR does not require reindexingdevdeploy_browserjob in the GitLab pipeline for this PR indevor this PR does not require reindexingdevanvildevor this PR does not require reindexinganvildevdeploy_browserjob in the GitLab pipeline for this PR inanvildevor this PR does not require reindexinganvildevOperator (mirroring)
devor this PR does not require mirroringdevanvildevor this PR does not require mirroringanvildevdevor this PR does not require mirroringdevanvildevor this PR does not require mirroringanvildevdevor this PR does not require mirroringdevanvildevor this PR does not require mirroringanvildevOperator
deploy:shared,deploy:gitlab,deploy:runner,API,reindex:partial,reindex:anvilprodandreindex:prodlabels to the next promotion PRs or this PR carries none of these labelsdeploy:shared,deploy:gitlab,deploy:runner,API,reindex:partial,reindex:anvilprodandreindex:prodlabels, from the description of this PR to that of the next promotion PRs or this PR carries none of these labelsShorthand for review comments
Lline is too longWline wrapping is wrongQbad quotesFother formatting problem