Skip to content

Fix race condition - envsubst $file | tee $file#558

Merged
openshift-merge-bot[bot] merged 1 commit intoopenstack-k8s-operators:mainfrom
hjensas:OSPRH-17012
Jun 2, 2025
Merged

Fix race condition - envsubst $file | tee $file#558
openshift-merge-bot[bot] merged 1 commit intoopenstack-k8s-operators:mainfrom
hjensas:OSPRH-17012

Conversation

@hjensas
Copy link
Copy Markdown
Contributor

@hjensas hjensas commented May 30, 2025

When running envsubst $file and pipe to tee $file there is a race causing the target file to be wiped.

The following script can be used to reproduce the race condition:

  #!/bin/sh
  echo -e "foo\nbar\nbaz" > /tmp/test_file
  count=0
  while [ -s /tmp/test_file ]; do
    envsubst < /tmp/test_file | tee /tmp/test_file > /dev/null
    ((count+=1))
  done

  if [ ! -s /tmp/test_file ]; then
    echo "File was wiped after $count run's of:  envsubst < \$file | tee \$file"
  fi
$ sh test.sh
File was wiped after 16 run's of:  envsubst < $file | tee $file
$ sh test.sh
File was wiped after 42 run's of:  envsubst < $file | tee $file
$ sh test.sh
File was wiped after 37 run's of:  envsubst < $file | tee $file
$ sh test.sh
File was wiped after 73 run's of:  envsubst < $file | tee $file
$ sh test.sh
File was wiped after 4 run's of:  envsubst < $file | tee $file

Jira: OSPRH-17012
Jira: OSPCIX-870

Copy link
Copy Markdown
Collaborator

@steveb steveb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oosh

/lgtm

@softwarefactory-project-zuul
Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/a511ac08567347068f8fb1ff66e7b175

✔️ noop SUCCESS in 0s
✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 19m 49s
podified-multinode-ironic-deployment FAILURE in 59m 36s

When running `envsubst $file` and pipe to `tee $file` there is a race
causing the targe file to be wiped.

The following script can be used to reproduce the race condition:
```
  #!/bin/sh
  echo -e "foo\nbar\nbaz" > /tmp/test_file
  count=0
  while [ -s /tmp/test_file ]; do
    envsubst < /tmp/test_file | tee /tmp/test_file > /dev/null
    ((count+=1))
  done

  if [ ! -s /tmp/test_file ]; then
    echo "File was wiped after $count run's of:  envsubst < \$file | tee \$file"
  fi
```

```
$ sh test.sh
File was wiped after 16 run's of:  envsubst < $file | tee $file
$ sh test.sh
File was wiped after 42 run's of:  envsubst < $file | tee $file
$ sh test.sh
File was wiped after 37 run's of:  envsubst < $file | tee $file
$ sh test.sh
File was wiped after 73 run's of:  envsubst < $file | tee $file
$ sh test.sh
File was wiped after 4 run's of:  envsubst < $file | tee $file
```

Jira: OSPRH-17012
Jira: OSPCIX-870
@hjensas
Copy link
Copy Markdown
Contributor Author

hjensas commented May 31, 2025

Looks ok in CI now: https://logserver.rdoproject.org/be0/rdoproject.org/be01bb61bcf94fa79f026408fc09c6dc/controller/ci-framework-data/logs/openstack-k8s-operators-openstack-must-gather/namespaces/openstack/pods/ironic-inspector-0/logs/inspector-pxe-init.log

+ export TMP_DNSMASQ_CFG=/var/tmp/dnsmasq.conf
+ TMP_DNSMASQ_CFG=/var/tmp/dnsmasq.conf
+ cp /var/lib/ironic/dnsmasq.conf /var/tmp/dnsmasq.conf
+ envsubst
+ tee /var/lib/ironic/dnsmasq.conf
# Disable listening for DNS
port=0
enable-tftp
tftp-root=/var/lib/ironic/tftpboot
...
+ export TMP_INSPECTOR_IPXE=/var/tmp/inspector.ipxe
+ TMP_INSPECTOR_IPXE=/var/tmp/inspector.ipxe
+ cp /var/lib/ironic/inspector.ipxe /var/tmp/inspector.ipxe
+ envsubst
+ tee /var/lib/ironic/inspector.ipxe
#!ipxe

:retry_boot
...

Copy link
Copy Markdown
Contributor

@abays abays left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci openshift-ci bot added the lgtm label Jun 2, 2025
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Jun 2, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abays, hjensas

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot openshift-merge-bot bot merged commit aed5d99 into openstack-k8s-operators:main Jun 2, 2025
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants