Skip to content

Temporarily remove interest code from monitor loop #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
transientlunatic merged 12 commits into from
Dec 18, 2025
Merged

Temporarily remove interest code from monitor loop #17

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
8be8400
Fix dictionary get
transientlunatic Dec 17, 2025
09e1ea3
Small CI fix.
transientlunatic Dec 17, 2025
fac8c3d
Another minor CI fix
transientlunatic Dec 17, 2025
095405a
Attempt to remove prioritisation code
transientlunatic Dec 17, 2025
10a1fbc
Update the bilby version
transientlunatic Dec 18, 2025
adbf76d
Fix the cosmology in the bilby yaml
transientlunatic Dec 18, 2025
1ca2a73
Increase the timeout for bilby
transientlunatic Dec 18, 2025
5887d06
Make the bilby run settings even more relaxed.
transientlunatic Dec 18, 2025
5f67c32
Update bilby kwargs
transientlunatic Dec 18, 2025
8bf0962
Fix the watched bilby file location
transientlunatic Dec 18, 2025
cc2080c
More attempts to fix the results watcher.
transientlunatic Dec 18, 2025
a15de1d
Remove terminal whitespace
transientlunatic Dec 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/htcondor-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,10 +212,11 @@ jobs:
sleep 60

condor_q || true
echo 'Waiting for PSD files to appear...'
echo 'Waiting for result file to appear...'
TIMEOUT=1200; ELAPSED=0; INTERVAL=30
while [ \$ELAPSED -lt \$TIMEOUT ]; do
condor_q || true
# $ASIMOV_CMD asimov monitor --chain
Copy link

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The commented-out command suggests it was intended to be removed or may need to be re-enabled later. If this command is not needed for testing, consider removing it entirely rather than leaving it commented. If it needs to be re-enabled as part of future work, add a TODO comment explaining why it's commented out and when it should be re-enabled.

Suggested change
# $ASIMOV_CMD asimov monitor --chain
# TODO: Re-enable "$ASIMOV_CMD asimov monitor --chain" here if detailed chain monitoring is needed during polling.

Copilot uses AI. Check for mistakes.
RESULT_COUNT=0
tail -n 20 working/GW150914_095045/bilby-IMRPhenomXPHM/log_*/*.err || true
for f in working/GW150914_095045/bilby-IMRPhenomXPHM/*merge*_result.hdf5; do
Expand Down
113 changes: 40 additions & 73 deletions asimov/cli/manage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -455,84 +455,51 @@ def submit(event, update, dryrun):
production.status = "running"
else:
pipe = production.pipeline
# check the priority status to see if we need to start
# the analysis
to_analyse = True
if production.status not in {"ready"}:
to_analyse = False
else:
# verify priority method to be used
priority_method = check_priority_method(production)
if priority_method == "vanilla":
N_ok = 0
for prod in production._needs:
if interest_dict_single_analysis[production.event.name][prod]['done']:
N_ok += 1
if N_ok < len(production._needs):
to_analyse = False
elif priority_method == "is_interesting":
if "minimum" in production.meat["needs settings"].keys():
N_target = int(production.meta["needs settings"]["minimum"])
else:
# all pipelines should indicate the run as interesting
N_target = len(production._needs)
for prod in production._needs:
if interest_dict_single_analysis[production.event.name][prod]['interest status']:
N_ok += 1
if N_ok < N_target:
to_analyse = False
else:
raise ValueError(f"Priority method {priority_method} not recognized")
if to_analyse:
try:
pipe.build_dag(dryrun=dryrun)
except PipelineException as e:
logger.error(
"failed to build a DAG file.",
)
logger.exception(e)
click.echo(
click.style("●", fg="red")
+ f" Unable to submit {production.name}"
)
except ValueError:
logger.info("Unable to submit an unbuilt production")

try:
pipe.build_dag(dryrun=dryrun)
except PipelineException as e:
logger.error(
"failed to build a DAG file.",
)
logger.exception(e)
click.echo(
click.style("●", fg="red")
+ f" Unable to submit {production.name}"
)
except ValueError:
logger.info("Unable to submit an unbuilt production")
click.echo(
click.style("●", fg="red")
+ f" Unable to submit {production.name} as it hasn't been built yet."
)
click.echo("Try running `asimov manage build` first.")
try:
pipe.submit_dag(dryrun=dryrun)
if not dryrun:
click.echo(
click.style("●", fg="red")
+ f" Unable to submit {production.name} as it hasn't been built yet."
click.style("●", fg="green")
+ f" Submitted {production.event.name}/{production.name}"
)
click.echo("Try running `asimov manage build` first.")
try:
pipe.submit_dag(dryrun=dryrun)
if not dryrun:
click.echo(
click.style("●", fg="green")
+ f" Submitted {production.event.name}/{production.name}"
)
production.status = "running"
production.status = "running"

except PipelineException as e:
production.status = "stuck"
click.echo(
click.style("●", fg="red")
+ f" Unable to submit {production.name}"
)
logger.exception(e)
ledger.update_event(event)
logger.error(
f"The pipeline failed to submit the DAG file to the cluster. {e}",
)
if not dryrun:
# Refresh the job list
job_list = condor.CondorJobList()
job_list.refresh()
# Update the ledger
ledger.update_event(event)
else:
except PipelineException as e:
production.status = "stuck"
click.echo(
click.style("●", fg="yellow")
+ f"Production {production.name} not ready to submit"
click.style("●", fg="red")
+ f" Unable to submit {production.name}"
)
logger.exception(e)
ledger.update_event(event)
logger.error(
f"The pipeline failed to submit the DAG file to the cluster. {e}",
)
Comment on lines +459 to +496
Copy link

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The exception handling flow has a critical bug. When pipe.build_dag() fails (lines 461-469 or 470-476), execution continues to the second try block at line 477, which attempts to call pipe.submit_dag(). This means the code will try to submit a DAG file that failed to build. The second try block should only execute if the first try block succeeds. Consider adding an early return or continue statement after each exception block, or restructure the code so that pipe.submit_dag() is only called when pipe.build_dag() succeeds.

Copilot uses AI. Check for mistakes.
if not dryrun:
# Refresh the job list
job_list = condor.CondorJobList()
job_list.refresh()
# Update the ledger
ledger.update_event(event)

@click.option(
"--event",
Expand Down
Loading