-
-
Notifications
You must be signed in to change notification settings - Fork 236
ENH: Adaptive Monte Carlo via Convergence Criteria #922
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Changes from all commits
3f483dc
6c79f80
424aeab
f6fbd1d
f6166b8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -800,6 +800,28 @@ | |
| ")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Alternatively, we can target an attribute using the method `MonteCarlo.simulate_convergence()` such that when the tolerance is met, the flight simulations would terminate early." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "test_dispersion.simulate_convergence(\n", | ||
| " target_attribute=\"apogee_time\",\n", | ||
| " target_confidence=0.95,\n", | ||
| " tolerance=0.5, # in seconds\n", | ||
| " max_simulations=1000,\n", | ||
| " batch_size=50\n", | ||
| " )" | ||
|
Comment on lines
+816
to
+822
|
||
| ] | ||
| }, | ||
| { | ||
| "attachments": {}, | ||
| "cell_type": "markdown", | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -525,6 +525,69 @@ def estimate_confidence_interval( | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| return res.confidence_interval | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| def simulate_convergence( | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| self, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| target_attribute="apogee_time", | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| target_confidence=0.95, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| tolerance=0.5, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| max_simulations=1000, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| batch_size=50, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| parallel=True, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| n_workers=4, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ): | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Comment on lines
+528
to
+537
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| """Run simulations cumulatively in batches until the confidence interval meets tolerance. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Parameters | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ---------- | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| target_attribute : str | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| The target attribute to track its convergence (e.g., "apogee", "apogee_time", etc.). | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| target_confidence : float, optional | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| The confidence level for the interval (between 0 and 1). Default is 0.95. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| tolerance : float, optional | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| The desired width of the confidence interval in seconds, meters, or other units. Default is 0.5. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| max_simulations : int, optional | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| The maximum number of simulations to run to avoid infinite loops. Default is 1000. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| batch_size : int, optional | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| The number of simulations to run in each batch. Default is 50. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| parallel : bool, optional | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Whether to run simulations in parallel. Default is True. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| n_workers : int, optional | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| The number of worker processes to use if running in parallel. Default is 8. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| The number of worker processes to use if running in parallel. Default is 8. | |
| The number of worker processes to use if running in parallel. Default is 4. |
Copilot
AI
Jan 6, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Returns section of the docstring is incorrect. It states that the method returns a single float value for the confidence interval width, but the actual implementation returns a list of confidence interval widths (one for each batch). The docstring should be updated to reflect that it returns a list of floats representing the confidence interval width history.
| confidence_interval_width : float | |
| The confidence interval width when the simulation stopped for either meeting the tolerance or maximum number of simulations. | |
| confidence_interval_width_history : list of float | |
| History of confidence interval widths, one value per batch of simulations. | |
| The last element corresponds to the width when the simulation stopped for | |
| either meeting the tolerance or reaching the maximum number of simulations. |
Copilot
AI
Jan 6, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The docstring describes the method purpose but could be more precise. The phrase "Run simulations cumulatively in batches" could be clarified to specify that it runs simulations until convergence is achieved or max_simulations is reached. Consider rephrasing to: "Run Monte Carlo simulations in batches until the confidence interval width converges within the specified tolerance or the maximum number of simulations is reached."
| """Run simulations cumulatively in batches until the confidence interval meets tolerance. | |
| Parameters | |
| ---------- | |
| target_attribute : str | |
| The target attribute to track its convergence (e.g., "apogee", "apogee_time", etc.). | |
| target_confidence : float, optional | |
| The confidence level for the interval (between 0 and 1). Default is 0.95. | |
| tolerance : float, optional | |
| The desired width of the confidence interval in seconds, meters, or other units. Default is 0.5. | |
| max_simulations : int, optional | |
| The maximum number of simulations to run to avoid infinite loops. Default is 1000. | |
| batch_size : int, optional | |
| The number of simulations to run in each batch. Default is 50. | |
| parallel : bool, optional | |
| Whether to run simulations in parallel. Default is True. | |
| n_workers : int, optional | |
| The number of worker processes to use if running in parallel. Default is 8. | |
| Returns | |
| ------- | |
| confidence_interval_width : float | |
| The confidence interval width when the simulation stopped for either meeting the tolerance or maximum number of simulations. | |
| """Run Monte Carlo simulations in batches until the confidence interval | |
| width converges within the specified tolerance or the maximum number of | |
| simulations is reached. | |
| Parameters | |
| ---------- | |
| target_attribute : str | |
| The target attribute to track its convergence (e.g., "apogee", | |
| "apogee_time", etc.). | |
| target_confidence : float, optional | |
| The confidence level for the interval (between 0 and 1). Default is | |
| 0.95. | |
| tolerance : float, optional | |
| The desired width of the confidence interval in seconds, meters, or | |
| other units. Default is 0.5. | |
| max_simulations : int, optional | |
| The maximum number of simulations to run to avoid infinite loops. | |
| Default is 1000. | |
| batch_size : int, optional | |
| The number of simulations to run in each batch. Default is 50. | |
| parallel : bool, optional | |
| Whether to run simulations in parallel. Default is True. | |
| n_workers : int, optional | |
| The number of worker processes to use if running in parallel. | |
| Default is 8. | |
| Returns | |
| ------- | |
| confidence_interval_widths : list of float | |
| Sequence of confidence interval widths computed after each batch of | |
| simulations, up to the point where the tolerance criterion is met | |
| or the maximum number of simulations is reached. |
Copilot
AI
Jan 6, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Calling import_outputs at the start of the method may not be necessary and could cause confusion. This loads outputs from an existing file, which may not align with the intent of a fresh convergence simulation. Consider removing this line or documenting why it's necessary to load previous outputs before starting the convergence simulation.
Copilot
AI
Jan 6, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The while loop condition could potentially result in running more simulations than max_simulations. If num_of_loaded_sims is less than max_simulations by less than batch_size, the next batch will run total_sims which equals max_simulations (due to the min() operation), but this could mean more work than expected if the confidence interval hasn't converged. Consider adding validation to ensure batch_size doesn't exceed max_simulations, and clarify the behavior when num_of_loaded_sims is not zero at the start.
| while (self.num_of_loaded_sims < max_simulations): | |
| total_sims = min(self.num_of_loaded_sims + batch_size, max_simulations) | |
| self.simulate( | |
| number_of_simulations=total_sims, | |
| if max_simulations <= 0: | |
| raise ValueError("max_simulations must be a positive integer.") | |
| if batch_size <= 0: | |
| raise ValueError("batch_size must be a positive integer.") | |
| if batch_size > max_simulations: | |
| warnings.warn( | |
| "batch_size is greater than max_simulations; " | |
| "capping batch_size to max_simulations.", | |
| UserWarning, | |
| ) | |
| batch_size = max_simulations | |
| while self.num_of_loaded_sims < max_simulations: | |
| remaining_sims = max_simulations - self.num_of_loaded_sims | |
| sims_to_run = min(batch_size, remaining_sims) | |
| if sims_to_run <= 0: | |
| break | |
| self.simulate( | |
| number_of_simulations=sims_to_run, |
Copilot
AI
Jan 6, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The import_outputs call within the loop is inefficient because it reloads and reprocesses the entire output file after each batch. Since simulate() with append=True already writes to the file and the output_file setter triggers set_outputs_log(), set_num_of_loaded_sims(), set_results(), and set_processed_results(), this additional import is redundant. Consider removing this line to improve performance.
| self.import_outputs(self.filename.with_suffix(".outputs.txt")) |
Copilot
AI
Jan 6, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The method does not validate that there are enough simulations to compute a meaningful confidence interval. The bootstrap method in estimate_confidence_interval requires a minimum number of samples to produce reliable results. Consider adding a check to ensure that at least a minimum number of simulations (e.g., batch_size or some reasonable threshold) have been completed before calculating the confidence interval, or document this requirement.
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -236,3 +236,31 @@ def invalid_data_collector(flight): | |||||
| monte_carlo_calisto.simulate(number_of_simulations=10, append=False) | ||||||
| finally: | ||||||
| _post_test_file_cleanup() | ||||||
|
|
||||||
|
|
||||||
| @pytest.mark.slow | ||||||
| def test_monte_carlo_simulate_convergence(monte_carlo_calisto): | ||||||
| """Tests the simulate_convergence method of the MonteCarlo class. | ||||||
|
|
||||||
| Parameters | ||||||
| ---------- | ||||||
| monte_carlo_calisto_pre_loaded : MonteCarlo | ||||||
|
||||||
| monte_carlo_calisto_pre_loaded : MonteCarlo | |
| monte_carlo_calisto : MonteCarlo |
Copilot
AI
Jan 6, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The tolerance value used in the test appears quite large (5.0 meters for apogee). For a test that verifies convergence behavior, consider using a more realistic tolerance value that would actually test the convergence logic, or add a comment explaining why this large tolerance is appropriate for this test case.
Copilot
AI
Jan 6, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The print statement should be removed from the test. Print statements in tests are generally not recommended as they clutter test output and don't provide value in automated testing. If debugging information is needed, use proper logging or remove the statement after debugging is complete.
| print(ci_history) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Inconsistent markdown link formatting. Lines 35 and 36 use parentheses for the URL part instead of square brackets, which is incorrect markdown syntax. The format should be #922 to match the pattern used in line 37 and other entries.