Skip to content

Conversation

@cscheid
Copy link
Collaborator

@cscheid cscheid commented Jan 21, 2025

With some A/B testing, we find evidence that the code in the PR is faster than that of main (I can share the script later)

$ N_TRIALS=3000 QUARTO_NO_TYPECHECK=1 ./run_experiment_2.py coalesce-raw.csv main tmp-2025-01-21-coalesce-raw quarto render --quiet docs/references/formats/presentations/revealjs.qmd
┏━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┓
┃ Variant ┃ Count ┃ Mean    ┃ Std Dev   ┃
┡━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━┩
│ A       │ 10    │ 2.51012 │ 0.0106346 │
│ B       │ 10    │ 2.49222 │ 0.0101653 │
└─────────┴───────┴─────────┴───────────┘
{'outcome': 'significant', 'winner': 'B', 'p_value': np.float64(0.0018407964064409569), 'total_samples': 20, 'alpha_spent': 0.05, 'statistics': {'counts': {'A': 10, 'B': 10}, 'means': {'A': 2.510115218162537, 'B': 2.4922211170196533}, 'variances': {'A': 0.00011309452203267512, 'B': 0.00010333396089645674}}}
Statistical Summary:
===================
Variant     Count         Mean      Std Dev
-------- -------- ------------ ------------
A              10     2.510115     0.010635
B              10     2.492221     0.010165

I used docs/references/formats/presentations/revealjs.qmd from quarto-web because I measured that file to be the one to take the longest time in a site render.

@cscheid
Copy link
Collaborator Author

cscheid commented Jan 21, 2025

GitHub is getting really confused about some commits. I'll close and try again from a separate branch.

@cscheid cscheid closed this Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants