Request discussion of proposed UX changes

[TopDevPros]

A good a/b test starts by setting the goals and criteria so you know when you've got an actionable result. Currently wagtail-experiments does not provide a way for the user to set params that are important for reliability.

The following changes should significantly enhance the value of wagtail-experiments to both high and low traffic sites.

Possible new settings in UX:

• The single biggest error in a/b testing appears to be deciding based on too small a sample. There's a lot of controversy over how small is too small. The consensus among pollsters is that 1000 responses is enough to represent 300+ million people, so at least we have an upper limit. It's also a good idea to limit the time frame to reduce influence by changing conditions.

  Minimum sample to recommend action: [ ]
  Maximum time frame: [ ]

• Goals reached after many intermediate pages usually aren't very relevant to the experiment.

  [x] Goal must be reached directly from an experiment page
  [ ] Intermediate pages before goal are accepted (not recommended, but may be current behavior)

• Sometimes you're testing different titles. Sometimes you're testing body changes with the same title. Both are important. If there are many alternative pages, we can make it easy to use the control page title.

  [x] Use control page title

Possible ways to reduce cluttering the UX:

• Hard code good defaults. But if users can't change them it will invite controversy about our defaults.

• Allow users to add preferences via settings.py. Marketers are the primary target market for wagtail-experiments, so this isn't the best option.

• Only show detailed settings if requested. The request could be on a Settings page.

• Put detailed settings on a Settings page. All experiments would share the same settings.