Improve resilience of posthog integration

[rhysyngsun]

Description/Context

Posthog had an outage which caused calls to their API to timeout.

We worked around this by setting IN_TEST_SUITE=True and FEATURES_DEFAULT=True because we were fortunate enough to have all features enabled. It took some time to evaluate if this was the correct way to address the issue because IN_TEST_SUITE sounds like it could have other undesirable effects.

Plan/Design

• Rename IN_TEST_SUITE to POSTHOG_ENABLED
• Investigate options to tune the timeout/retry settings in posthog
  • The default appears to be 3 retries w/ 15 second timeouts, pushing us well over the 30 second Heroku request timeout.
  • The client itself appears to have settings for these: https://github.com/PostHog/posthog-python/blob/master/posthog/client.py#L32
  • These aren't all exposed through module-level properties, but it looks like we can set default_client
  • See also how the library itself creates a default default_client: https://github.com/PostHog/posthog-python/blob/master/posthog/__init__.py#L441-L455
  • Handle errors from the call to Posthog
• Document procedures for Posthog outages

[pdpinch]

How similar is this to the work @jkachel is doing on mitodl/mit-learn#693

[collinpreston]

@rhysyngsun what does "pushing us well over the 30 second Heroku request timeout." mean?

[rhysyngsun]

Heroku has a 30 second request timeout: https://devcenter.heroku.com/articles/request-timeout

[collinpreston]

This should not be automatically closed. This can be closed once the changes added in mitodl/ol-django#152 are integrated with MITx Online.