Placeholder issue to track this process. Current idea: add a prefers_async annotation to data blocks that need to be processed "offline". Develop an API spec for how a block should respond as its being processed, and let the UI poll and reload this. Then use APScheduler as our initial backend for offline tasks, following its inclusion in #1371.
Future work: use more robust/decentralised backend like RMQ/Arq + celery, but for now APScheduler will give us 90% of the benefit with limited extra work.