[DataStore] Implement mutations in batches or upload data in the background

[kubatatami]

Is your feature request related to a problem? Please describe.

Users open our app once or twice a day. On app startup we fetch data from physical device and save it in DataStore. For each day we may have ~2000 rows to save. We observe that DataStore send mutation queries for each row one by one and only when app is the foreground. Each mutation takes 300-800ms so to send all data to backend we have to keep app open for almost 20 minutes while our user have the app open for a few minutes only. In result we don't have a recent data on the backend and not synced data grows on mobile device.

Describe the solution you'd like

If Amplify SDK send mutation queries in batches (let's say 50 rows per request) then uploading ~2000 rows will be significantly faster and it will solve our issue.

Describe alternatives you've considered

Alternatively, if Amplify SDK let us to upload data when app is in the background it will also solve our issue. Our app keeps BLE connection with external device so it's active in the background for many minutes per day - enough to send all ~2000 rows.

Is the feature request related to any of the existing Amplify categories?

DataStore

Additional context

No response

[github-actions]

This has been identified as a feature request. If this feature is important to you, we strongly encourage you to give a 👍 reaction on the request. This helps us prioritize new features most important to you. Thank you!

[lawmicha]

Thank you for raising the feature request. Our team will prioritize it and will post an update on the issue once we have more information.

[madej10]

Hi @lawmicha any update on this? we're seeing #3437 a lot too so we would be very interested in this feature

[fzy-github]

@lawmicha I am experimenting with BackgroundTasks on iOS in order to trigger DataStore syncing to cloud and have a question.

I am trying a naive approach that will "simply" call DataStore.stop() followed by DataStore.start() methods and will observe outboxStatus event until it reaches empty: true state.

So far I have observed that I am able to call start/stop methods but I am unable to receive outbox events. Is there any code in DataStore which might prevent it running when app is in background state?

[lawmicha]

Hi @fzy-spyro, there isn't anything in DataStore that I'm aware of that depends on the background/foreground state of the app. I wonder what the start sequence logs look like when you call start() from the background task, perhaps it's not emitting the event because it didn't sync the mutation? Do you see any other events emitted? the events are sent through the Hub category in Amplify.

[lawmicha]

@madej10 / @kubatatami DataStore currently does not have any special implementation that checks the app states to see whether it's in foreground or background, I think the path forward in this feature request are one or a combination of:

1. DataStore to own the implementations of the background tasks that will execute in the background. This may or may not be opt-in, such that the developers may have to enable background mode of some sort for DataStore to utilize this behavior.

2. DataStore to provide APIs which are to be called during background task execution, the developer writes the background tasks and as part of the code, call into something like DataStore.performBackgroundSync() , where the API is a slim down version of the DataStore sync sequence (probably wouldn't have to estasblish the subscriptions for example).

A note about batching, DataStore calls AppSync APIs create/update/delete, so batching may be difficult because the AppSync API to perform batch mutations simply doesn't exist. If AppSync does provide a batch API, we may be able to reconcile the entire pages pretty quickly, as we do with the full/delta sync response which is also a page of model data.