Database Sync

[Paul-kp]

I see sync was added as of V3.2.0.
Can you provide details about its operation so I can update the KeePass list of apps that sync?
https://sourceforge.net/p/keepass/discussion/329220/thread/9969adb36a/

cheers, Paul

[J-Jamet]

https://github.com/Kunzisoft/KeePassDX/wiki/File-Manager-and-Sync

[Paul-kp]

I read that but it's not obvious how it works in practice. I don't have a spare Android device to test it on.

cheers, Paul

[J-Jamet]

In practice, it is the user who chooses the separate application that will be used to synchronize his files (any type of files), for example, a Cloud application.

KeePassDX completely delegates the retrieval of file data, so once the file stream is retrieved by the URI, KeePassDX only handles the merge of new deciphered data and the save.

As synchronization is the combination of

• file recovery (by any protocol) and conflict binary file management (by the thirst party file manager app)
• and the merging of encrypted data (by KeePassDX)

synchronization is managed in 2 different applications, KeePassDX and another one chosen by the user.

[nomis]

If KeePassDX opens the file from the SAF and I then edit the same file on another device, how does KeePassDX know to synchronise the changes from the two versions instead of overwriting them?

When I use RCX to access a WebDAV folder, KeePassDX is just overwriting the changes made elsewhere while the file was open. The other application is also using the same WebDAV service and writing to the file in place so KeePassDX should be aware that the file has been changed.

What's at fault here and is this supported by the Android SAF API or not? For this to work correctly KeePassDX would need to be informed that there's a newer version of the file so that it can try to merge it.

This is especially important when using caching (not used during the above test) because a temporary network failure could mean that an out of date version of the file is loaded (with no warning for the user) and then passwords will be deleted when it's saved. Without caching this application is unusable over slower or higher latency connections (e.g. mobile data from another country).

[nomis]

If KeePassDX opens the file from the SAF and I then edit the same file on another device, how does KeePassDX know to synchronise the changes from the two versions instead of overwriting them?

It is not KeePassDX that has this logic. KeePassDX does not know how the file is synchronized, it only knows when it is synchronized. It is the role of the file manager to properly perform this task. https://github.com/Kunzisoft/KeePassDX/wiki/File-Manager-and-Sync

The file manager can't do this. Only KeePassDX can decrypt the contents of both versions of the file and merge changes.

When I use RCX to access a WebDAV folder, KeePassDX is just overwriting the changes made elsewhere while the file was open. The other application is also using the same WebDAV service and writing to the file in place so KeePassDX should be aware that the file has been changed.

If RCX tells KeePassDX that the files it manages have modified metadata, so the modification date has been changed, then KeePassDX detects a change and a possible merge. If the metadata of the URI is not modified then KeePassDX cannot know it.

What's at fault here and is this supported by the Android SAF API or not? For this to work correctly KeePassDX would need to be informed that there's a newer version of the file so that it can try to merge it.

Yes it is supported by Android SAF since it has been tested with KeePassDX.

Is there logging available in KeePassDX that would prove that it was provided with out of date metadata when it went to save the file?

Without caching this application is unusable over slower or higher latency connections (e.g. mobile data from another country).

That's why a dedicated sync application is being designed as such.

I assume FileSync won't be able to decrypt the file so it has no way to perform synchronisation when the file has been modified elsewhere.

I don't understand how you will achieve this without either caching the file within KeePassDX or using two SAF URIs for the same database ("always available local storage" and "remote storage").

If KeePassDX supported two SAF URIs for the same database then a specific dedicated sync application would be unnecessary because KeePassDX could synchronise the two versions using any SAF provider (as long as the correct metadata is returned). Only KeePassDX is able to decrypt the file in order to synchronise between two versions.

How would the following scenarios be handled? Assume that when "the network is unavailable" that the remote storage may also have been down for an extended period of time such that a SAF provider that normally pushes all changes immediately to local storage on the Android device also has an out of date version of the database.

1. Immediately open the database even if the network is unavailable, high latency or low bandwidth

There's no way for any application to know what the network conditions are in advance so it must always be loaded first from local storage (because this is always fast). The latest version would be loaded from remote storage (in the background) and merged. If there's a conflict then a notification would ask the user how to resolve it.

Unless the SAF has a way for the application to request either a cached or up to date version from the same URI this is going to require two URIs or in-app storage.

Note: it is not acceptable to decide "the network is slow, therefore the user will have to wait longer and may have to retry". It's bad enough trying to use services over a slow connection; that doesn't need to be made worse by also having to re-download the entire password database before being able to login.

If the background load from remote storage failed then it won't be possible to save changes remotely because the remote file may be out of date.

Unless the SAF has a way for the application to indicate that it is saving changes based on an out of date version that will later be merged by the application (because only KeePassDX can decrypt the file to do that) this is going to require two URIs or in-app storage.

2. Save the database when the network is unavailable (or so unreliable that the file can't be remotely read or written successfully)

If the SAF stores the file locally when the network is down then the remote file may also have been changed before the network is back up.

Unless the SAF is able to request that the application do a merge of two file versions (because only KeePassDX can decrypt the file to do that) this is going to require two URIs or in-app storage (and explicit "do not store the file locally" behaviour when writing to the remote SAF URI).

[J-Jamet]

Thank you @nomis for you feedback. It's interesting to see your point of view, I will try to answer as best I can.

The file manager can't do this. Only KeePassDX can decrypt the contents of both versions of the file and merge changes.

When we talk about synchronization, we are talking about :
1/ correctly retrieve data and manage conflicts from two different sources at the file metadata level
2/ managing the merge of data from two different database sources, and therefore of file content

Any cloud file manager should be able to handle synchronization conflicts regardless of the file type to correctly do the 1/. Here .kdbx files are considered as binary, there can be several strategies to manage conflicts at this level depending of file manager.
For 2/, The way SAF works is simple, there is a URI provided to KeePassDX and if the data of this URI is updated then KeePassDX can merge it with the data it previously received from the same URI.

So KeePassDX doesn't know how the file is synchronized and what transfer protocol is used, and it doesn't care, it just wants the URI data.

Is there logging available in KeePassDX that would prove that it was provided with out of date metadata when it went to save the file?

Yes, this is how the merge works in KeePassDX. If the metadata of the URI is updated, a message appears indicating that the data has changed, the user has the choice to merge the data, ignore it and keep the old data, or overwrite the old data. The old data is the data in RAM that was loaded the first time it was opened. There is no cache and no data retention in KeePassDX when a database is closed.

I assume FileSync won't be able to decrypt the file so it has no way to perform synchronisation when the file has been modified elsewhere.

I don't understand how you will achieve this without either caching the file within KeePassDX or using two SAF URIs for the same database ("always available local storage" and "remote storage").

You take the problem backwards, a URI is dedicated to a single database, but the principle of synchronization is that several files in different places represent the same database. The problem is the resolution of file conflicts, if we are sure that the files have been modified one after the other and that the previous file was up to date then no worries, the file manager can provide the latest version through the URI. If there is a conflict between two binary files, then simply give the user the choice of action:

• Either ignore one of the two files whose data will be lost, if the user is confident (not the solution we want but adopted by current file managers by default without user consent)
• Or provide a dynamic conflict resolution principle by editor (here by KeePassDX) which will take care of the merge. The first file A will be linked to the URI for the content provider, then after opening the file by the editor, the second file of location B can be manually linked by the user to the existing URI (which will unlink file A) and allow the merge in the editor and resolve conflict. (the same principle as git)

How would the following scenarios be handled? Assume that when "the network is unavailable" that the remote storage may also have been down for an extended period of time such that a SAF provider that normally pushes all changes immediately to local storage on the Android device also has an out of date version of the database.

I plan in FileSync to make a cache and history system:

• if no file is yet available, it is not offered by the SAF
• if a file is available in the history, propose the last available file
• if an update takes time by the network, the last file of the history is available by the URI until the file is completely downloaded, it thus replaces the source of data for the proposed URI and updates metadata.

There's no way for any application to know what the network conditions are in advance so it must always be loaded first from local storage (because this is always fast). The latest version would be loaded from remote storage (in the background) and merged. If there's a conflict then a notification would ask the user how to resolve it.

Exactly

Unless the SAF has a way for the application to request either a cached or up to date version from the same URI this is going to require two URIs or in-app storage.

It is from the file manager that the file change for the same URI will take place

Note: it is not acceptable to decide "the network is slow, therefore the user will have to wait longer and may have to retry". It's bad enough trying to use services over a slow connection; that doesn't need to be made worse by also having to re-download the entire password database before being able to login.

I agree

If the background load from remote storage failed then it won't be possible to save changes remotely because the remote file may be out of date.

Why not? It will just make another history file. If the file retrieved from the network corresponds to a different file from the history, it will cause a conflict to resolve. I'm thinking about a branch system in the history.

Unless the SAF has a way for the application to indicate that it is saving changes based on an out of date version that will later be merged by the application (because only KeePassDX can decrypt the file to do that) this is going to require two URIs or in-app storage.

No, if the file manager changes the file that points to the same URI.

If the SAF stores the file locally when the network is down then the remote file may also have been changed before the network is back up.

Indeed

Unless the SAF is able to request that the application do a merge of two file versions (because only KeePassDX can decrypt the file to do that) this is going to require two URIs or in-app storage (and explicit "do not store the file locally" behaviour when writing to the remote SAF URI).

No, same thing, the file manager can change the file that points to the URI already provided.

[nomis]

1/ correctly retrieve data and manage conflicts from two different sources at the file metadata level
Any cloud file manager should be able to handle synchronization conflicts regardless of the file type to correctly do the 1/. Here .kdbx files are considered as binary, there can be several strategies to manage conflicts at this level depending of file manager.

There is no acceptable strategy to handle a conflict of the KeePass database as a binary file. No user should be presented with the options of "lose data from Version A" or "lose data from Version B". This is probably the core of the problem with handling database syncing because you appear to consider this acceptable?

2/ managing the merge of data from two different database sources, and therefore of file content

I'm not proposing merging two different KeePass databases, only two versions of the same KeePass database.

Other applications can do this but I'm not interested in using that feature here.

Is there logging available in KeePassDX that would prove that it was provided with out of date metadata when it went to save the file?

Yes, this is how the merge works in KeePassDX.

In order to prove your claim that the SAF is at fault for providing the wrong metadata, I need log messages from KeePassDX that state what metadata it received from the SAF when the file was loaded and when the file was saved. The log messages would also indicate why the decision to overwrite the file was made.

If the log messages need to be enabled then I need advice on how to do that. If the log messages don't exist then I need advice on where to add it.

The reason for using log messages is that this is for debug only and a UI for "overwriting the file" isn't needed or expected.

You take the problem backwards, a URI is dedicated to a single database, but the principle of synchronization is that several files in different places represent the same database. The problem is the resolution of file conflicts, if we are sure that the files have been modified one after the other and that the previous file was up to date then no worries, the file manager can provide the latest version through the URI.

No, I'm just looking ahead to the only way to solve this without resorting to manual conflict resolution.

If there is a conflict between two binary files, then simply give the user the choice of action:

• Either ignore one of the two files whose data will be lost, if the user is confident (not the solution we want but adopted by current file managers by default without user consent)

As stated above, this is not acceptable. (I agree it's the default but that's how the SAF is designed. There's an expectation that devices are always online with fast reliable connectivity.)

• Or provide a dynamic conflict resolution principle by editor (here by KeePassDX) which will take care of the merge. The first file A will be linked to the URI for the content provider, then after opening the file by the editor, the second file of location B can be manually linked by the user to the existing URI (which will unlink file A) and allow the merge in the editor and resolve conflict. (the same principle as git)

Why would anyone want to resolve conflicts by having to manually set up a second URI for the other version of the file so that they can merge them? This isn't a difficult file format to merge automatically. No file manager works like this and I don't think anyone wants to go to this level of manual effort.

This is why I want two SAF URIs for the same database ("always available local storage" and "remote storage") because it is easy to maintain a copy in local storage that can be used to manage conflicts automatically.

How would the following scenarios be handled? Assume that when "the network is unavailable" that the remote storage may also have been down for an extended period of time such that a SAF provider that normally pushes all changes immediately to local storage on the Android device also has an out of date version of the database.

I plan in FileSync to make a cache and history system:

• if no file is yet available, it is not offered by the SAF
• if a file is available in the history, propose the last available file
• if an update takes time by the network, the last file of the history is available by the URI until the file is completely downloaded, it thus replaces the source of data for the proposed URI and updates metadata.

As far as I'm aware the SAF doesn't have a means of offering multiple versions of a file or asking the user what to do here, and it has to make a decision quickly (#1562) so it will always provide the latest cached version even if that's out of date?

I also don't expect the user to have to make an explicit decision every time. For example, I don't necessarily know the state of the network if I'm on a train - it might work or it might not but I'd prefer not to delay the application to find out.

There's no way for any application to know what the network conditions are in advance so it must always be loaded first from local storage (because this is always fast). The latest version would be loaded from remote storage (in the background) and merged. If there's a conflict then a notification would ask the user how to resolve it.

Exactly

You appear to agree but only from the perspective of the file manager. Only the editor can resolve conflicts and the SAF isn't able to provide this type of functionality.

Unless the SAF has a way for the application to request either a cached or up to date version from the same URI this is going to require two URIs or in-app storage.

It is from the file manager that the file change for the same URI will take place

But the file manager doesn't know the difference between "the editor has been opened and needs a cached version immediately" and "the editor can now retrieve newer content in the background"?

If the background load from remote storage failed then it won't be possible to save changes remotely because the remote file may be out of date.

Why not? It will just make another history file. If the file retrieved from the network corresponds to a different file from the history, it will cause a conflict to resolve. I'm thinking about a branch system in the history.

As above, this is a binary file. I don't want to be resolving conflicts by discarding one version of it.

Unless the SAF has a way for the application to indicate that it is saving changes based on an out of date version that will later be merged by the application (because only KeePassDX can decrypt the file to do that) this is going to require two URIs or in-app storage.

No, if the file manager changes the file that points to the same URI.

How does the file manager know the difference between "the out of date cached file that was being loaded is being saved with changes" and "the latest up to date version of the file is being saved with changes"?

Unless the SAF is able to request that the application do a merge of two file versions (because only KeePassDX can decrypt the file to do that) this is going to require two URIs or in-app storage (and explicit "do not store the file locally" behaviour when writing to the remote SAF URI).

No, same thing, the file manager can change the file that points to the URI already provided.

All you appear to be doing is repeatedly storing multiple copies of a binary file for the user to manually resolve. This is a password database format that can be edited from multiple devices at once. It's not appropriate to be trying to manage it as a binary file in any file manager.

[J-Jamet]

There is no acceptable strategy to handle a conflict of the KeePass database as a binary file. No user should be presented with the options of "lose data from Version A" or "lose data from Version B". This is probably the core of the problem with handling database syncing because you appear to consider this acceptable?

No, I'm just saying that's what most file managers do now because they don't merge on unknown (and even known) file formats. But if you don't find it acceptable as it is (me niether), why is it default everywhere in many cloud applications and no one is improving the concept of these apps?

I'm not proposing merging two different KeePass databases, only two versions of the same KeePass database.

Me neither, you misread, I'm talking about different sources from the same database (even if it doesn't really matter).

In order to prove your claim that the SAF is at fault for providing the wrong metadata

That's not what I'm saying, the storage access framework works well in Android system. It's the file providers of the buggy file managers that provide the metadata to the SAF that don't do their job.
Here is the code that checks the metadata info of the database file. You can debug as you wish, if you find bugs, I'll be happy to fix them :

KeePassDX/app/src/main/java/com/kunzisoft/keepass/services/DatabaseTaskNotificationService.kt

Line 154 in 428efd2

fun checkDatabaseInfo() {

No, I'm just looking ahead to the only way to solve this without resorting to manual conflict resolution.

Well, you will not make me add internet permissions to KeePassDX by saying that there is only one solution. If you want your "only" way, so simply choose KeePass2Android, which is to make the application a file manager and implement all the third-party APIs. This is precisely what I don't want to do and it is precisely to avoid doing this that KeePassDX was created. I list the reasons for this choice here : #1157 #342

And it's not fully manual, it's merging two files from two history branches, it's not complicated for the user to understand if there is the right UI and it will also facilitate conflicts for all types of files.

Why would anyone want to resolve conflicts by having to manually set up a second URI for the other version of the file so that they can merge them? This isn't a difficult file format to merge automatically.

I don't only resonate with this file format but with all file formats. Again, if you want an all-in-one solution, you just got the wrong app.

No file manager works like this and I don't think anyone wants to go to this level of manual effort.

If file managers don't work like this, that's precisely what causes the data loss issues that people complain about. So maybe it's time to improve the concept instead of pretending it's normal, don't you think?
If it's well presented with a nice UX/UI and solves many problems with all other file types, I don't see why. This is a very good solution for me but it must be done well.

This is why I want two SAF URIs for the same database ("always available local storage" and "remote storage") because it is easy to maintain a copy in local storage that can be used to manage conflicts automatically.

If you want to create a file manager with two URIs it is possible, yes. It will suffice to do "merge from ..." instead of "merge".
It's really the same concept as the split history branches I was talking about earlier except you use two URIs instead of linking one file after another to the same URI.

As far as I'm aware the SAF doesn't have a means of offering multiple versions of a file or asking the user what to do here, and it has to make a decision quickly (#1562) so it will always provide the latest cached version even if that's out of date?

From the file manager we can provide any file to the SAF, not necessarily the obsolete file of a cache, these are the rules for changing automatic URI links that must be done once the file is updated.

I also don't expect the user to have to make an explicit decision every time. For example, I don't necessarily know the state of the network if I'm on a train - it might work or it might not but I'd prefer not to delay the application to find out.

It is obvious that the maximum synchronization decisions must be automated, there will be problems only if a device modifies a file which is not synchronized on the network and that another device also modifies it. In reality, manual conflict handling will happen infrequently, as file hashes and dates will provide a solid branch basis for history. Minimal management for conflict is normal for me.

You appear to agree but only from the perspective of the file manager. Only the editor can resolve conflicts and the SAF isn't able to provide this type of functionality.

I repeat, we can send the base file in the URI and the file to be merged in the same URI once the database is open. The notification requesting to merge already in place in KeePassDX then does the merge job.

But the file manager doesn't know the difference between "the editor has been opened and needs a cached version immediately" and "the editor can now retrieve newer content in the background"?

I don't understand what you mean.
There is no notion of background data with the editor. The data from a new version of the file synchronized from the network will be retrieved asynchronously with a load bar in the file manager. Once the file is fully donwloaded, the URI will simply link to this new file, so the file's metadata will be changed in the SAF.
Then, either KeePassDX was open and is notified to offer a reload, or it was closed and a new opening will directly offer the downloaded content. Once the editor saves the data, a new file will be created and this will invalidate the last history file.

There will only be a conflict if the hash string of the history file does not match the last file. If this is the case, a merge resolution is requested by pointing to the URI of the last local history file and then pointing to the downloaded file.

As above, this is a binary file. I don't want to be resolving conflicts by discarding one version of it.

Me neither, that's why I've been talking about history from the beginning.

How does the file manager know the difference between "the out of date cached file that was being loaded is being saved with changes" and "the latest up to date version of the file is being saved with changes"?

Because there will be a history with branches (and hashes for each file in the history), the same as git. A conflict resolution procedure will arise if the history is no longer the same from one device to another. After that it's a question of protocol and implementation.

All you appear to be doing is repeatedly storing multiple copies of a binary file for the user to manually resolve. This is a password database format that can be edited from multiple devices at once. It's not appropriate to be trying to manage it as a binary file in any file manager.

There will only be conflict resolution if there is a conflict, which will happen very infrequently :

• if the user modifies his file on several devices at exactly the same time
• or if one of the devices does not reconnect to the network before the time the user will edit it on another device.
  These are clearly normal cases where the user can be asked to perform the conflict resolution / merge procedure.

[nomis]

There is no acceptable strategy to handle a conflict of the KeePass database as a binary file. No user should be presented with the options of "lose data from Version A" or "lose data from Version B". This is probably the core of the problem with handling database syncing because you appear to consider this acceptable?

No, I'm just saying that's what most file managers do now because they don't merge on unknown (and even known) file formats. But if you don't find it acceptable as it is (me niether), why is it default everywhere in many cloud applications and no one is improving the concept of these apps?

Because that would require amending the SAF to support asking the application resolving the conflict. It would have to attempt this in the background and then ask the user (with a notification) what to do to resolve it. If multiple applications modified the file how does the provider know which one should resolve it?

I don't think the SAF is ever going to get this kind of interface because it's for simple file types (images/videos) that are accessed by more than one application but are rarely edited. It's not really intended for a database. Anything more complicated (e.g. real-time Google Docs/Sheets editing) will use its own private APIs to the storage provider.

The only way to resolve this is for the KeePass application to store a local copy of the database, because it can do merging of the content between multiple copies of the database. (I don't care how it does this, but I would use in-app storage. It could be an optional feature.)

You agree that it's unacceptable but because it's the default behaviour of other cloud applications you can't/won't do anything to avoid it? Unless you extend the SAF API it appears that the user would have to manually resolve the conflict by instructing the editor to open version A and then merge data from version B.

No, I'm just looking ahead to the only way to solve this without resorting to manual conflict resolution.

Well, you will not make me add internet permissions to KeePassDX by saying that there is only one solution.

Nothing I've asked for requires this. I've already suggested that KeePassDX could use a "local storage URI" and "remote storage URI" so that it can resolve the conflicts itself (because the storage provider can't resolve binary conflicts).

And it's not fully manual, it's merging two files from two history branches, it's not complicated for the user to understand if there is the right UI and it will also facilitate conflicts for all types of files.

I'm not sure what UI you're imagining for an encrypted binary file format that a user could possibly use.

Why would anyone want to resolve conflicts by having to manually set up a second URI for the other version of the file so that they can merge them? This isn't a difficult file format to merge automatically.

I don't only resonate with this file format but with all file formats. Again, if you want an all-in-one solution, you just got the wrong app.

It's impossible to do this over two apps because the SAF doesn't expose that kind of functionality.

No file manager works like this and I don't think anyone wants to go to this level of manual effort.

If file managers don't work like this, that's precisely what causes the data loss issues that people complain about. So maybe it's time to improve the concept instead of pretending it's normal, don't you think? If it's well presented with a nice UX/UI and solves many problems with all other file types, I don't see why. This is a very good solution for me but it must be done well.

Because it's impossible to do this with a binary file format that the storage provider doesn't understand. What you're proposing either requires a SAF API to request the application that accessed the file resolves the conflict or asks the user to pick one of two versions to keep.

As far as I'm aware the SAF doesn't have a means of offering multiple versions of a file or asking the user what to do here, and it has to make a decision quickly (#1562) so it will always provide the latest cached version even if that's out of date?

From the file manager we can provide any file to the SAF, not necessarily the obsolete file of a cache, these are the rules for changing automatic URI links that must be done once the file is updated.

There's no UI or API in the SAF to accommodate this. Even if there was, having to select which version of the file you get every time you open it is not user-friendly.

The best user experience would involve loading the cached database in the foreground and then automatically merging the latest database in the background (when possible).

I also don't expect the user to have to make an explicit decision every time. For example, I don't necessarily know the state of the network if I'm on a train - it might work or it might not but I'd prefer not to delay the application to find out.

It is obvious that the maximum synchronization decisions must be automated, there will be problems only if a device modifies a file which is not synchronized on the network and that another device also modifies it.

This assumes that all files are replicated to all devices as soon as possible. This is only practical when using a large cloud storage provider. The rest of us have to cope with the storage provider and/or the Android device having no network connectivity for a period of time and make changes (e.g. WiFi passwords) without losing data.

You appear to agree but only from the perspective of the file manager. Only the editor can resolve conflicts and the SAF isn't able to provide this type of functionality.

I repeat, we can send the base file in the URI and the file to be merged in the same URI once the database is open. The notification requesting to merge already in place in KeePassDX then does the merge job.

How can the file manager possibly know that when KeePassDX asks for the file content the first time it wants a cached version but when it asks for the file content a second time (before saving or simply to give the user the latest content) it wants the latest version? I don't see anything in the SAF that does this.

But the file manager doesn't know the difference between "the editor has been opened and needs a cached version immediately" and "the editor can now retrieve newer content in the background"?

I don't understand what you mean. There is no notion of background data with the editor. The data from a new version of the file synchronized from the network will be retrieved asynchronously with a load bar in the file manager. Once the file is fully donwloaded, the URI will simply link to this new file, so the file's metadata will be changed in the SAF.
Then, either KeePassDX was open and is notified to offer a reload,

Where is this notification API for doing this in the SAF?

or it was closed and a new opening will directly offer the downloaded content. Once the editor saves the data, a new file will be created and this will invalidate the last history file.

This still doesn't solve the problem of overwriting the file (on save) based on opening a cached version before the SAF has been able to retrieve the latest version from the network. There's no way of knowing what the SAF is up to in the background.

You're expecting the SAF to provide a cached version on open and the latest version before it is saved but there's no SAF API to do this. There's no way of requesting that a caching storage provider returns the up to date content or an error if it's unable to do so. Even the refresh function can only ask the storage provider to try to refresh the content. It doesn't tell you that it tried but failed to refresh it.

If it fails to refresh the content then there's then no way to safely save the open database, because it will cause a binary conflict that the storage provider can't resolve on its own.

How does the file manager know the difference between "the out of date cached file that was being loaded is being saved with changes" and "the latest up to date version of the file is being saved with changes"?

Because there will be a history with branches (and hashes for each file in the history), the same as git. A conflict resolution procedure will arise if the history is no longer the same from one device to another. After that it's a question of protocol and implementation.

If the file manager won't communicate with the editor to resolve the conflict then the user's choice is still going to be "here are two versions of the file, you will lose data if you pick one of them".

All you appear to be doing is repeatedly storing multiple copies of a binary file for the user to manually resolve. This is a password database format that can be edited from multiple devices at once. It's not appropriate to be trying to manage it as a binary file in any file manager.

There will only be conflict resolution if there is a conflict, which will happen very infrequently :

• or if one of the devices does not reconnect to the network before the time the user will edit it on another device.
  These are clearly normal cases where the user can be asked to perform the conflict resolution / merge procedure.

This assumes the "large cloud provider" mode of operation I mentioned above. I can't have my file provider regularly polling in all network environments to find out if the file has changed.

In other environments there will be a cached version of the file that may be out of date. It's ridiculous to expect the user to manually resolve binary file conflicts that could be resolved automatically by the application that understands the database format.

---

If you want your "only" way, so simply choose KeePass2Android, which is to make the application a file manager and implement all the third-party APIs.

There's nothing here that requires doing that. Keepass2Android has a cache that works if there's no connectivity (i.e. it's still impractical on slow networks without manually selecting the cache) but it has a convoluted build process so I'm looking for an alternative.

The existence of a local cache is separate from how the remote content is accessed. It would be possible to make Keepass2Android use the SAF and it would be possible to add a local cache to KeePassDX.

Using a single-URI SAF with caching on the provider side without an API to properly control it isn't going to work well for a binary file.

[J-Jamet]

One thing is non-negotiable, KeePassDX must not contain any data, even encrypted, from the user's keepass database. It's only an editor, it has been indicated from the beginning, that's what the application was designed to do. And I've explained why, so that it's never the entry point for any attack. Once the database is closed, KeePassDX no longer contains anything of interest. This allows you to put the file on an OTG usb key and only the data will be on the key.

Remember that the aim of KeePassDX is to manage all your database data in a single file.
So yes, this has its huge advantages, but it also has its limitations: synchronization must be based on the principle of a binary file.
For sure it's a pain for that but if the purpose of the user is to correctly synchronize each piece of data, then there are much more suitable tools that will work every time. https://github.com/android-password-store/Android-Password-Store

If we're still using synchronization tools that are incapable of providing solutions (even manual ones) to their users in case of conflict, there will always be problems in any case. Based on this constraints, I propose the solutions I've already explained to solve the global binary sync problem.
Cached files are already available in file manager applications, but they're poorly managed, as we've seen with Google Drive. How many times do users complain that their cache isn't up to date, and this has nothing to do with KeePassDX.

From what you say, SAF is very limited, but it's powerful enough to let different data through if you ask it to send different data, so enabling a merge if different data are sent manually at different time.
On the other hand, the UX must be clear enough for the user to understand the notion of conflict and data merge with a separate application. But currently, all synchronization applications don't care about this. That's what I want to change.

The aim is to keep any type of file that can be merged by the same URI of the content provider, and that's completely feasible. If you don't like it, it'll be done with two different URIs when KeePassDX is able to manage the opening and merging of several databases simultaneously. It requires a lot of work, but it's not technically unfeasible.

[nomis]

From what you say, SAF is very limited, but it's powerful enough to let different data through if you ask it to send different data, so enabling a merge if different data are sent manually at different time. On the other hand, the UX must be clear enough for the user to understand the notion of conflict and data merge with a separate application. But currently, all synchronization applications don't care about this. That's what I want to change.

I don't want to use a synchronisation application that has to be manually told to send different data at different times.

The aim is to keep any type of file that can be merged by the same URI of the content provider, and that's completely feasible. If you don't like it, it'll be done with two different URIs when KeePassDX is able to manage the opening and merging of several databases simultaneously. It requires a lot of work, but it's not technically unfeasible.

I can't tell if you agree with me or not - I realise it's not trivial to do, but can I eventually have a URI using local storage for offline use and a URI using remote storage for online use and automatically use/sync both of them based on availability? (A UI only being required when there's a conflict that can't be resolved automatically.)