(Feature request)sync server implementation in rust

[dobefore]

Describe feature

Hi!This is a great project.
I would like to implement sync server logic in rust,which could be build into more platforms.

Would you mind providing me some suggestions or js code guidance on where can i find conterpart of js implementaion?
Much appreciated!

Additional Information

No response

[zadam]

Hi, I guess this would be a good starting point: https://github.com/zadam/trilium/blob/master/src/routes/api/sync.js

There's no documentation of the protocol aside from its implementation, but it's also not very complicated protocol ...

[sigaloid]

I too have been interested in this - I have a bit of experience with server backends as a Libreddit maintainer for a little while now. @dobefore, would you be willing to join forces?

@zadam: I have a question. Assuming the sync protocol is replicated, what functionality would be taken away? I assume web app functionality would be gone. Would it be possible to statically host the web app in a way that still provides all of the JS/CSS resources necessary? ie, is the web version's only necessary endpoints the actual app code + sync protocol routes? Or are there other routes that would need to be recreated?

[dobefore]

@zadam ，Thanks!

@sigaloid ,I sure would like to. Yet I am afraid I have little spare time to finish this.

[zadam]

@sigaloid

Would it be possible to statically host the web app in a way that still provides all of the JS/CSS resources necessary?

The web version needs all the backend code (which is not small), not just the sync protocol part and static resources.

Desktop and web versions are almost identical, the desktop version has just a small electron integration added.

[sigaloid]

Thank you!

Would you mind giving a short explanation on how the syncing architecture works? If I wanted to replicate just the server-side syncing functionality, what do I need to have set up? I assume a functional database, probably having executed the schema.sql in db/.

In addition, I'm a bit confused about what code the server would need to replicate and which is just for the client to sync, though I understand that the differentiation of client/server might be a little blurry given that the server can also sync to another server. For example, do all of these routes need to be implemented for a client to be able to trigger a sync to a server? https://github.com/zadam/trilium/blob/master/src/routes/routes.js#L210-L221

I believe I also have to replicate the /api/setup routes, because these routes are how the server is set up with the client. Is this correct?

I appreciate any guidance you would be willing to give.

[zadam]

For example, do all of these routes need to be implemented for a client to be able to trigger a sync to a server? https://github.com/zadam/trilium/blob/master/src/routes/routes.js#L210-L221

The first five are called from GUI/Options and not part of sync protocol. The rest needs to be implemented + few others - some of the /setup/sync*, /login/sync. You need to execute the schema.sql but then also be able to run DB migrations (db/migrations), since the DB changes from time to time. The DB migrations might be tricky since some of them are actually a JavaScript migration code. Those won't change the DB schema, but may e.g. rename entities (where equivalent SQL might be too complex).

I understand that the differentiation of client/server might be a little blurry given that the server can also sync to another server.

[sigaloid]

Thank you so much for that, it was very helpful.

The DB migrations might be tricky since some of them are actually a JavaScript migration code.

I assume that if I implement the code as-is, I don't need to implement past migration code, correct? Just in the future I need to implement any new migrations?

Also in the case of the server websockets component, it looks like the local app doesn't require it - is it true that the websockets only is used in the web app?

Final question: Can you describe what the cls.getComponentId does, in cls.js? It looks like namespace is a hashmap of some kind, and each route sets a value within it when it's called. But I can't figure out what it does, nor its relevance to the server syncing alone - entityChangesService.addEntityChange relies on it, so it's pretty far down the dependency tree and looks like lots of things use the entity changes service.

I mapped out each function and its dependents as well, for each route required to implement the sync (I even used Trilium to map it :P). https://gist.githubusercontent.com/sigaloid/89c1f113d2d502b3d5f4a42f7de45d12/raw/8ee4021dd696cd8299c78a68165d18ab61f656eb/Rewrite.svg

[zadam]

I assume that if I implement the code as-is, I don't need to implement past migration code, correct? Just in the future I need to implement any new migrations?

Yes. It will need to be able to upgrade the existing DBs to newer versions.

Also in the case of the server websockets component, it looks like the local app doesn't require it - is it true that the websockets only is used in the web app?

No, websockets are used for desktop app as well.

Can you describe what the cls.getComponentId does, in cls.js?

entityChanges are used for 2 things - sync between Trilium instances using sync protocol and to sync the frontend from backend via websocket. The componentId identifies the frontend component which triggered the data change - the change will be sent via websocket to the app, and the component can use it to answer "was it me who caused this change?". Useful e.g. for CKEditor instances - if it was this instance who sent the update upon keystroke, it's already updated and doesn't need to reload (all other instances having the note loaded will have to reload).