[OUTDATED] Tarantool backup/restore

[nshy]

This RFC is outdated. New one is here.

Reviewers:

• Expert: @locker
• Expert: @Gerold103
• Expert: @Serpentian
• Team Lead: @sergepetrenko
• CTO: @sergos

TOC

• Changelog
• Links
• Document aims
• Use cases
• Backup consistency
• Scope
• Single instance
  • Backup
    • Incremental backup
  • Recovery
  • Introspection
• Synchronous replicaset
  • Incremental backup
• Non synchronous replicaset
• Cluster
  • vshard

Changelog

• v4 12/02/2026: Kept box.backup.start() result backward compatible, added more info to box.backup.info(), dropped requirement making backup from master for synchronous replicaset.
• v3 23/01/2026: Described incremental backup and introduced box.backup.info().
• v2 04/12/2025: Described steps to backup/restore explicitly. Added technical details to make sure xlogs has only commited data in case of synchronous replicaset. Added multimaster/asynchronous master-replica replicaset backup/restore. Described backup in case of vshard. Made misc changes to improve document structure.
• v1: Added initial version.

Links

Github issue #11729 (which holds further references also) .

Document aims

The purpose of the document is to describe how to backup and restore Tarantool at instance, replicaset and cluster level. We describe API only at instance level (vshard is exception, it is an example of cluster backup), all other steps are should done by backup agent. Also we expect tt CLI will provide more user-friendly interface for backup/restrore on base of this RFC.

Not all described below is how Tarantool currently works and rather is how we plan to make it work in terms of backup/restore.

Use cases

Backup is done to restore after all data is lost.

Other known use cases:

1. Restore cluster after full data lost when there is standby cluster. If standby cluster if distant, then restoring lost cluster from it will take more time then first restoring from local backup and then copying only difference from standby cluster. The same can be applied to replicaset spread among datacenters when restoring a lost replica.

Backup consistency

We do not elaborate here making replicaset/cluster backup consistent in terms it represents state at some moment in global time as there is no such yet. However replica/shards has data at the "moment" of backup start. It may differ from replica to replica and from shard to shard due to network latencies, replica failures, internal events which may delay backup start (see technical details for asynchronous replicaset backup).

Scope

Tickets mention PITR (point in time recovery). Current argeement is that incremental backups will provide enough points of consistent recovery so we don't need anything more fine grained.

Single instance

Backup

When box.backup.start() is called WAL is rotated and function returns a list of files required to restore to the current point. It is last snapshot and all WAL files after it up to rotated. Backup agent is supposed to copy the listed files. It may optimize backup storage and copy only new WAL files if there was no new snapshot since the last backup. After files are copied call box.backup.stop().

Example:

> box.backup.start()
---
- - ./00000000000000001111.snap
  - ./00000000000000001111.xlog
  - ./00000000000000002222.xlog
...

So to backup such replicaset we need next steps:

1. Call box.backup.start() on instance.
2. Copy files listed in the above call where required.
3. Call `box.backup.stop().

The above is example is for memtx only spaces. In case of there are vinyl spaces the list of data files will include *.vylog, *.index and *.run files.

Technical details

In case of vinyl the list will not include *.index and *.run files not included in *.vylog. For example, files dumped since last snapshot. We are going to use *.xlog to restore data written since last snapshot. Currently vinyl cannot dump memory during xlog recovery, which may be triggered, so restoring from such a backup is not possible currently. We are going either to support dump during recovery or require that recovery should be done without xlog and then xlog should be applied using tooling (tt).

Incremental backup

In case of incremental backup we should be able to backup only difference since last full/incremental backup. In this case backup procedure is similar to what described in regular backup but backup is started by box.backup.start(type='full') or by box.backup.start(type='incremental') call.

If one requests incremental backup when there was no previous full or incremental then full backup is done. Also data files for incremental backup are kept for limited amount time set by new configuration option backup_gc_timeout. If incremental backup is requested after this period of time then also full backup is done.

To facilitate client check whether full or incremental backup was done we add key type to table returned by box.backup.info(). Like in case of starting backup it can have value full or incremental. If backup is not full nor incremental (regular) then type = 'default'.

Introspection

New box.backup.info() function will return list of backup files like in box.backup.start() and extra information. If there is no active backup then nil is returned.

Example:

> box.backup.info()
---
files:
  - ./00000000000000001111.snap
  - ./00000000000000001111.xlog
  - ./00000000000000002222.xlog
type: full
mode: default
vclock_start:
  - 555
  - 222
vclock_end:
  - 888
  - 1334
time: 1770903102
...

Here:

• vclock_start - vclock of the first statement in the backup files
• vclock_end - vclock of the statement in the backup files
• time - Unix-time when backup was started
• type - type of the backup (default/full/incremental check incremental backup section)
• mode - check backup of non synchronous replicaset section

Recovery

To recover an instance one need to put all data files (listed on backup in box.backup.start()) in working directory of instance before start if backup is full. If backup is incremental then one need to put files from full backup and all incremental backups since that up to required point.

Synchronous replicaset

In this case it is enough to backup only master. We cannot have too outdated data in this case. Notion of master can be up-to-date or not. The latter case is when there is new term and new master this this term and this instance does not know it yet and consider itself a master. If master is up-to-date then it holds all the committed data up to now. If master is not up-to-date then the replicaset can hold new committed data but as master can continue to consider itself a master only for election timeout the amount of this data is limited.

In this case it enough to backup only single replica. It can be master or replica if it is a follow state.

So to backup such replicaset we need next steps:

1. Backup selected replica as described in single instance backup.
2. Store each replica instance UUID in backup metadata (instance UUID is available through box.info.uuid for example).

To restore such replicaset we need next steps:

1. Copy backup data files in each replica working directory.
2. Fix instance UUID in each data file header to the UUID saved in backup metadata. Data file header is plain text. One need to replace value for header key Instance.

More technical details on replicaset recovery from single instance backup are in #12039.

Incremental backup

Incremental backup will be not possible if backup is started on replica different from the previous incremental/full backup. Though the incremental backup will not fail - the full backup will be done.

There is another issue we need to take care due to changing replica being backed up. For example we make full backup F1 at replica A, then we make full backup F2 at replica B, then we request incremental backup I3 at replica A again. I3 is a difference since F1 which is not yet garbage collected by chance. Client probably expects that the I3 holds the difference from F2. Obviously client can tackle this situation and do not request incremental backup after changing the replica being backed up.

Technical details

Without extra precautions the xlogs can have uncommitted transactions. These transactions can be rolled back later in replicaset history but on restore they can be applied. So we may have statements after restore that never be visible in replicaset history. We can avoid that if we wait all uncommitted transactions that get into backup xlog to be committed. If they get rolled back then box.backup.start() should raise error. There should be special error code, so the client can retry starting backup on this error as error is transient.

Non synchronous replicaset

This can be multimaster replicaset and asynchronous master-replica replicaset. In both cases making backup of only a single instance from replicaset as described above can miss some data. For example, in case of multimaster the replication can be paused due to long standing conflict, so instances can have different statements. If we backup only one of the instances we miss statements from the other that are not replicated. As conflict can exist for a long period of time we can miss data in backups for this period.

So to backup such replicaset we need next steps:

1. Backup every instance of replicaset as described in single instance backup. Start backup with box.backup.start({mode='replicaset'}), then call box.backup.info() to get extra information for backup. It will have vclock_start and vclock_end keys in this mode. These are vclock range of statements present in the data files of the backup. Backup agent should check that intervals of all replicas are overlapped for each vclock component. In this case there will be no rebootstrap after restore. In case the condition is not met, the backup should be restarted (box.backup.stop()/box.backup.start({mode='replicaset'})).

Example:

> box.backup.info()
---
files:
  - 1: ./00000000000000000777.xlog
  - 2: ./00000000000000001111.snap
  - 3: ./00000000000000001111.xlog
  - 4: ./00000000000000002222.xlog
type: full
vclock_start:
  - 555
  - 222
vclock_end:
  - 888
  - 1334
...

To restore such replicaset we need next steps:

1. Copy corresponding backup data files for each replica to their working directory.

Technical details

In backup mode 'replicaset' we list all extra xlogs the other replicas need to connect without rebootstrap, besides last snapshot and xlogs after it.

There still a chance that rebootstrap will be required. This can happen due to race. We make backup of instance A, then we make backup of instance B. Before that B advances gc vclock for A. So backup of instance B can miss some statements required for A. We can deal with that by inspecting vclock intervals present in box.backup.info() output. We add vclock_start and vclock_end to the box.backup.info() output in mode='replicaset'. There will be no rebootstrap if intervals of all replicas are overlapped for each vclock component. This check should be done by backup agent.

Cluster

Mere backup of every replicaset in cluster without extra coordination may be inconsistent for 2 reasons.

1. Rebalancing, when data is moved from one replicaset to another. The data being migrated may be lost or duplicated.
2. Cross-shards transactions. The changes may be applied in order unexpected to application.

We can take full cluster write lock during backup to exclude both cases but this way backup may impact cluster performance significantly. At the replicaset level backup is lightweight.

Another approach can handle issue 1 but not 2. We can abort/finish in progress data migrations and disable new ones before starting replicasets backup. After it is started data migrations are enabled again. This can be done fast and does not reduce cluster performance. As to issue 2 we can only rely on application in the latter approach, that the application can restore consistently by itself somehow.

vshard

In case of vshard we can use vshard.router.map_callrw() to start backup on every shard. This way all in progress rebalancing will be finished before starting backup. vshard consists of synchronous replicasets, so we need synchronous replicaset backup (as described in section above) for every shard.

So to backup vshard cluster we need next steps:

1. Call vshard.router.map_callrw() with function box.backup.start(). Make each shard backup as described in section for synchronous replicaset backup.

To restore cluster we need next steps:

1. Restore each shard as described in section for synchronous replicaset backup.

[sergepetrenko]

Restore cluster after full data lost when there is standby cluster. If standby cluster if distant in terms of data replication we can restore damaged cluster from local backup and only then align it with standby cluster.

Let's state the benefit of doing so: AFAIU we assume replication will take more time than local backup recovery.
Also the same applies to restoring a single node of a replica set, if other nodes are distant.

[nshy]

Rewrited the passage to mention explicitly time, mentioned replicaset also.

[Serpentian]

It seems, that this will be the main RFC, which unites the replicaset and cluster backup. Currently, it's very unclear to me, what we're doing, way too many questions.

Strict overview of motivation?

Let's firstly figure out, why do we implement that and what do we want to achieve at the end. We must strictly describe the goals of the RFC (e.g. do users wanna see the Point-In-Time recovery or not, according to the https://jira.vk.team/browse/TNTP-2825 they do) and the guarantees, we give to users.

For guarantees, we can check backup tools for other databases:

• PostgreSQL Patroni/Etcd — pgBackRest / WAL archiving
• MySQL Galera — Percona XtraBackup
• Cassandra — nodetool snapshot

And please, include the links to the associated github and jira tickets, it's very difficult to find them now.

How will the process look for the end user?

We must determine, how the backup process will look like for the end user. Is he going to take the tool from SDK, configure it and start the backup? In that case the tool should automatically move the needed files to the configured servers. Then a user just calls the tool one more time and it restores the cluster from a backup?

Or do we expect user to call some vshard/aeon function, that will return, which files should be copied and from which server, user manually goes to every instance, copies files to some servers and then uses tool to restore the cluster?

Or is it going TCM or/and ATE?

From the first glance, it loooks like we need all.

API of replicaset/cluster

Then we should define, how API of the replicaset/cluster will look like, this will be called by a user or our tool.

Will we use box.backup.start/info/end(), as it's proposed here.

Will writing the metadata (e.g. timestamp, instance info) of the backup to a file be a separate API? Or box.backup.info{write = true?

Will we have vshard/aeon.router.backup()?

Review

Rebalancing, when data is moved from one replicaset to another. The data being migrated may be lost or duplicated.

It may happen in VShard, if it's done without any protections:

1. Backup tool comes to rs1, doesn't find data there, makes backup
2. rs2 sends data to rs1
3. Backup tool comes to rs2, doesn't find data there, makes backup.
4. Data is lost

For that we could use already existing vshard.router.map_callrw, which prohibits rebalancing and makes sure, that all buckets are writable on the instance. It give guarantees, that the request will be done everywhere or nowhere. Write lock is not needed.

We can abort in progress data migrations and disable new ones before starting
replicasets backup

It's not possible in VShard now and I'm not sure it's possible to implement that at all, while preserving the safety. If that's needed, we'll have to write careful RFC for that to investigate.

[Gerold103]

I agree with almost everything what Nikita said. Only will leave meta-comments here:

Or do we expect user to call some vshard/aeon function, that will return, which files should be copied and from which server, user manually goes to every instance, copies files to some servers and then uses tool to restore the cluster?

Yes. Any sort of consistent cluster backup is only possible, when the replicasets can't exchange data. And this can only be done now by non-core frameworks like vshard and whatever else is on top of the core at the same level (Aeon I guess?).

Inside replicaset we could in theory invent something like pause GC on all replicas, collect xlogs, and then "merge" them. Could be done with a tool. Maybe even with the built-in xlog Lua module with some code on top of it.

But in a cluster we won't have this available. Because no vclock-like logical clock exists in the whole cluster, allowing to see which nodes did which changes.

This backup thing will need to be cluster-tech-aware. VShard and Aeon will need to be explicitly supported.

We can abort in progress data migrations and disable new ones before starting
replicasets backup

It's not possible in VShard now

Abortion isn't possible, but it is quite possible to wait for the end of the current migrations and not allow new ones. Via vshard.router.map_callrw(). No? During this call no bucket moves will be happening.

[nshy]

This backup thing will need to be cluster-tech-aware. VShard and Aeon will need to be explicitly supported.

As consistent cluster backup is not now possible a different approach is worked out in current backup vision doc. This RFC is outdated now, new version in moved here.

Abortion isn't possible, but it is quite possible to wait for the end of the current migrations and not allow new ones. Via vshard.router.map_callrw(). No? During this call no bucket moves will be happening.

Thanx! @Serpentian suggested to use vshard.router.map_callrw() too. It's usage is described in vision doc.

[nshy]

Strict overview of motivation?

Improved this part, hopefully addressed all the questions.

How will the process look for the end user?
API of replicaset/cluster

The high level API is outside of this RFC goals. Here we only describe Tarantool API that can be used for backup/restore by backup agent.

At this point it is not clear why we should add extra API to fetch instance config. It is already should be known due to config in Tarantool 3 or one can call box.cfg{} to fetch it.

Review

Thanx for suggestion! This part was abstract so added concrete steps for vshard (using vshard.router.map_callrw() as suggested).

[locker]

So to backup such replicaset we need next steps:

1. Backup current replicaset master as described in single instance backup.
2. Store each replica instance UUID in backup metadata (instance UUID is available through box.info.uuid for example).

Is it necessary? Can't we recreate a replicaset with different UUIDs, maybe even with a different replication factor?

[nshy]

Yeah looks like it is optional. But it may be required if there are bindings to UUIDs from somewhere else.

[locker]

For example, in case of multimaster the replication can be paused due to long standing conflict, so instances can have different statements.

Do we actually support multimaster setups? Is it possible to configure one with Tarantool 3.0 config?

[nshy]

Yeah, we have a master-master section in documentation for Tarantool 3.0.

[locker]

1. I don't see any point in implementing incremental backup API in Tarantool core because snapshots+xlogs already do incremental backups, in fact. Introducing something like checkpoint_interval for backups seems to be too much: checkpoint_interval should be configured to do incremental backups properly. Vinyl produces incremental backups internally basing on data size without setting the time interval explicitly. All we need to do is rotate xlog and include it into backup.
2. Extending box.backup.start() return value isn't backward-compatible (think of pairs). Let's leave it as is. box.backup.info() should include the vclock, time, status, maybe something else + list of files. The list of files should be return in a separate field.
3. I don't see much point in going into low-level details when describing replicaset backup. We just need to ensure that it's possible to restore a replicaset from a replica backup and how to do that. How to choose a replica for a backup is up to the backup application. They may always choose the first replica in the replica set or the leader or have an option.

[sergepetrenko]

checkpoint_interval should be configured to do incremental backups properly

AFAIU it's typical for backup applications to perform full backups once a day, and cover all the intermediate time slots of a day with incremental backups.

One of the ideas is to collect incremental backups each 15 minutes to be able to do point-in-time recovery with up to 15 minute granularity (of course, choosing the time interval will be the job of the backup tool or its configuration, not tarantool).

I don't see any point in implementing incremental backup API in Tarantool core because snapshots+xlogs already do incremental backups, in fact

I'm not sure the single API box.backup.start{type='full'} will handle all the cases with incremental backup. I see at least one problem:
Imagine the user has checkpoint_count = 1, checkpoint_interval=C and the backup tool has incremental backup collection interval B.
Regardless of C and B values it's possible, that a full backup is collected at moment T1, when data dir contains:

111.snap
111.xlog
222.xlog -- just rotated because of a backup.start() request

Then at the moment T1 + B we want to collect an incremental backup, but the data directory contents are now:

333.snap
333.xlog
444.xlog -- just rotated.

All the operations between 222.xlog and 333.snap will be lost. So it's impossible to collect an incremental backup without holding a reference for 222.xlog somewhere, in _gc_consumers space probably.

This is what type='incremental' handle is for, AFAIU.

[nshy]

I don't see any point in implementing incremental backup API in Tarantool core because snapshots+xlogs already do incremental backups, in fact. Introducing something like checkpoint_interval for backups seems to be too much: checkpoint_interval should be configured to do incremental backups properly. Vinyl produces incremental backups internally basing on data size without setting the time interval explicitly. All we need to do is rotate xlog and include it into backup.

Kept incremental backup API as discussed on f2f discussion.

Extending box.backup.start() return value isn't backward-compatible (think of pairs). Let's leave it as is. box.backup.info() should include the vclock, time, status, maybe something else + list of files. The list of files should be return in a separate field.

Fixed. However I did not added status field, not sure what it for.

I don't see much point in going into low-level details when describing replicaset backup. We just need to ensure that it's possible to restore a replicaset from a replica backup and how to do that. How to choose a replica for a backup is up to the backup application. They may always choose the first replica in the replica set or the leader or have an option.

Fixed.

[sergepetrenko]

If backup is not full nor incremental (regular) then type = 'default'

How is it different from a full backup? Isn't it better to just have 2 options, either 'full' or 'incremental'?

Introspection
New box.backup.info() function will return the same information as box.backup.start() if there is active backup or nil otherwise

Does this info survive a node restart?

Store each replica instance UUID in backup metadata (instance UUID is available through box.info.uuid for example).

I think this info should be returned via box.backup.start()/info() as well -- basically putting box.space._cluster:select{} contents there is enough.

box.backup.start({mode='replicaset'})

AFAIU this mode only differs in backup.start()/backup.info() output -- it adds vclock_start and vclock_end fields. Right? If yes, maybe better return these fields in all backup modes and drop mode = 'replicaset' altogether?

[nshy]

How is it different from a full backup? Isn't it better to just have 2 options, either 'full' or 'incremental'?

In case of 'full' backup we want to pin snapshot it is rooted from. So when next snapshot is done we will be able on fetch only difference on next incremental backup.

Does this info survive a node restart?

I guess not. Currently checkpoint pins of backup are not restored after restart, so backup cannot be continued after restart generally speaking. However it looks like a useful feature. Should we add restart support to the RFC? We can put pins in existing _gc_consumers and have a timeout for purging backup pins from there (there is backup_base_gc_timeout for incremental backups, we can rename it to backup_gc_timeout I guess).

AFAIU this mode only differs in backup.start()/backup.info() output -- it adds vclock_start and vclock_end fields. Right? If yes, maybe better return these fields in all backup modes and drop mode = 'replicaset' altogether?

There is another difference, in 'replicaset' mode we provide more xlogs to avoid rebootstrap on restoring from backup. Or we should hide non synchronous replicaset section to move it out of current RFC consideration?

[sergepetrenko]

In case of 'full' backup we want to pin snapshot it is rooted from. So when next snapshot is done we will be able on fetch only difference on next incremental backup.

So, 'regular' backup fetches the same data as 'full' one, but doesn't pin snapshot/xlogs, right? Is it for the sake of users who don't do incremental backups?

I guess not. Currently checkpoint pins of backup are not restored after restart, so backup cannot be continued after restart generally speaking. However it looks like a useful feature. Should we add restart support to the RFC? We can put pins in existing _gc_consumers and have a timeout for purging backup pins from there (there is backup_base_gc_timeout for incremental backups, we can rename it to backup_gc_timeout I guess).

Ok, I see. I think we shouldn't do this unless someone asks for it explicitly.

There is another difference, in 'replicaset' mode we provide more xlogs to avoid rebootstrap on restoring from backup. Or we should hide non synchronous replicaset section to move it out of current RFC consideration?

Oh, so we find the oldest xlogs still needed by some replicas and return them as well?
Ok, I see now, thanks!

After a second thought all these modes do make sense. Although it's a bit confusing that there are so many of them.

[nshy]

So, 'regular' backup fetches the same data as 'full' one, but doesn't pin snapshot/xlogs, right? Is it for the sake of users who don't do incremental backups?

Exactly.

[Serpentian]

Sorry, I still have a lot of questions regarding that feature)

Also we expect tt CLI will provide more user-friendly interface for backup/restrore on base of this RFC.

Still, same comment as in the https://github.com/orgs/tarantool/discussions/12039. I don't think we should rely on tt team to figure this all out, if we decided to design such document. Since now it's not obvious, how the restore process will look like to the end user. Instead, we can invite them to review the document and think about the API together.

When box.backup.start() is called WAL is rotated

It also suspends all removal of outdated backups (I'd rather state it in the RFC) and it may be a problem. I'm afraid, that there may be users, which will go to every master in cluster and call smth like:

function backup_start()
    box.backup.start()
    return box.backup.info()
end

After that they'll go to every instance and copy the files and then try executing the following on all masters:

function backup_end()
    box.backup.end()
end

But it may fail on some of them (no connection e.g.) and then we'll get the instances, which never delete the backup files, that will sooner or later lead to incident, disk memory is not infinite.

If we expect user to use the backup as follows in terms of cluster:

function backup_start()
    local files = box.backup.start()
    -- do smth with `fio` module
    box.backup.end()
end

I'd propose introducing force close of backup on exit from function call, as we do it for transactions. However, it may have performance impact, which I don't really like.

If we intentionally allow user to split start() and end() between funcitons, I'd consider introducing timeout option to box.backup.start(). We can make it equal infinity on 3.X for backward compatibility and switch under compat to something less in 4.X.

box.backup.start(type='full')

This one is not backward compatible, the first argument to box.backup.start() is number already, so it'll be:

box.backup.start(0, {type = 'full'})
-- or
box.backup.start(nil, {type = 'full'})

should be able to backup only difference since last full/incremental backup

Did I understood u correctly, that if we had the following snap done before:

- - ./00000000000000001111.snap
  - ./00000000000000001111.xlog
  - ./00000000000000002222.xlog

And new xlog file appeared, then during incremental backup only it'll be returned:

  - ./00000000000000005555.xlog

Then the question, how will the tarantool remember, what files were in the last backup. Will this info survive instance restart? If so, where it'll be persisted.

Also data files for incremental backup are kept for limited amount time set by new configuration option backup_base_gc_timeout. If incremental backup is requested after this period of time then also full backup is done.

I didn't parse that sentence, could you elaborate please?

To facilitate client check whether full or incremental backup was done we add key type

> box.backup.start()
---
---
- 1: ./00000000000000000777.xlog
  2: ./00000000000000001111.snap
  3: ./00000000000000001111.xlog
  4: ./00000000000000002222.xlog
  type: 'full'

Key-value and index-based in the same table looks way too crutchy. In terms of box.backup.start() we're limited with backward compatibility. I'd rather return only the list of files from box.backup.start() and everything else and also files from box.backup.info(), in which we can do whatever we want:

> box.backup.info()
---
---
- files: - ./00000000000000000777.xlog
         - ./00000000000000001111.snap
         - ./00000000000000001111.xlog
         - ./00000000000000002222.xlog
- type: 'full'
<and so on...>

WDYT?

Incremental backup

And the last question about icremental backups. Why do we want it at all? Do we have any users, which request it? It's way too complicated to implement in tarantool such thing and to client application it costs nothing: he will just send a list of files he already has to the function, will start a new backup and skip the files, which are already persisted. That's it.

I'd think twice, do we want it in core) Moreover, as you already stated, we have problems with master changes.

Start backup with box.backup.start({mode='replicaset'}). Backup start will return vclock_start and vclock_end in this mode

And why don't we want to always return that info and not introduce that replicaset mode? It doesn't differ from any other, AFACS. Only the return values is changed.

In case the condition is not met, the backup should be restarted (box.backup.stop()/box.backup.start({mode='replicaset'}))

Hmm, and it'll be considered successfull full backup and will affect incremental once after it? Even when user doesn't like it and didn't copy the files? That strange. Maybe we need smth like box.backup.abort then, but I'd still consider not implementing incremental backups.

This can be multimaster replicaset and asynchronous master-replica replicaset. In both cases making backup of only a single instance from replicaset as described above can miss some data

Multi-master setup doesn't exist in productions, but the async cluster is obviously the most popular. And here's the question, does our clients take backups from all instances in every replicaset? I'm pretty sure, they don't.

These are vclock range of statements present in the data files of the backup. Backup agent should check that intervals of all replicas are overlapped for each vclock component.

I didn't understand, what is meant here and pretty sure tt team also won`t be able to parse. Could you elaborate?

In backup mode 'replicaset' we list all extra xlogs the other replicas need to connect without rebootstrap, besides last snapshot and xlogs after it.

Will we take that info from gc_consumers space?

1. Call vshard.router.map_callrw() with function ``box.backup.start()`. Make each shard backup as described in section for synchronous replicaset backup.

This is exactly what I described in the first comment, very dangerous.

Nits

• If standby cluster if is distant ...
• The latter case is when there is new term and new master this this ...

If it's not difficult for you, let's please numerate the parts of RFC, e.g.

## 1. Document aims
## 5. Single instance
### 5.1 Backup

It's not easy to parse that font differance between chapters and not very obvious, what subsection I'm reading.

[nshy]

Still, same comment as in the https://github.com/orgs/tarantool/discussions/12039. I don't think we should rely on tt team to figure this all out, if we decided to design such document. Since now it's not obvious, how the restore process will look like to the end user. Instead, we can invite them to review the document and think about the API together.

I guess cluster backup/restore will be done in a different product, like ATE or Tarantool DB. We here only to describe how to accomplish this tasks using Tarantool/vshard API, probably some new tt commands could help but they are not for end user too. tt commands that may be introduced will be quite simple, I guess we don't need to consider them in RFC.

I'd propose introducing force close of backup on exit from function call, as we do it for transactions. However, it may have performance impact, which I don't really like.

Done in new RFC which superseds this one.

This one is not backward compatible, the first argument to box.backup.start() is number already...

Discussed f2f that having a table or a number as first argument fits in Tarantool API.

Then the question, how will the tarantool remember, what files were in the last backup. Will this info survive instance restart? If so, where it'll be persisted.

Mentioned this point in new RFC (we discussed this point with @sergepetrenko, decided we don't need to add persistence right now given requirements).

Key-value and index-based in the same table looks way too crutchy. In terms of box.backup.start() we're limited with backward compatibility. I'd rather return only the list of files from box.backup.start() and everything else and also files from box.backup.info(), in which we can do whatever we want.

Already done in this RFC (and new RFC therefore) after similar comment from @locker.

And the last question about icremental backups. Why do we want it at all? Do we have any users, which request it? It's way too complicated to implement in tarantool such thing and to client application it costs nothing: he will just send a list of files he already has to the function, will start a new backup and skip the files, which are already persisted. That's it.

I'd think twice, do we want it in core) Moreover, as you already stated, we have problems with master changes.

In new RFC we do not track incremental backup base in Tarantool (however as discussed f2f it does not look to be complicated).

And why don't we want to always return that info and not introduce that replicaset mode? It doesn't differ from any other, AFACS. Only the return values is changed.

Done. Also replicaset mode is not mentioned in new RFC. I guess if we need this one then it is only a matter of adding info about gc consumers to box.backup.info().

Hmm, and it'll be considered successfull full backup and will affect incremental once after it? Even when user doesn't like it and didn't copy the files? That strange. Maybe we need smth like box.backup.abort then, but I'd still consider not implementing incremental backups.

box.backup.abort() is not required after we decided that client will track backup base.

Multi-master setup doesn't exist in productions, but the async cluster is obviously the most popular. And here's the question, does our clients take backups from all instances in every replicaset? I'm pretty sure, they don't.

Such setup was requested by Alexander Klenov.

I didn't understand, what is meant here and pretty sure tt team also won`t be able to parse. Could you elaborate?
Will we take that info from gc_consumers space?

Dropped this parts as there is no replicaset mode in the new RFC.