Skip to content
This repository was archived by the owner on Jul 10, 2025. It is now read-only.

Commit 25c4309

Browse files
committed
Adding modifications for merged proposal
1 parent 512619d commit 25c4309

File tree

1 file changed

+46
-3
lines changed

1 file changed

+46
-3
lines changed

rfcs/20200505-transactional-fs.md

Lines changed: 46 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ For a coarse granularity adding `StartTransaction` `EndTransaction` and `GetTran
6868
```cpp
6969
class Filesystem {
7070
// Transaction Token API extensions
71-
virtual Status GetTransactionTokenForFile(const string& file_name,TransactionToken* token) = 0
71+
virtual Status GetTransactionTokenForFile(const string& file_name,TransactionToken* token) = 0;
7272
virtual Status StartTransaction(const string& transaction_name, TransactionToken* token) = 0;
7373
virtual Status EndTransaction(TransactionToken* token) = 0;
7474

@@ -80,7 +80,7 @@ class Filesystem {
8080

8181
// Creating directories
8282
virtual Status CreateDir(const string& dirname, TransactionToken* token=nullptr) = 0;
83-
virtual Status RecursivelyCreateDir(const string& dirname), TransactionToken* token=nullptr;
83+
virtual Status RecursivelyCreateDir(const string& dirname, TransactionToken* token=nullptr);
8484

8585
// Deleting
8686
virtual Status DeleteFile(const string& fname, TransactionToken* token=nullptr) = 0;
@@ -695,6 +695,49 @@ Status MergeFiles(const string& fname, const string& dirname,
695695
- Added Alternatives and Changes during review sections
696696
- Added optional `DecodeTransactionToken` method
697697

698-
Seed this with open questions you require feedback on from the RFC process.
698+
## Final design proposal
699+
700+
During the review meeting it has been decided to merge wrapped filesystem approach and stateful token approach. Then the final proposal is as shown below. The `WrappedFileSystem` and `Filesystem` classes described above are to be extended with three new methods,
701+
702+
```cpp
703+
class WrappedFileSystem : public Filesystem {
704+
// Other methods are ommited for brevity
705+
virtual Status AddToTransaction(const string& uri,
706+
TransactionToken* token = nullptr) {
707+
return fs_->AddToTransaction(uri, (token ? token : token_));
708+
}
709+
710+
virtual Status GetTokenOrStartTransaction(const string& uri,
711+
TransactionToken*& token) {
712+
return fs_->GetTokenOrStartTransaction(uri, token);
713+
}
714+
715+
virtual Status DecodeTransaction(const TransactionToken* token = nullptr,
716+
string* decoded_string) {
717+
return fs_->DecodeTransaction((token ?: token : token_), decoded_string);
718+
}
719+
//...
720+
};
721+
```
722+
723+
Then the current C API's `TF_FilesystemOps` table is to be extended by 6 new function pointers.
724+
725+
```cpp
726+
struct TF_FilesystemOps{
727+
// Existing members are not modified
728+
// Transaction management
729+
void (*const StartTransaction)(TF_Filesystem*, TransactionToken**);
730+
void (*const EndTransaction)(TF_Filesystem*, TransactionToken*);
731+
void (*const AddToTransaction)(TF_Filesystem* fs, const char* file_name, TransactionToken** token);
732+
void (*const GetTransactionTokenForFile)(TF_Filesystem* fs, const char* file_name, TransactionToken** token);
733+
void (*const GetOrStartTransactionTokenForFile)(TF_Filesystem* fs, const char* file_name, TransactionToken** token);
734+
// Optional Transaction Debugging
735+
void (*const DecodeTransactionToken)(const TF_Filesystem*, const TransactionToken*, char**);
736+
};
737+
```
738+
739+
The new functions will be null pointers until respective plugins implement them. `ModularFilesystem` implementation will check whether a plugin implements the transactions and will ignore the transaction if it is not implemented, possibly after producing a log message, thus falling back to current transactionless state. Since these function pointers will be added after existing pointers, already compiled plugins will keep functioning and they can be gradually start supporting transactions. Any filesystem plugin that start supporting transaction will be used by the framework.
740+
741+
With these final modifications, there is no need to carry transaction tokens through different compilation units or ops in the graph. For example checkpointing logic can be implemented in an atomic-like way by simply modifying the save and merge ops to use same transaction using accessed file names and directories.
699742

700743
[filesystem_plugin]: https://github.com/tensorflow/community/blob/master/rfcs/20190506-filesystem-plugin-modular-tensorflow.md

0 commit comments

Comments
 (0)