Skip to content

Comments

fix: precaching of all transactions causes errors#766

Open
NickUfer wants to merge 13 commits intomatrix-org:developfrom
NickUfer:tx_registeredusers_cache
Open

fix: precaching of all transactions causes errors#766
NickUfer wants to merge 13 commits intomatrix-org:developfrom
NickUfer:tx_registeredusers_cache

Conversation

@NickUfer
Copy link

@NickUfer NickUfer commented Jan 3, 2022

This PR changes the storage implementation to only load users and transaction ids when they are needed. Recent queries for txIds and users are cached. This fix also reduces memory consumption.

Fixes #765

@verymilan
Copy link
Contributor

Kinda sad to see that this is still marked as draft and has no responses so far...

@NickUfer NickUfer force-pushed the tx_registeredusers_cache branch from f06b46f to a17907f Compare November 16, 2022 17:44
@NickUfer
Copy link
Author

Tested it, works fine

Nov-16 21:37:58.622 [DiscordStore] verbose: User registration status added to registeredUsersCache. 
                    userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
                    
[...]

Nov-16 21:38:27.140 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache. 
                        userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:27.640 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache. 
                        userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:27.698 [DiscordBot] verbose: i:redacted c:redacted g:redacted
Nov-16 21:38:27.699 [UserSync] verbose: State update requested for 000000000000000000
Nov-16 21:38:27.701 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache. 
                        userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:27.706 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache. 
                        userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:28.140 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache. 
                        userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:28.640 [DiscordStore] verbose: User registration status retrieved from registeredUsersCache. 
                        userId: '@_discord_000000000000000000:matrix.redacted.invalid' ; registered: true
Nov-16 21:38:28.764 [DiscordStore] verbose: Txn completion status added to txnsCompletionStatusCache. 
                    txnId: '51' ; completed: false
Nov-16 21:38:28.765 [bot-sdkAppservice] info: [ 'Processing transaction 51' ]
Nov-16 21:38:28.766 [bot-sdkAppservice] info: [ 'Processing event of type m.room.message' ]
Nov-16 21:38:28.768 [DiscordStore] verbose: Txn completion status added to txnsCompletionStatusCache. 
                            txnId: '51' ; completed: true

Lets see what the CI says, otherwise should be ready to review

@NickUfer NickUfer marked this pull request as ready for review November 16, 2022 21:44
@NickUfer NickUfer requested a review from a team as a code owner November 16, 2022 21:44
@NickUfer
Copy link
Author

Added log.silly for cache pruning, works too

Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='54' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='55' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='56' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='57' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='58' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='59' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='60' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='61' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='62' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='63' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='64' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.713 [Timedcache] silly: removed entry with key='65' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='66' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='67' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='68' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='69' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='70' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='71' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='72' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='73' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='74' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='75' from cache 'txnsCompletionStatusCache'
Nov-16 22:07:10.714 [Timedcache] silly: removed entry with key='@_discord_bot:matrix.redacted.invalid' from cache 'registeredUsersCache'

test: fixes tests to close DiscordStorage after usage
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

discord -> matrix no longer working, unclear error message

2 participants