Firestore: Improve efficiency of memory persistence when processing a large number of writes #6233

dconeybe · 2024-09-05T16:37:16Z

Replace ArrayList with ArrayDeque so that erasing the first element does not incur O(n) cost, as when erasing the entire ArrayList this inefficiency results in O(n^2) cost. Googlers see b/364354267 and cl/671065694 for details.

Ported from firebase/firebase-ios-sdk#13572

…e efficient erasing of first element

github-actions · 2024-09-05T16:37:29Z

Release note changes

The following release notes were modified. Please ensure they look correct.

Release Notes

firebase-firestore

### {{firestore}} version 25.1.1 {: #firestore_v25-1-1}

* {{changed}} Improve efficiency of memory persistence when processing a large number of writes. GitHub [#6233](//github.com/firebase/firebase-android-sdk/issues/6233){: .external}

#### {{firestore}} Kotlin extensions version 25.1.1 {: #firestore-ktx_v25-1-1}

The Kotlin extensions library transitively includes the updated
`firebase-firestore` library. The Kotlin extensions library has no additional
updates.

google-oss-bot · 2024-09-05T16:41:39Z

Coverage Report ¹

Affected Products

firebase-firestore
Overall coverage changed from 46.05% (2eb7cd7) to 46.03% (78cec76) by -0.02%.

Test Logs

https://storage.googleapis.com/firebase-sdk-metric-reports/Lap00ZT1VI.html

github-actions · 2024-09-05T16:42:58Z

Unit Test Results

  186 files +  152   186 suites +152 4m 47s ⏱️ + 3m 49s
1 234 tests +1 046 1 218 ✔️ +1 030 16 💤 +16 0 ❌ ±0
2 492 runs +2 116 2 460 ✔️ +2 084 32 💤 +32 0 ❌ ±0

Results for commit b9784b9. ± Comparison against base commit 2eb7cd7.

This pull request removes 188 and adds 1234 tests. Note that renamed tests count towards both.

com.google.firebase.appdistribution.impl.AabUpdaterTest ‑ updateAppTask_emptyLocationHeader_setsDownloadFailure
com.google.firebase.appdistribution.impl.AabUpdaterTest ‑ updateAppTask_isNotRedirectResponse_setsDownloadFailure
com.google.firebase.appdistribution.impl.AabUpdaterTest ‑ updateAppTask_missingLocationHeader_setsDownloadFailure
com.google.firebase.appdistribution.impl.AabUpdaterTest ‑ updateAppTask_onAppResume_setsUpdateCancelled
com.google.firebase.appdistribution.impl.AabUpdaterTest ‑ updateAppTask_whenAabReleaseAvailable_redirectsToPlay
com.google.firebase.appdistribution.impl.AabUpdaterTest ‑ updateAppTask_whenOpenConnectionFails_setsNetworkFailure
com.google.firebase.appdistribution.impl.AabUpdaterTest ‑ updateApp_whenCalledMultipleTimesWithAAB_onlyMakesOneRequest
com.google.firebase.appdistribution.impl.ApkInstallerTests ‑ installActivityDestroyed_setsInstallError
com.google.firebase.appdistribution.impl.ApkInstallerTests ‑ installApk_currentActivityNotNull_InstallIntentOnCurrentActivity
com.google.firebase.appdistribution.impl.ApkInstallerTests ‑ whenCalledMultipleTimes_onlyEmitsOneIntent
…

com.google.firebase.firestore.AggregateQuerySnapshotTest ‑ createWithCountShouldReturnInstanceWithTheGivenQueryAndCount
com.google.firebase.firestore.AggregateQueryTest ‑ testSourceMustNotBeNull
com.google.firebase.firestore.BlobTest ‑ testComparison
com.google.firebase.firestore.BlobTest ‑ testEquals
com.google.firebase.firestore.BlobTest ‑ testMutableBytes
com.google.firebase.firestore.CollectionReferenceTest ‑ testEquals
com.google.firebase.firestore.DocumentChangeTest ‑ randomTests
com.google.firebase.firestore.DocumentChangeTest ‑ testAdditions
com.google.firebase.firestore.DocumentChangeTest ‑ testChangesWithSortOrderChange
com.google.firebase.firestore.DocumentChangeTest ‑ testDeletions
…

♻️ This comment has been updated with latest results.

google-oss-bot · 2024-09-05T20:02:25Z

Size Report ¹

Affected Products

firebase-firestore
Type Base (2eb7cd7) Merge (78cec76) Diff
aar 1.44 MB 1.44 MB +17 B (+0.0%)
apk (release) 11.3 MB 11.3 MB +52 B (+0.0%)

Test Logs

https://storage.googleapis.com/firebase-sdk-metric-reports/K6LkZHA6T1.html

google-oss-bot · 2024-09-05T20:17:21Z

Startup Time Report ¹

Note: Layout is sometimes suboptimal due to limited formatting support on GitHub. Please check this report on GCS.

Notes

This report is for comparing the base commit (2eb7cd7) and the CI merge commit (78cec76)
Please check below reports for each individual commit to find more details (Perfetto traces, histograms, detailed measurements)
- 2eb7cd7: https://storage.googleapis.com/firebase-sdk-metric-reports/LCsYH3Bm26/index.html
- 78cec76: https://storage.googleapis.com/firebase-sdk-metric-reports/llTZU5UBd5/index.html

Startup Times

`fire-fst`

Device Statistics Distributions

oriole-32

Percentile	`2eb7cd7`	`78cec76`	Diff	Significant (?)
p10	324 ±32 μs	314 ±15 μs	-9.78 μs (-3.0%)	NO
p25	342 ±59 μs	322 ±16 μs	-19.3 μs (-5.6%)	NO
p50	365 ±86 μs	335 ±18 μs	-29.7 μs (-8.1%)	NO
p75	402 ±117 μs	365 ±41 μs	-37.5 μs (-9.3%)	NO
p90	459 ±196 μs	434 ±86 μs	-24.9 μs (-5.4%)	NO

20 test runs in comparison

Commit	Test Runs
`2eb7cd7`	2024-09-05_14:43:18.113963_ZYPh 2024-09-05_14:43:18.114016_bvUl 2024-09-05_14:43:18.114028_gZli 2024-09-05_14:43:18.114037_jaqw 2024-09-05_14:43:18.114045_JJgj 2024-09-05_14:43:18.114053_GhUl 2024-09-05_14:43:18.114061_jxsd 2024-09-05_14:43:18.114073_HJeq 2024-09-05_14:43:18.114084_FpAF 2024-09-05_14:43:18.114092_xCVW
`78cec76`	2024-09-05_20:03:54.231529_VCaW 2024-09-05_20:03:54.231574_WldU 2024-09-05_20:03:54.231585_ocle 2024-09-05_20:03:54.231594_eRnz 2024-09-05_20:03:54.231598_GhMI 2024-09-05_20:03:54.231603_kPce 2024-09-05_20:03:54.231606_Deyr 2024-09-05_20:03:54.231612_MGwe 2024-09-05_20:03:54.231620_bSRl 2024-09-05_20:03:54.231636_RFlj

redfin-30

Percentile	`2eb7cd7`	`78cec76`	Diff	Significant (?)
p10	611 ±48 μs	603 ±43 μs	-7.81 μs (-1.3%)	NO
p25	632 ±56 μs	623 ±50 μs	-8.67 μs (-1.4%)	NO
p50	654 ±70 μs	651 ±65 μs	-3.29 μs (-0.5%)	NO
p75	692 ±90 μs	682 ±78 μs	-9.26 μs (-1.3%)	NO
p90	739 ±108 μs	732 ±111 μs	-6.91 μs (-0.9%)	NO

20 test runs in comparison

Commit	Test Runs
`2eb7cd7`	2024-09-05_14:43:18.113963_ZYPh 2024-09-05_14:43:18.114016_bvUl 2024-09-05_14:43:18.114028_gZli 2024-09-05_14:43:18.114037_jaqw 2024-09-05_14:43:18.114045_JJgj 2024-09-05_14:43:18.114053_GhUl 2024-09-05_14:43:18.114061_jxsd 2024-09-05_14:43:18.114073_HJeq 2024-09-05_14:43:18.114084_FpAF 2024-09-05_14:43:18.114092_xCVW
`78cec76`	2024-09-05_20:03:54.231529_VCaW 2024-09-05_20:03:54.231574_WldU 2024-09-05_20:03:54.231585_ocle 2024-09-05_20:03:54.231594_eRnz 2024-09-05_20:03:54.231598_GhMI 2024-09-05_20:03:54.231603_kPce 2024-09-05_20:03:54.231606_Deyr 2024-09-05_20:03:54.231612_MGwe 2024-09-05_20:03:54.231620_bSRl 2024-09-05_20:03:54.231636_RFlj

`timeToInitialDisplay`

Device Statistics Distributions

oriole-32

Percentile	`2eb7cd7`	`78cec76`	Diff	Significant (?)
p10	202 ±9 ms	206 ±5 ms	+4.07 ms (+2.0%)	NO
p25	210 ±11 ms	212 ±5 ms	+1.82 ms (+0.9%)	NO
p50	220 ±20 ms	219 ±5 ms	-1.10 ms (-0.5%)	NO
p75	229 ±25 ms	228 ±5 ms	-1.60 ms (-0.7%)	NO
p90	240 ±31 ms	242 ±7 ms	+2.04 ms (+0.8%)	NO

20 test runs in comparison

Commit	Test Runs
`2eb7cd7`	2024-09-05_14:43:18.113963_ZYPh 2024-09-05_14:43:18.114016_bvUl 2024-09-05_14:43:18.114028_gZli 2024-09-05_14:43:18.114037_jaqw 2024-09-05_14:43:18.114045_JJgj 2024-09-05_14:43:18.114053_GhUl 2024-09-05_14:43:18.114061_jxsd 2024-09-05_14:43:18.114073_HJeq 2024-09-05_14:43:18.114084_FpAF 2024-09-05_14:43:18.114092_xCVW
`78cec76`	2024-09-05_20:03:54.231529_VCaW 2024-09-05_20:03:54.231574_WldU 2024-09-05_20:03:54.231585_ocle 2024-09-05_20:03:54.231594_eRnz 2024-09-05_20:03:54.231598_GhMI 2024-09-05_20:03:54.231603_kPce 2024-09-05_20:03:54.231606_Deyr 2024-09-05_20:03:54.231612_MGwe 2024-09-05_20:03:54.231620_bSRl 2024-09-05_20:03:54.231636_RFlj

redfin-30

Percentile	`2eb7cd7`	`78cec76`	Diff	Significant (?)
p10	251 ±14 ms	268 ±4 ms	+17.1 ms (+6.8%)	NO
p25	256 ±15 ms	273 ±4 ms	+16.9 ms (+6.6%)	NO
p50	263 ±16 ms	281 ±5 ms	+17.5 ms (+6.6%)	NO
p75	278 ±33 ms	290 ±7 ms	+12.1 ms (+4.4%)	NO
p90	292 ±37 ms	304 ±12 ms	+11.4 ms (+3.9%)	NO

20 test runs in comparison

Commit	Test Runs
`2eb7cd7`	2024-09-05_14:43:18.113963_ZYPh 2024-09-05_14:43:18.114016_bvUl 2024-09-05_14:43:18.114028_gZli 2024-09-05_14:43:18.114037_jaqw 2024-09-05_14:43:18.114045_JJgj 2024-09-05_14:43:18.114053_GhUl 2024-09-05_14:43:18.114061_jxsd 2024-09-05_14:43:18.114073_HJeq 2024-09-05_14:43:18.114084_FpAF 2024-09-05_14:43:18.114092_xCVW
`78cec76`	2024-09-05_20:03:54.231529_VCaW 2024-09-05_20:03:54.231574_WldU 2024-09-05_20:03:54.231585_ocle 2024-09-05_20:03:54.231594_eRnz 2024-09-05_20:03:54.231598_GhMI 2024-09-05_20:03:54.231603_kPce 2024-09-05_20:03:54.231606_Deyr 2024-09-05_20:03:54.231612_MGwe 2024-09-05_20:03:54.231620_bSRl 2024-09-05_20:03:54.231636_RFlj

https://storage.googleapis.com/firebase-sdk-metric-reports/wraM7lNnoV/index.html

dconeybe · 2024-09-06T20:51:24Z

Closing PR as this work has been shelved and tracked by b/365092047

dconeybe added 2 commits September 5, 2024 12:33

MemoryMutationQueue.java: use ArrayDeque instead of ArrayList for mor…

ef27e83

…e efficient erasing of first element

CHANGELOG.md entry added (NOTE: update PR number once PR is created)

a91ed1a

dconeybe added api: firestore main-merge-ack labels Sep 5, 2024

dconeybe self-assigned this Sep 5, 2024

dconeybe mentioned this pull request Sep 5, 2024

Firestore: Improve efficiency of memory persistence when processing a large number of writes firebase/firebase-ios-sdk#13572

Merged

google-oss-bot added the size/S label Sep 5, 2024

CHANGELOG.md: update PR number

645efb7

Start switch to TreeMap

c636193

google-oss-bot added size/M and removed size/S labels Sep 5, 2024

Finish TreeMap implementation

edff85b

google-oss-bot added size/L and removed size/M labels Sep 5, 2024

cleanup

b9784b9

dconeybe closed this Sep 6, 2024

dconeybe deleted the dconeybe/MemoryMutationQueueDeque branch September 6, 2024 20:51

firebase locked and limited conversation to collaborators Oct 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Firestore: Improve efficiency of memory persistence when processing a large number of writes #6233

Firestore: Improve efficiency of memory persistence when processing a large number of writes #6233

Uh oh!

dconeybe commented Sep 5, 2024

Uh oh!

github-actions bot commented Sep 5, 2024 •

edited

Loading

Uh oh!

google-oss-bot commented Sep 5, 2024 •

edited

Loading

`firebase-firestore`

Uh oh!

github-actions bot commented Sep 5, 2024 •

edited

Loading

Uh oh!

google-oss-bot commented Sep 5, 2024

`firebase-firestore`

Uh oh!

google-oss-bot commented Sep 5, 2024

`fire-fst`

`timeToInitialDisplay`

Uh oh!

dconeybe commented Sep 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Firestore: Improve efficiency of memory persistence when processing a large number of writes #6233

Firestore: Improve efficiency of memory persistence when processing a large number of writes #6233

Uh oh!

Conversation

dconeybe commented Sep 5, 2024

Uh oh!

github-actions bot commented Sep 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release note changes

Uh oh!

google-oss-bot commented Sep 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report 1

Affected Products

firebase-firestore

Test Logs

Uh oh!

github-actions bot commented Sep 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unit Test Results

Uh oh!

google-oss-bot commented Sep 5, 2024

Size Report 1

Affected Products

firebase-firestore

Test Logs

Uh oh!

google-oss-bot commented Sep 5, 2024

Startup Time Report 1

Notes

Startup Times

fire-fst

timeToInitialDisplay

Uh oh!

dconeybe commented Sep 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions bot commented Sep 5, 2024 •

edited

Loading

google-oss-bot commented Sep 5, 2024 •

edited

Loading

Coverage Report ¹

`firebase-firestore`

github-actions bot commented Sep 5, 2024 •

edited

Loading

Size Report ¹

`firebase-firestore`

Startup Time Report ¹

`fire-fst`

`timeToInitialDisplay`