Merge branch 'next-major' into feature/string-compression

Author: jedelbo

CHANGELOG.md

@@ -21,6 +21,34 @@
 
 ----------------------------------------------
 
+# 14.10.3 Release notes
+
+### Enhancements
+* "Next launch" metadata file actions are now performed in a multi-process safe manner ([#7576](https://github.com/realm/realm-core/pull/7576)).
+
+### Fixed
+* Fixed a change of mode from Strong to All when removing links from an embedded object that links to a tombstone. This affects sync apps that use embedded objects which have a `Lst<Mixed>` that contains a link to another top level object which has been deleted by another sync client (creating a tombstone locally). In this particular case, the switch would cause any remaining link removals to recursively delete the destination object if there were no other links to it. ([#7828](https://github.com/realm/realm-core/issues/7828), since 14.0.0-beta.0)
+* Fixed removing backlinks from the wrong objects if the link came from a nested list, nested dictionary, top-level dictionary, or list of mixed, and the source table had more than 256 objects. This could manifest as `array_backlink.cpp:112: Assertion failed: int64_t(value >> 1) == key.value` when removing an object. ([#7594](https://github.com/realm/realm-core/issues/7594), since v11 for dictionaries)
+* Fixed the collapse/rejoin of clusters which contained nested collections with links. This could manifest as `array.cpp:319: Array::move() Assertion failed: begin <= end [2, 1]` when removing an object. ([#7839](https://github.com/realm/realm-core/issues/7839), since the introduction of nested collections in v14.0.0-beta.0)
+* wait_for_upload_completion() was inconsistent in how it handled commits which did not produce any changesets to upload. Previously it would sometimes complete immediately if all commits waiting to be uploaded were empty, and at other times it would wait for a server roundtrip. It will now always complete immediately. ([PR #7796](https://github.com/realm/realm-core/pull/7796)).
+* `realm_sync_session_handle_error_for_testing` parameter `is_fatal` was flipped changing the expected behavior. (#[7750](https://github.com/realm/realm-core/issues/7750)).
+
+### Breaking changes
+* None.
+
+### Compatibility
+* Fileformat: Generates files with format v24. Reads and automatically upgrade from fileformat v10. If you want to upgrade from an earlier file format version you will have to use RealmCore v13.x.y or earlier.
+
+-----------
+
+### Internals
+* Fixed `Table::remove_object_recursive` which wouldn't recursively follow links through a single `Mixed` property. This feature is exposed publicly on `Table` but no SDK currently uses it, so this is considered internal. ([#7829](https://github.com/realm/realm-core/issues/7829), likely since the introduction of Mixed)
+* Upload completion is now tracked in a multiprocess-compatible manner ([PR #7796](https://github.com/realm/realm-core/pull/7796)).
+* The local realm will assume the the client file ident of the fresh realm during a client reset. ([PR #7850](https://github.com/realm/realm-core/pull/7850))
+* Building using C++20 on Windows.
+
+----------------------------------------------
+
 # 14.10.2 Release notes
 
 ### Enhancements
@@ -517,7 +545,7 @@
 * Fixed equality queries on a Mixed property with an index possibly returning the wrong result if values of different types happened to have the same StringIndex hash. ([6407](https://github.com/realm/realm-core/issues/6407) since v11.0.0-beta.5).
 * If you have more than 8388606 links pointing to one specific object, the program will crash. ([#6577](https://github.com/realm/realm-core/issues/6577), since v6.0.0)
 * Query for NULL value in Dictionary<Mixed> would give wrong results ([6748])(https://github.com/realm/realm-core/issues/6748), since v10.0.0)
-* A Realm generated on a non-apple ARM 64 device and copied to another platform (and vice-versa) were non-portable due to a sorting order difference. This impacts strings or binaries that have their first difference at a non-ascii character. These items may not be found in a set, or in an indexed column if the strings had a long common prefix (> 200 characters). ([PR #6670](https://github.com/realm/realm-core/pull/6670), since 2.0.0-rc7 for indexes, and since since the introduction of sets in v10.2.0)
+* A Realm generated on a non-apple ARM 64 device and copied to another platform (and vice-versa) were non-portable due to a sorting order difference. This impacts strings or binaries that have their first difference at a non-ascii character. These items may not be found in a set, or in an indexed column if the strings had a long common prefix (> 200 characters). ([PR #6670](https://github.com/realm/realm-core/pull/6670), since 2.0.0-rc7 for indexes, and since the introduction of sets in v10.2.0)
 
 ### Breaking changes
 * Support for upgrading from Realm files produced by RealmCore v5.23.9 or earlier is no longer supported.

CMakeLists.txt

@@ -129,6 +129,9 @@ if(MSVC)
     # We use these in our AtomicSharedPtr implementation and it can't move
     # to atomic<shared_ptr<T>> because NotificationToken relies on movability.
     add_compile_options(/D_SILENCE_CXX20_OLD_SHARED_PTR_ATOMIC_SUPPORT_DEPRECATION_WARNING)
+
+    # Enable __cplusplus macro
+    add_compile_options(/Zc:__cplusplus)
 else()
     add_compile_options(-Wall -Wextra -Wempty-body -Wparentheses -Wunknown-pragmas -Wunreachable-code -Wunused-parameter -Wno-missing-field-initializers)
     # TODO: Remove this when fixed

Package.swift

@@ -3,7 +3,7 @@
 import PackageDescription
 import Foundation
 
-let versionStr = "14.10.2"
+let versionStr = "14.10.3"
 let versionPieces = versionStr.split(separator: "-")
 let versionCompontents = versionPieces[0].split(separator: ".")
 let versionExtra = versionPieces.count > 1 ? versionPieces[1] : ""

dependencies.yml

@@ -1,5 +1,5 @@
 PACKAGE_NAME: realm-core
-VERSION: 14.10.2
+VERSION: 14.10.3
 OPENSSL_VERSION: 3.2.0
 ZLIB_VERSION: 1.2.13
 # https://github.com/10gen/baas/commits

evergreen/config.yml

@@ -438,7 +438,7 @@ functions:
       script: |-
         if ls crash-*> /dev/null 2>&1; then
           echo "Found crash file"
-          #Rename the crash file and the realm file. 
+          #Rename the crash file and the realm file.
           #If there is a crash, this will signal that something needs to be uploaded.
           mv crash-* realm-fuzzer-crash.txt
           mv fuzz-realm.realm fuzzer_realm.realm
@@ -467,7 +467,7 @@ functions:
       bucket: mciuploads
       permissions: public-read
       content_type: application/x-binary
-      display_name: Realm File 
+      display_name: Realm File
       optional: true
 
   - command: shell.exec
@@ -523,7 +523,7 @@ functions:
         set -o verbose
 
         TOP_DIR=$(pwd)
-        
+
         HANG_ANALYZER_PATH=$TOP_DIR/evergreen/hang_analyzer
         REQUIREMENTS_PATH=$HANG_ANALYZER_PATH/requirements.txt
 
@@ -1174,7 +1174,7 @@ tasks:
             ./build/test/${cmake_build_type|Debug}-iphonesimulator/realm-combined-tests.app \
             'io.realm.CombinedTests' \
             $TEST_RESULTS_FILE
- 
+
 - name: generate-sync-corpus
   tags: [ "for_nightly_tests" ]
   exec_timeout_secs: 1800
@@ -1401,6 +1401,19 @@ buildvariants:
   - name: compile_test
   - name: lint
 
+- name: ubuntu-cpp20
+  display_name: "Ubuntu C++20"
+  run_on: ubuntu2404-arm64-large
+  expansions:
+    fetch_missing_dependencies: On
+    c_compiler: "/opt/clang+llvm/bin/clang"
+    cxx_compiler: "/opt/clang+llvm/bin/clang++"
+    clang_format: "/opt/clang+llvm/bin/clang-format"
+    extra_flags: -DCMAKE_CXX_STANDARD=20
+  tasks:
+  - name: compile_test
+  - name: lint
+
 - name: ubuntu-no-session-multiplexing
   display_name: "Ubuntu (Sync Multiplexing Disabled)"
   run_on: ubuntu2204-arm64-large
@@ -1477,7 +1490,7 @@ buildvariants:
     fetch_missing_dependencies: On
     c_compiler: "/opt/clang+llvm/bin/clang"
     cxx_compiler: "/opt/clang+llvm/bin/clang++"
-    run_with_encryption: 1 
+    run_with_encryption: 1
     enable_asan: On
     cmake_build_type: RelWithDebInfo
   tasks:
@@ -1500,7 +1513,7 @@ buildvariants:
     cxx_compiler: "./clang_binaries/bin/clang++"
     python3: /opt/mongodbtoolchain/v3/bin/python3
   tasks:
-  - name: compile_test_and_package 
+  - name: compile_test_and_package
   - name: benchmarks
   - name: generate-sync-corpus
 
@@ -2004,5 +2017,3 @@ buildvariants:
 #     no_tests: On
 #   tasks:
 #   - name: compile_only
-
-

src/external/bson/bson-decimal128.c

@@ -374,7 +374,7 @@ _mul_64x64 (uint64_t left,                 /* IN */
  * Returns:
  *    The lowercased character.
  */
-char
+static char
 _dec128_tolower (char c)
 {
    if (isupper (c)) {
@@ -395,7 +395,7 @@ _dec128_tolower (char c)
  * Returns:
  *    true if the strings are equal, false otherwise.
  */
-bool
+static bool
 _dec128_istreq (const char *a, /* IN */
                 const char *b /* IN */)
 {

src/external/bson/bson-iter.c

@@ -631,7 +631,7 @@ _bson_iter_next_internal (bson_iter_t *iter,    /* INOUT */
       subtype = *(iter->raw + iter->d2);
 
       if (subtype == BSON_SUBTYPE_BINARY_DEPRECATED) {
-         int32_t binary_len;
+         uint32_t binary_len;
 
          if (l < 4) {
             iter->err_off = o;

src/realm/object-store/sync/sync_session.cpp

@@ -912,8 +912,7 @@ void SyncSession::create_sync_session()
     session_config.proxy_config = sync_config.proxy_config;
     session_config.simulate_integration_error = sync_config.simulate_integration_error;
     session_config.flx_bootstrap_batch_size_bytes = sync_config.flx_bootstrap_batch_size_bytes;
-    session_config.session_reason =
-        client_reset::is_fresh_path(m_config.path) ? sync::SessionReason::ClientReset : sync::SessionReason::Sync;
+    session_config.fresh_realm_download = client_reset::is_fresh_path(m_config.path);
     session_config.schema_version = m_config.schema_version;
 
     if (sync_config.on_sync_client_event_hook) {

src/realm/sync/client.cpp

@@ -169,6 +169,10 @@ class SessionWrapper final : public util::AtomicRefCountBase, DB::CommitListener
 
     const SessionReason m_session_reason;
 
+    // If false, QUERY and MARK messages are allowed but UPLOAD messages will not
+    // be sent to the server.
+    const bool m_allow_upload_messages;
+
     const uint64_t m_schema_version;
 
     std::shared_ptr<SubscriptionStore> m_flx_subscription_store;
@@ -716,6 +720,13 @@ uint64_t SessionImpl::get_schema_version() noexcept
     return m_wrapper.m_schema_version;
 }
 
+bool SessionImpl::upload_messages_allowed() noexcept
+{
+    // Can only be called if the session is active or being activated
+    REALM_ASSERT_EX(m_state == State::Active || m_state == State::Unactivated, m_state);
+    return m_wrapper.m_allow_upload_messages;
+}
+
 void SessionImpl::initiate_integrate_changesets(std::uint_fast64_t downloadable_bytes, DownloadBatchState batch_state,
                                                 const SyncProgress& progress, const ReceivedChangesets& changesets)
 {
@@ -1150,7 +1161,9 @@ SessionWrapper::SessionWrapper(ClientImpl& client, DBRef db, std::shared_ptr<Sub
     , m_progress_handler(std::move(config.progress_handler))
     , m_connection_state_change_listener(std::move(config.connection_state_change_listener))
     , m_debug_hook(std::move(config.on_sync_client_event_hook))
-    , m_session_reason(m_client_reset_config ? SessionReason::ClientReset : config.session_reason)
+    , m_session_reason(m_client_reset_config || config.fresh_realm_download ? SessionReason::ClientReset
+                                                                            : SessionReason::Sync)
+    , m_allow_upload_messages(!config.fresh_realm_download)
     , m_schema_version(config.schema_version)
     , m_flx_subscription_store(std::move(flx_sub_store))
     , m_migration_store(std::move(migration_store))

src/realm/sync/client.hpp

@@ -420,9 +420,9 @@ class Session {
         /// through Client::run().
         util::UniqueFunction<ConnectionStateChangeListener> connection_state_change_listener;
 
-        /// The purpose of this sync session. Reported to the server for informational purposes and has no functional
-        /// effect.
-        SessionReason session_reason = SessionReason::Sync;
+        /// Is this session being opened for a realm whose path ends in ".fresh"? If so,
+        /// it will be downloading a fresh copy of the realm data from the server.
+        bool fresh_realm_download = false;
 
         /// Schema version
         ///