Merge pull request #7436 from realm/support-new-datatypes

Support new datatypes:
- UUID
- RealmAny
- RealmMap
- RealmSet

Author: cmelchior

CHANGELOG.md

@@ -1,3 +1,71 @@
+## 10.6.0-BETA.2 (2021-06-14)
+
+### Breaking Changes
+* ``MapChangeSet.getDeletionsCount()` has been replaced with `MapChangeSet.getDeletions()` that return the keys for entries that has been deleted instead of just the number of deleted entries.
+* Primary keys now have automatic indexes again. Indexes was removed in v10.0.0 because they were not needed, but it caused issues when upgrading from a pre v10 version of Realm, and in some cases resulted in large delays when upgrading the fileformat. (Issue [#7426](https://github.com/realm/realm-java/issues/7426), since 10.0.0).
+
+### Enhancements
+* Allow `insert` and `insertOrUpdate` operations on `RealmObject` or `RealmObject` collections containing `RealmDictionary` or `RealmSet` fields.
+* Added support for `RealmDictionary` in `DynamicRealmObject` with `setDictionary(String fieldName, RealmDictionary<?> dictionary)`, `getDictionary(String fieldName, Class<?> primitiveType)`, and `getDictionary(String fieldName)`.
+* Added support for `RealmSet` in `DynamicRealmObject` with `setRealmSet(String fieldName, RealmSet<?> realmSet)`, `getRealmSet(String fieldName, Class<?> primitiveType)`, and `getRealmSet(String fieldName)`.
+
+### Fixed
+* Removed wrong `@Nullable` annotation on `RealmQuery.maxRealmAny()`.
+* Fixed `RealmAny.getValueClass()` returning the `RealmObject` proxy class instead of the model class on a `RealmAny` referencing a managed `RealmObject`.
+
+### Compatibility
+* File format: Generates Realms with format v22. Unsynced Realms will be upgraded from Realm Java 2.0 and later. Synced Realms can only be read and upgraded if created with Realm Java v10.0.0-BETA.1.
+* APIs are backwards compatible with all previous release of realm-java in the 10.6.y series.
+* Realm Studio 11.0.0-alpha.0 or above is required to open Realms created by this version.
+
+### Internal
+* Updated to Realm Core 11.0.2, commit a30382469eb72c0cf1824b44e7062071c2f3f3a9.
+* Updated to Gradle 6.8.3.
+
+
+## 10.6.0-BETA.1 (2021-05-17)
+
+### Breaking Changes
+* [RealmApp] Sync protocol version increased to 3. This version adds support for the new data types introduced in file format version 21.
+* File format version bumped to 21. In this version we support new basic datatypes `UUID` and `RealmAny`, as well as `RealmSet` and `RealmMap` collections with string-based keys (i.e. `RealmDictionary`).
+* Queries no longer do nullability checks on non-nullable fields, so using `null` as an argument will not throw an `IllegalArgumentException`.
+* String query filters `contains`, `beginsWith`, `endsWith`, and `like`, now throw a null pointer exception on null values.
+* The query builder no longer throw `IllegalStateException` but `IllegalArgumentException`.
+* The `distinct` query filter on unsupported fields no longer throws an exception when applied through when querying across relationships.
+* The `distinct` query filter no longer throws an exception when applied on non-existent fields.
+
+### Enhancements
+* Added support for `java.util.UUID` as supported field in model classes.
+* Added support for `java.util.UUID` as a primary key.
+* Added support for `RealmAny` as supported field in model classes. A `RealmAny` is used to represent a polymorphic Realm value or Realm Object, is indexable but cannot be used as a primary key. See [Javadoc for RealmAny](https://docs.mongodb.com/realm-sdks/java/latest/io/realm/RealmAny.html).
+* Added support for `RealmDictionary` as supported field in model classes. A `RealmDictionary` is a `Map` of strings to values - all types under the `RealmAny` umbrella can be used as values. See [Javadoc for RealmDictionary](https://docs.mongodb.com/realm-sdks/java/latest/io/realm/RealmDictionary.html) and [Javadoc for RealmMap](https://docs.mongodb.com/realm-sdks/java/latest/io/realm/RealmMap.html). `RealmDictionary` is not yet supported by any of the `Realm.insert` and `Realm.createFromJson` methods - This support will be added in a future release.
+* Added support for `RealmSet` as supported field in model classes. A `RealmSet` is a collection that implements the Java `Set` interface and contains no duplicate values - all types under the `RealmAny` umbrella can be used as values. See [Javadoc for RealmSet](https://docs.mongodb.com/realm-sdks/java/latest/io/realm/RealmSet.html). `RealmSet` is not yet supported by any of the `Realm.insert` and `Realm.createFromJson` methods - This support will be added in a future release.
+* Allow UTF8 encoded characters in property names in string-based queries ([#4467](https://github.com/realm/realm-core/issues/4467))
+* The error message when the initial steps of opening a Realm file fails is now more descriptive.
+* Make conversion of Decimal128 to/from string work for numbers with more than 19 significant digits. ([#4548](https://github.com/realm/realm-core/issues/4548))
+* Remove type coercion on bool and ObjectId when doing queries.
+* Allow passing arguments into string-based query predicates.
+* Queries across relationships now support the `between` operator.
+* Queries on numerical fields (byte, short, int, long, float, double, decimal128) now accept any numerical value as an argument.
+* `isEmpty` query filter can now be applied on `RealmList` and `RealmObject` fields.
+
+### Fixed
+* Fix assertion failures such as "!m_notifier_skip_version.version" or "m_notifier_sg->get_version() + 1 == new_version.version" when performing writes inside change notification callbacks. Previously refreshing the Realm by beginning a write transaction would skip delivering notifications, leaving things in an inconsistent state. Notifications are now delivered recursively when needed instead. ([Cocoa #7165](https://github.com/realm/realm-cocoa/issues/7165)).
+* Fixed name aliasing not working in sort/distinct clauses when doing string-based queries. ([#4550](https://github.com/realm/realm-core/issues/4550), never before working).
+* Potential/unconfirmed fix for crashes associated with failure to memory map (low on memory, low on virtual address space). For example ([#4514](https://github.com/realm/realm-core/issues/4514)).
+* Syncing large Decimal128 values will cause "Assertion failed: cx.w[1] == 0" ([#4519](https://github.com/realm/realm-core/issues/4519), since v10.0.0)
+* Classes names "class_class_..." were not handled correctly when doing queries ([#4480](https://github.com/realm/realm-core/issues/4480))
+* Fix collection notification reporting for modifications. This could be observed by receiving the wrong indices of modifications on sorted or distinct results, or notification blocks sometimes not being called when only modifications have occurred. ([#4573](https://github.com/realm/realm-core/pull/4573) since v6).
+
+### Compatibility
+* File format: Generates Realms with format v21. Unsynced Realms will be upgraded from Realm Java 2.0 and later. Synced Realms can only be read and upgraded if created with Realm Java v10.0.0-BETA.1.
+* APIs are backwards compatible with all previous release of realm-java in the 10.6.y series.
+* Realm Studio 11.0.0-alpha.0 or above is required to open Realms created by this version.
+
+### Internal
+* Updated to Realm Core 11.0.0-beta.4, commit: d50aef63a8aaf435e3afed82b589b47d8e1ab1ab.
+
+
 ## 10.5.0 (2021-05-07)
 
 ### Breaking Changes
@@ -26,7 +94,7 @@
 
 ## 10.4.0 (2021-03-26)
 
-All releases from 10.4.0 and forward are now found on `mavenCentral()` instead of `jcenter()`. 
+All releases from 10.4.0 and forward are now found on `mavenCentral()` instead of `jcenter()`.
 
 A minimal supported setup will therefore now look like this:
 
@@ -76,7 +144,7 @@ for more information.
 * None.
 
 ### Fixes
-* RxJava Flowables/Observables and Coroutine Flows would crash if they were created from a `RealmList` and the parent object holding the list was deleted. Now, the stream is disposed/closed instead. (Issue [#7242](https://github.com/realm/realm-java/issues/7242)) 
+* RxJava Flowables/Observables and Coroutine Flows would crash if they were created from a `RealmList` and the parent object holding the list was deleted. Now, the stream is disposed/closed instead. (Issue [#7242](https://github.com/realm/realm-java/issues/7242))
 * Fixes Realm models default values containing objects with a PK might crash with a `RealmPrimaryKeyConstraintException`. (Issue [#7269] (https://github.com/realm/realm-java/issues/7269))
 
 ### Compatibility

Dockerfile

@@ -14,7 +14,7 @@ ENV ANDROID_HOME /opt/android-sdk-linux
 # Need by cmake
 ENV ANDROID_NDK_HOME /opt/android-ndk
 ENV ANDROID_NDK /opt/android-ndk
-ENV PATH ${PATH}:${ANDROID_HOME}/emulator:${ANDROID_HOME}/tools:${ANDROID_HOME}/tools/bin:${ANDROID_HOME}/platform-tools
+ENV PATH ${PATH}:${ANDROID_HOME}/emulator:${ANDROID_HOME}/cmdline-tools/latest:${ANDROID_HOME}/cmdline-tools/latest/bin:${ANDROID_HOME}/platform-tools
 ENV PATH ${PATH}:${NDK_HOME}
 ENV NDK_CCACHE /usr/bin/ccache
 ENV CCACHE_CPP2 yes
@@ -52,9 +52,12 @@ RUN DEBIAN_FRONTEND=noninteractive \
     && apt-get clean
 
 # Install the Android SDK
+# See https://developer.android.com/studio/index.html#downloads for latest version
 RUN cd /opt && \
-    wget -q https://dl.google.com/android/repository/sdk-tools-linux-3859397.zip -O android-tools-linux.zip && \
-    unzip android-tools-linux.zip -d ${ANDROID_HOME} && \
+    wget -q https://dl.google.com/android/repository/commandlinetools-linux-7302050_latest.zip -O android-tools-linux.zip && \
+    mkdir --parents ${ANDROID_HOME}/cmdline-tools/latest && \
+    unzip android-tools-linux.zip -d ${ANDROID_HOME}/cmdline-tools/latest && \
+    mv ${ANDROID_HOME}/cmdline-tools/latest/cmdline-tools/* ${ANDROID_HOME}/cmdline-tools/latest/ && \
     rm -f android-tools-linux.zip
 
 # Grab what's needed in the SDK

Jenkinsfile

@@ -7,9 +7,9 @@ import groovy.json.JsonOutput
 // CONSTANTS
 
 // Branches from which we release SNAPSHOT's. Only release branches need to run on actual hardware.
-releaseBranches = ['master', 'next-major', 'v10', 'releases']
+releaseBranches = ['master', 'next-major', 'support-new-datatypes', 'releases']
 // Branches that are "important", so if they do not compile they will generate a Slack notification
-slackNotificationBranches = [ 'master', 'releases', 'next-major', 'v10' ]
+slackNotificationBranches = [ 'master', 'releases', 'next-major', 'support-new-datatypes' ]
 // WARNING: Only set to `false` as an absolute last resort. Doing this will disable all integration
 // tests.
 enableIntegrationTests = true
@@ -236,7 +236,20 @@ def runBuild(buildFlags, instrumentationTestTarget) {
       if (isReleaseBranch) {
         signingFlags = "-PsignBuild=true -PsignSecretRingFile=\"${SIGN_KEY}\" -PsignPassword=${SIGN_KEY_PASSWORD}"
       }
-      sh "./gradlew assemble ${buildFlags} ${signingFlags} --stacktrace"
+      // Work around https://github.com/realm/realm-java/issues/7476 by building each artifact independantly instead 
+      // of using Gradle to call down into sub projects (which seems to trigger a bug somewhere).
+      sh """
+        cd realm-annotations
+        ./gradlew publishToMavenLocal ${buildFlags} ${signingFlags} --stacktrace
+        cd ../realm-transformer
+        ./gradlew publishToMavenLocal ${buildFlags} ${signingFlags} --stacktrace
+        cd ../library-build-transformer
+        ./gradlew publishToMavenLocal ${buildFlags} ${signingFlags} --stacktrace
+        cd ../gradle-plugin
+        ./gradlew publishToMavenLocal ${buildFlags} ${signingFlags} --stacktrace
+        cd ../realm
+        ./gradlew publishToMavenLocal ${buildFlags} ${signingFlags} --stacktrace
+      """
     }
   }
 
@@ -308,7 +321,7 @@ def runBuild(buildFlags, instrumentationTestTarget) {
     },
     'Gradle Plugin' : {
       try {
-        gradle('gradle-plugin', 'check --debug')
+        gradle('gradle-plugin', 'check')
       } finally {
         storeJunitResults 'gradle-plugin/build/test-results/test/TEST-*.xml'
       }

dependencies.list

@@ -4,7 +4,7 @@ REALM_CORE=10.3.3
 
 # Version of MongoDB Realm used by integration tests
 # See https://github.com/realm/ci/packages/147854 for available versions
-MONGODB_REALM_SERVER=2021-03-09
+MONGODB_REALM_SERVER=2021-04-22
 
 # Common Android settings across projects
 GRADLE_BUILD_TOOLS=4.0.0
@@ -13,7 +13,7 @@ KOTLIN=1.3.72
 KOTLIN_COROUTINES=1.3.9
 
 # Common classpath dependencies
-gradle=6.5
+gradle=6.8.3
 ndkVersion=22.0.7026061
 BUILD_INFO_EXTRACTOR_GRADLE=4.17.0
 GRADLE_BINTRAY_PLUGIN=1.8.5

examples/gradle/wrapper/gradle-wrapper.jar

@@ -1,5 +1,5 @@
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-6.5-all.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-6.8.3-bin.zip
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists

examples/gradle/wrapper/gradle-wrapper.properties

@@ -82,6 +82,7 @@ esac
 
 CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
 
+
 # Determine the Java command to use to start the JVM.
 if [ -n "$JAVA_HOME" ] ; then
     if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
@@ -129,6 +130,7 @@ fi
 if [ "$cygwin" = "true" -o "$msys" = "true" ] ; then
     APP_HOME=`cygpath --path --mixed "$APP_HOME"`
     CLASSPATH=`cygpath --path --mixed "$CLASSPATH"`
+    
     JAVACMD=`cygpath --unix "$JAVACMD"`
 
     # We build the pattern for arguments to be converted via cygpath
@@ -154,19 +156,19 @@ if [ "$cygwin" = "true" -o "$msys" = "true" ] ; then
         else
             eval `echo args$i`="\"$arg\""
         fi
-        i=$((i+1))
+        i=`expr $i + 1`
     done
     case $i in
-        (0) set -- ;;
-        (1) set -- "$args0" ;;
-        (2) set -- "$args0" "$args1" ;;
-        (3) set -- "$args0" "$args1" "$args2" ;;
-        (4) set -- "$args0" "$args1" "$args2" "$args3" ;;
-        (5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;;
-        (6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;;
-        (7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;;
-        (8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;;
-        (9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;;
+        0) set -- ;;
+        1) set -- "$args0" ;;
+        2) set -- "$args0" "$args1" ;;
+        3) set -- "$args0" "$args1" "$args2" ;;
+        4) set -- "$args0" "$args1" "$args2" "$args3" ;;
+        5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;;
+        6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;;
+        7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;;
+        8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;;
+        9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;;
     esac
 fi
 
@@ -175,14 +177,9 @@ save () {
     for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done
     echo " "
 }
-APP_ARGS=$(save "$@")
+APP_ARGS=`save "$@"`
 
 # Collect all arguments for the java command, following the shell quoting and substitution rules
 eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS "\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\"" org.gradle.wrapper.GradleWrapperMain "$APP_ARGS"
 
-# by default we should be in the correct project dir, but when run from Finder on Mac, the cwd is wrong
-if [ "$(uname)" = "Darwin" ] && [ "$HOME" = "$PWD" ]; then
-  cd "$(dirname "$0")"
-fi
-
 exec "$JAVACMD" "$@"

examples/gradlew

@@ -29,6 +29,9 @@ if "%DIRNAME%" == "" set DIRNAME=.
 set APP_BASE_NAME=%~n0
 set APP_HOME=%DIRNAME%
 
+@rem Resolve any "." and ".." in APP_HOME to make it shorter.
+for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi
+
 @rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
 set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m"
 
@@ -81,6 +84,7 @@ set CMD_LINE_ARGS=%*
 
 set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
 
+
 @rem Execute Gradle
 "%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %CMD_LINE_ARGS%
 

examples/gradlew.bat

@@ -1,5 +1,5 @@
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-6.5-all.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-6.8.3-bin.zip
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists