Refactor to NPM Packages (#894)

* Update schema to right folders and namespaces
Move SDK to package. TODO Create script to upgrade package to new SpatialOS version
Delete CoreSDK downloader
Add some temporary helper scripts

* Fix code generation for packages that are dependencies only

* Generate schema descriptor
Fix local deployment launcher

* Checkout sdk packages
Initial version of sdk package update scripts. Still need to move and improve
Move packaged schema from `Schema` to `.schema` to avoid needless meta files.

* Update scripts to download into the right folders

* Move SDK Update scripts and combine
Update SDK Script to clean unneeded binaries

* Fix update sdk script when run from outside scripts folder
move republish script

* Have npm ignore the gitignore files
Add missing .gitignore file

* Add Schema Descriptor path to GdkToolsConfiguration
Fix window drawing to not be a hack

* Fix code generation for test-project

* Fixed error in download for mobile SDK

* Fixed metafile names

* add CoreSDK OSX meta files

* Set repo to cloudsmith.io
Split scripts into multiple

* update sdk downloading scripts for 13.7.1

* update update script

* add empty plugins dir

* Rename all packages to io.improbable.*

* Remove package specific git ignore from root

* rename package dependencies to new naming style

* Update application identifiers

* Fix NPM CI (#1015)

* upgrade guide

* Update package definition descriptions

* Update NPM Registry url in publish script

* Update gitignore for code generator

* Remove old gitignore entry for test-project schema from gdk packages

Author: zeroZshadow

.buildkite/premerge.steps.yaml

@@ -13,7 +13,7 @@ common: &common
     - "environment=production"
     - "permission_set=builder"
     - "platform=windows"
-    - "queue=v3-1558109315-a75758857a3cdfbe-------z"
+    - "queue=v3-1562257704-92dadf3bbbe69f0b-------z"
     - "scaler_version=2"
     - "minimum_instances=1"
     - "agent_count=1"

.buildkite/release-qa.steps.yaml

@@ -13,7 +13,7 @@ common: &common
     - "environment=production"
     - "permission_set=builder"
     - "platform=windows"
-    - "queue=v3-1558109315-a75758857a3cdfbe-------z"
+    - "queue=v3-1562257704-92dadf3bbbe69f0b-------z"
     - "scaler_version=2"
     - "minimum_instances=1"
     - "agent_count=1"

.gitignore

@@ -47,8 +47,6 @@ workers/unity/Assets/Plugins/Improbable.meta
 
 _ReSharper.Caches
 
-workers/unity/Packages/com.improbable.gdk.tools/.CodeGenerator/GdkCodeGenerator/Generated
-
 # Jetbrains
 workers/unity/Assets/Plugins/Editor/Jetbrains
 workers/unity/Assets/Plugins/Editor/Jetbrains.meta
@@ -65,7 +63,6 @@ test-project/Assets/Plugins/Improbable.meta
 test-project/ProjectSettings/UnityConnectSettings.asset
 test-project/Assets/Generated/Source.meta
 test-project/Assets/Generated/Source/**/*.*
-test-project/Assets/.Schema/from_gdk_packages
 !test-project/Assets/Generated/Source/improbable/gdk/tests/**/*.cs
 
 .shared-ci/

CHANGELOG.md

@@ -5,17 +5,21 @@
 ### Breaking Changes
 
 - Renamed the `buildTarget` command line argument to `buildEnvironment`. [#1012](https://github.com/spatialos/gdk-for-unity/pull/1012)
+- All GDK packages now have an `io.improbable` prefix instead of `com.improbable`.
 
 ### Added
 
 - Added the `LinkedGameObjectMap` class for finding the `GameObject`(s) linked with a specified `EntityId`. [#1013](https://github.com/spatialos/gdk-for-unity/pull/1013)
     - This can be used with the `[Require]` annotation to inject it into your `MonoBehaviours` provided you are using the `GameObjectCreation` feature module. For example: `[Require] private LinkedGameObjectMap gameObjectMap;`
 - Added the ability for the build system to build specific targets of a given build environment. [#1012](https://github.com/spatialos/gdk-for-unity/pull/1012)
     - Use the `buildTargetFilter` command line argument to pass in a comma delimited list of build targets to filter for. For example, `+buildTargetFilter win,macos`.
+- Added two new GDK packages: `io.improbable.worker.sdk` and `io.improbable.worker.sdk.mobile` which contain the underlying Worker SDK packages for Windows/MacOS/Linux and Android/iOS respectively.
 
 ### Internal
 
 - Stopped throwing a `Test Exception` in playground.
+- The embedded `DownloadCoreSdk` `dotnet` project has been removed from `Improbable.Gdk.Tools`.
+- The `PluginPostprocessor` tool has been removed from `Improbable.Gdk.Tools`.
 
 ## `0.2.4` - 2019-06-28
 

UPGRADE_GUIDE.md

@@ -2,6 +2,28 @@
 
 ## From `0.2.4` to `0.2.5`
 
+### NPM Packages
+
+From release `0.2.5` onward, all GDK packages and feature modules will be delivered through the Unity Package Manager. To upgrade your project to use these packages do the following:
+
+1. Open the `Packages/manifest.json` file inside your Unity project.
+2. Add the following JSON snippet to the this file, under the `"registry: "https://packages.unity.com"` line:
+    ```json
+    "scopedRegistries": [
+      {
+        "name": "Improbable",
+        "url": "https://npm.improbable.io/gdk-for-unity/",
+        "scopes": [
+          "io.improbable"
+        ]
+      }
+    ]
+    ```
+3. Under the `dependencies` field, find all `com.improbable.*` packages and change the names to `io.improbable.*`.
+4. For all these packages, change the value from `file:<something>` to `0.2.5`.
+
+> **Note:** You no longer need to checkout the `gdk-for-unity` repository side-by-side your project to sideload these packages.
+
 ### Build system changes
 
 When calling the build system from the command line, you must change the `buildTarget` argument to `buildEnvironment`. Previously you may have called the build system like so:

ci/bootstrap.sh

@@ -22,3 +22,6 @@ pushd "${SHARED_CI_DIR}"
     git fetch --depth 20 origin master
     git checkout "${PINNED_SHARED_CI_VERSION}"
 popd
+
+# Download local copy of the SDK packages.
+./init.sh

ci/publish_packages.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+
+set -e -u -o pipefail
+
+cd "$(dirname "$0")/../"
+
+PKG_ROOT="workers/unity/Packages"
+REGISTRY="https://npm.improbable.io/gdk-for-unity/"
+
+# Re-publish all packages
+pushd $PKG_ROOT
+    for dir in */;
+    do
+        echo $dir
+        pushd $dir
+            npm publish --registry="${REGISTRY}"
+        popd
+    done
+popd

ci/test.sh

@@ -35,10 +35,10 @@ cleanUnity "$(pwd)/test-project"
 
 echo "Code Generator Testing"
 
-dotnet test --logger:"nunit;LogFilePath=${CODE_GEN_LIB_TEST_RESULTS_FILE}" workers/unity/Packages/com.improbable.gdk.tools/.CodeGenerator/CodeGeneration/CodeGeneration.csproj
+dotnet test --logger:"nunit;LogFilePath=${CODE_GEN_LIB_TEST_RESULTS_FILE}" workers/unity/Packages/io.improbable.gdk.tools/.CodeGenerator/CodeGeneration/CodeGeneration.csproj
 CODE_GEN_LIB_TEST_RESULT=$?
 
-dotnet test --logger:"nunit;LogFilePath=${CODE_GENERATOR_TEST_RESULTS_FILE}" workers/unity/Packages/com.improbable.gdk.tools/.CodeGenerator/GdkCodeGenerator/GdkCodeGenerator.csproj
+dotnet test --logger:"nunit;LogFilePath=${CODE_GENERATOR_TEST_RESULTS_FILE}" workers/unity/Packages/io.improbable.gdk.tools/.CodeGenerator/GdkCodeGenerator/GdkCodeGenerator.csproj
 CODE_GENERATOR_TEST_RESULT=$?
 
 echo "Editmode Testing"

init.ps1

@@ -0,0 +1,41 @@
+cd $PSScriptRoot
+
+$PkgRoot = $PSScriptRoot + "/workers/unity/Packages"
+$SdkPath = $PkgRoot + "/io.improbable.worker.sdk"
+$SdkMobilePath = $PkgRoot + "/io.improbable.worker.sdk.mobile"
+
+$SdkVersion = Get-Content ($SdkPath + "/package.json") | jq -r '.version'
+
+function UpdatePackage($type, $identifier, $path, $removes)
+{
+    spatial package get $type $identifier $SdkVersion "$path" --unzip --force --json_output
+
+    if ($removes -ne $null)
+    {
+        $removes.Split(";") | ForEach {
+            rm "$path/$_"
+        }
+    }
+}
+
+UpdatePackage worker_sdk core-dynamic-x86_64-linux "$SdkPath/Plugins/Improbable/Core/Linux/x86_64"
+UpdatePackage worker_sdk core-bundle-x86_64-macos "$SdkPath/Plugins/Improbable/Core/OSX"
+UpdatePackage worker_sdk core-dynamic-x86_64-win32 "$SdkPath/Plugins/Improbable/Core/Windows/x86_64" "CoreSdkDll.lib"
+
+UpdatePackage worker_sdk csharp-c-interop "$SdkPath/Plugins/Improbable/Sdk/Common" "Improbable.Worker.CInterop.pdb"
+
+UpdatePackage schema standard_library "$SdkPath/.schema"
+
+UpdatePackage tools schema_compiler-x86_64-win32 "$SdkPath/.schema_compiler"
+UpdatePackage tools schema_compiler-x86_64-macos "$SdkPath/.schema_compiler"
+UpdatePackage tools schema_compiler-x86_64-linux "$SdkPath/.schema_compiler"
+
+#Update Mobile SDK
+UpdatePackage worker_sdk core-static-fullylinked-arm-ios "$SdkMobilePath/Plugins/Improbable/Core/iOS/arm" "CoreSdkStatic.lib;libCoreSdkStatic.a.pic"
+UpdatePackage worker_sdk core-static-fullylinked-x86_64-ios "$SdkMobilePath/Plugins/Improbable/Core/iOS/x86_64" "CoreSdkStatic.lib;libCoreSdkStatic.a.pic"
+
+UpdatePackage worker_sdk core-dynamic-arm64v8a-android "$SdkMobilePath/Plugins/Improbable/Core/Android/arm64"
+UpdatePackage worker_sdk core-dynamic-armv7a-android "$SdkMobilePath/Plugins/Improbable/Core/Android/armv7"
+UpdatePackage worker_sdk core-dynamic-x86-android "$SdkMobilePath/Plugins/Improbable/Core/Android/x86"
+
+UpdatePackage worker_sdk csharp-c-interop-static "$SdkMobilePath/Plugins/Improbable/Sdk/iOS" "Improbable.Worker.CInteropStatic.pdb"

init.sh

@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+set -e -u -o pipefail
+
+cd "$(dirname "$0")"
+
+PKG_ROOT="workers/unity/Packages"
+SDK_PATH="${PKG_ROOT}/io.improbable.worker.sdk"
+SDK_MOBILE_PATH="${PKG_ROOT}/io.improbable.worker.sdk.mobile"
+
+SDK_VERSION="$(cat "${SDK_PATH}"/package.json | jq -r '.version')"
+
+update_package() {
+    local type=$1
+    local identifier=$2
+    local path=$3
+
+    spatial package get $type $identifier $SDK_VERSION "${path}" --unzip --force --json_output
+
+    local files=${4:-""}
+    for file in $(echo $files | tr ";" "\n"); do
+        rm "${path}/${file}"
+    done
+}
+
+# Update Core SDK
+update_package worker_sdk core-dynamic-x86_64-linux "${SDK_PATH}/Plugins/Improbable/Core/Linux/x86_64"
+update_package worker_sdk core-bundle-x86_64-macos "${SDK_PATH}/Plugins/Improbable/Core/OSX"
+update_package worker_sdk core-dynamic-x86_64-win32 "${SDK_PATH}/Plugins/Improbable/Core/Windows/x86_64" "CoreSdkDll.lib"
+
+update_package worker_sdk csharp-c-interop "${SDK_PATH}/Plugins/Improbable/Sdk/Common" "Improbable.Worker.CInterop.pdb"
+
+update_package schema standard_library "${SDK_PATH}/.schema"
+
+update_package tools schema_compiler-x86_64-win32 "${SDK_PATH}/.schema_compiler"
+update_package tools schema_compiler-x86_64-macos "${SDK_PATH}/.schema_compiler"
+update_package tools schema_compiler-x86_64-linux "${SDK_PATH}/.schema_compiler"
+
+#Update Mobile SDK
+update_package worker_sdk core-static-fullylinked-arm-ios "${SDK_MOBILE_PATH}/Plugins/Improbable/Core/iOS/arm" "CoreSdkStatic.lib;libCoreSdkStatic.a.pic"
+update_package worker_sdk core-static-fullylinked-x86_64-ios "${SDK_MOBILE_PATH}/Plugins/Improbable/Core/iOS/x86_64" "CoreSdkStatic.lib;libCoreSdkStatic.a.pic"
+
+update_package worker_sdk core-dynamic-arm64v8a-android "${SDK_MOBILE_PATH}/Plugins/Improbable/Core/Android/arm64"
+update_package worker_sdk core-dynamic-armv7a-android "${SDK_MOBILE_PATH}/Plugins/Improbable/Core/Android/armv7"
+update_package worker_sdk core-dynamic-x86-android "${SDK_MOBILE_PATH}/Plugins/Improbable/Core/Android/x86"
+
+update_package worker_sdk csharp-c-interop-static "${SDK_MOBILE_PATH}/Plugins/Improbable/Sdk/iOS" "Improbable.Worker.CInteropStatic.pdb"