Chrimc/id (#739)

* Initial check-in.

* Allof Working.

* Move definitions local.

* Launch to sdk.

* Everything working for new merge.

* schema: working

* Working except that allof expansion does not expand
local references.

* Simplification and some analysis working.

* Add extends.

* Everything working.

* Update help

* Fix typo

* Expand allOf and validate final schema.

* Missing as warning and validate schema.

* Update tests.

* Update test to pass on server.

* Normalize paths.

* Check normalize for mac

* Update for mac paths and lint.

* Address review comments.

* Update documentation and fix $id bugs.

* Address review commehts

* Fix bad merge.

Author: Chris McConnell

packages/dialog/README.md

@@ -11,8 +11,9 @@ This package is intended for Microsoft use only and should be consumed through @
 # Commands
 <!-- commands -->
 * [`bf `](#bf-)
-* [`bf dialog:merge GLOB1 [GLOB2] [GLOB3] [GLOB4] [GLOB5] [GLOB6] [GLOB7] [GLOB8] [GLOB9]`](#bf-dialogmerge-glob1-glob2-glob3-glob4-glob5-glob6-glob7-glob8-glob9)
-* [`bf dialog:verify GLOB1 [GLOB2] [GLOB3] [GLOB4] [GLOB5] [GLOB6] [GLOB7] [GLOB8] [GLOB9]`](#bf-dialogverify-glob1-glob2-glob3-glob4-glob5-glob6-glob7-glob8-glob9)
+* [`bf dialog`](#bf-dialog)
+* [`bf dialog:merge PATTERNS`](#bf-dialogmerge-patterns)
+* [`bf dialog:verify PATTERNS`](#bf-dialogverify-patterns)
 
 ## `bf `
 
@@ -26,89 +27,60 @@ OPTIONS
   -h, --help  show CLI help
 ```
 
-_See code: [src/commands/index.ts](https://github.com/microsoft/botframework-cli/src/commands/index.ts)_
+_See code: [src\commands\index.ts](https://github.com/microsoft/botframework-cli/tree/master/packages/dialog/blob/v1.0.0/src\commands\index.ts)_
 
-## `bf dialog:merge GLOB1 [GLOB2] [GLOB3] [GLOB4] [GLOB5] [GLOB6] [GLOB7] [GLOB8] [GLOB9]`
+## `bf dialog`
 
-```
-USAGE
-  $ bf dialog:merge GLOB1 [GLOB2] [GLOB3] [GLOB4] [GLOB5] [GLOB6] [GLOB7] [GLOB8] [GLOB9]
-
-OPTIONS
-  -b, --branch=branch  [default: master] The branch to use for the meta-schema component.schema.
-  -h, --help           show CLI help
-  -o, --output=output  [default: app.schema] Output path and filename for merged schema. [default: app.schema]
-
-  -u, --update         Update .schema files to point the <branch> component.schema and regenerate component.schema if
-                       baseComponent.schema is present.
-
-  --verbose            output verbose logging of files as they are processed
-```
-
-_See code: [src/commands/dialog/merge.ts](https://github.com/microsoft/botframework-cli/src/commands/dialog/merge.ts)_
-
-## `bf dialog:verify GLOB1 [GLOB2] [GLOB3] [GLOB4] [GLOB5] [GLOB6] [GLOB7] [GLOB8] [GLOB9]`
+Dialog related commands for working with .schema and .dialog files.
 
 ```
 USAGE
-  $ bf dialog:verify GLOB1 [GLOB2] [GLOB3] [GLOB4] [GLOB5] [GLOB6] [GLOB7] [GLOB8] [GLOB9]
+  $ bf dialog
 
 OPTIONS
-  -h, --help  show CLI help
-  --verbose   Show verbose output
+  -h, --help  Dialog command help
 ```
 
-_See code: [src/commands/dialog/verify.ts](https://github.com/microsoft/botframework-cli/src/commands/dialog/verify.ts)_
-<!-- commandsstop -->
-* [`bf dialog:merge [FILE]`](#bf-dialogmerge-file)
-* [`bf dialog:verify [FILE]`](#bf-dialogverify-file)
-
-## `bf dialog:merge [FILE]`
+_See code: [src\commands\dialog\index.ts](https://github.com/microsoft/botframework-cli/tree/master/packages/dialog/blob/v1.0.0/src\commands\dialog\index.ts)_
 
-The bf dialog:merge creates an merged schema file which represents merging of all of the component
-schemas and the SDK schemas together into an application .schema file.
+## `bf dialog:merge PATTERNS`
 
-The file pattern can be an arbitrary GLOB patterns for finding .schema files (such as myfolder/**/*.schema), but
-the better way to use it is to invoke it in the folder that has a project in it (either .csproj or packages.json).
-In that case the project file will be analyzed for all dependent folders and .schema files will be merged to create
-the app.schema for the project.
+Merge component .schema files into an app.schema.
 
 ```
 USAGE
-  $ bf dialog:merge GLOB1 [GLOB2] [GLOB3] [GLOB4] [GLOB5] [GLOB6] [GLOB7] [GLOB8] [GLOB9]
+  $ bf dialog:merge PATTERNS
+
+ARGUMENTS
+  PATTERNS  Any number of glob regex patterns to match .schema, .csproj, or package.json files.
 
 OPTIONS
-  -b, --branch=branch  The branch to use for the meta-schema component.schema. [default: master] 
-  -h, --help              show CLI help
-  -o, --output=output  Output path and filename for merged schema. [default: app.schema]
-  -u, --update         Update .schema files to point the <branch> component.schema and regenerate component.schema if baseComponent.schema is present.
-  --verbose            output verbose logging of files as they are processed
-```
+  -h, --help           show CLI help
+  -o, --output=output  [default: app.schema] Output path and filename for merged schema.
+  -v, --verbose        Show verbose logging of files as they are processed.
 
-Example:
-```
-bf dialog:merge -o app.schema
+EXAMPLES
+  $ bf dialog:merge *.csproj
+  $ bf dialog:merge libraries/**/*.schema **/*.csproj -o app.schema
 ```
 
-_See code: [src/commands/dialog/merge.ts](https://github.com/microsoft/botframework-cli/blob/v0.0.0/src/commands/dialog/merge.ts)_
+_See code: [src\commands\dialog\merge.ts](https://github.com/microsoft/botframework-cli/tree/master/packages/dialog/blob/v1.0.0/src\commands\dialog\merge.ts)_
 
-## `bf dialog:verify [FILE]`
+## `bf dialog:verify PATTERNS`
 
-The dialog:verify command is used to validate that all of the .dialog file resources for a project are valid based on the 
-applications app.schema file (see dialog:merge command).
+Verify .dialog files match their app.schema.
 
 ```
 USAGE
-  $ bf dialog:verify GLOB1 [GLOB2] [GLOB3] [GLOB4] [GLOB5] [GLOB6] [GLOB7] [GLOB8] [GLOB9]
+  $ bf dialog:verify PATTERNS
+
+ARGUMENTS
+  PATTERNS  Any number of glob regex patterns to match .dialog files.
 
 OPTIONS
   -h, --help  show CLI help
   --verbose   Show verbose output
 ```
 
-Example:
-``` 
-bf dialog:verify test/**/*.dialog
-```
-
-_See code: [src/commands/dialog/verify.ts](https://github.com/microsoft/botframework-cli/blob/v0.0.0/src/commands/dialog/verify.ts)_
+_See code: [src\commands\dialog\verify.ts](https://github.com/microsoft/botframework-cli/tree/master/packages/dialog/blob/v1.0.0/src\commands\dialog\verify.ts)_
+<!-- commandsstop -->

packages/dialog/package.json

@@ -24,7 +24,7 @@
       "@oclif/plugin-help"
     ]
   },
-  "repository": "microsoft/botframework-cli",
+  "repository": "https://github.com/microsoft/botframework-cli/tree/master/packages/dialog",
   "scripts": {
     "postpack": "rimraf oclif.manifest.json",
     "posttest": "tslint -p test -t stylish",

packages/dialog/src/commands/dialog/merge.ts

@@ -18,7 +18,7 @@ export default class DialogMerge extends Command {
     static flags: flags.Input<any> = {
         debug: flags.boolean({ char: 'd', description: 'Generate debug files.', hidden: true, default: false}),
         help: flags.help({ char: 'h' }),
-        output: flags.string({ char: 'o', description: 'Output path and filename for merged schema. [default: app.schema]', default: 'app.schema', required: false }),
+        output: flags.string({ char: 'o', description: 'Output path and filename for merged schema.', default: 'app.schema', required: false }),
         verbose: flags.boolean({ char: 'v', description: 'Show verbose logging of files as they are processed.', default: false }),
     }
 

packages/dialog/src/commands/dialog/readme.md

@@ -2,24 +2,20 @@
 
 ## Merge
 
-Ths will merge together [Microsoft Bot Builder](https://github.com/Microsoft/BotBuilder) .schema JSON schema files into a single JSON schema file. You can point to the files either directly with a glob pattern or indirectly through a glob pattern that matches a package.json, packages.config or \*.csproj file. The .schema files should have a unique filename that is used to refer to that type using `$kind`. The .schema files can optionally include a `$schema: "https://raw.githubusercontent.com/Microsoft/botbuilder-tools/SchemaGen/packages/DialogSchema/src/dialogSchema.schema"` which defines the schema they are validated against. Within a schema definition you can use `$role:"implements(<kind>)` to project the type definition into interface types defined using `$role:"interface"` while merging. To refer to a type in a property, just use `"$kind":"<kind>"`. The merger combines all of the component .schema files into a single .schema file that has resolved all external `$ref`, merged `allOf` and connected together schemas through `$role` and `$kind`.
+This will merge together [Microsoft Bot Builder](https://github.com/Microsoft/BotBuilder) .schema JSON schema files into a single JSON schema file. You can point to the files either directly with a glob pattern or indirectly through a glob pattern that matches a package.json, packages.config or \*.csproj file. The .schema files should have a unique filename that is used to refer to that type using `$kind`. The .schema files should include a `$schema: "https://raw.githubusercontent.com/microsoft/botframework-sdk/master/schemas/component/component.schema"` which defines the schema they are validated against. You can access common definitions by using `$ref: "schema:#/definitions/stringExpression"` in a property definition.  At the top-level in a component schema you can use `$role:"implements(<kind>)` to project the type definition into interface types defined using `$role:"interface"` while merging. To extend an existing .schema file use `$role: "extends(<kind>)"`.  To refer to a type in a property, use `"$kind":"<kind>"` and the corresponding kind will be wired into your schema.  The merger combines all of the component .schema files into a single .schema file that has resolved all external `$ref`, merged `allOf` and connected together schemas through `$role` and `$kind`.
 
-In addition to types you can also mark properties with a `$role` which will define the underlying type and restrictions. This is also useful information for UI tools to help you construct values. Roles include:
-
-- `$role: "expression"` which marks a string property which is expected to contain an expression string.
-- `$role: "lg"` which marks a string property which is used for language generation and can refer to LG templates.
-- `$role: "memoryPath"` which marks a string property which is expected to contain a path in memory like `user.name`.
+You can also mark properties with a `$role: "expression"` to indicate to tooling that an expression is allowed for that property. 
 
 For example look at these files:
 
 - [IRecognizer.schema](test/schemas/IRecognizer.schema) defines the place holder for `IRecognizer` including a default option which is a bare string.
 - [Recognizer.schema](test/schemas/Recognizer.schema) includes `$role:"implements(IRecognizer)"` which extends the `IRecognizer` definition when merged.
-- [root.schema](test/schemas/root.schema) is a schema file that includes `$kind:"IRecognizer"` in order to make use of the `IRecognizer` place holder.
-- [app.schema](test/examples/app.schema) was created by this tool shows how all of these definitions are merged together. In particular if you look at `IRecognizer` you will see the definition that includes a string, or the complete definition of `Recognizer`.
+- [root.schema](test/schemas/root.schema) is a schema file that includes `$kind:"IRecognizer"` in order to make use of the `IRecognizer` place holder.  The `$role: []` ensures that this is available as a top-level object.
+- [app.schema](test/schemas/app.schema) was created by this tool shows how all of these definitions are merged together. In particular if you look at `IRecognizer` you will see the definition that includes a string, or the complete definition of `Recognizer`.
 
 [root.dialog](test/examples/root.dialog) Shows how you could use the resulting schema to enter in JSON schema and get intellisense.
 
 ## Verify
 
-The verify command will check .dialog files to ensure that they all meet the schema.
+The verify command will check .dialog files to ensure that they all meet their $schema.
 

packages/dialog/src/library/schemaMerger.ts

@@ -194,7 +194,7 @@ export default class SchemaMerger {
                 }
                 let finalSchema: any = {
                     $schema: this.metaSchemaId,
-                    $id: ppath.resolve(this.output),
+                    $id: `file:///${ppath.resolve(this.output).replace(/\\/g, '/')}#`,
                     type: 'object',
                     title: 'Component kinds',
                     description: 'These are all of the kinds that can be created by the loader.',
@@ -208,6 +208,7 @@ export default class SchemaMerger {
                 // Convert all remote references to local ones
                 finalSchema = await parser.bundle(finalSchema as parser.JSONSchema, this.schemaProtocolResolver())
                 finalSchema = this.expandAllOf(finalSchema)
+                this.removeId(finalSchema)
                 if (this.debug) {
                     await fs.writeJSON(this.output + '.expanded', finalSchema, this.jsonOptions)
                 }
@@ -758,6 +759,16 @@ export default class SchemaMerger {
         return allof(bundle)
     }
 
+    // Remove any child $id because their references have been changed to be local
+    removeId(bundle: any) {
+        walkJSON(bundle, (val, _obj, path) => {
+            if (path && val.$id) {
+                delete val.$id
+            }
+            return false
+        })
+    }
+
     // Verify schema has title/description everywhere and interfaces exist.
     verifySchema(schema: any): void {
         this.log('Verifying schema')
@@ -766,7 +777,7 @@ export default class SchemaMerger {
             this.currentKind = entry.$ref.substring(entry.$ref.lastIndexOf('/') + 1)
             let definition = schema.definitions[this.currentKind]
             let verifyProperty = (val, path) => {
-                if (!val.$id) {
+                if (!val.$schema) {
                     if (val.$ref) {
                         val = clone(val)
                         let ref = ptr.get(schema, val.$ref)
@@ -804,7 +815,7 @@ export default class SchemaMerger {
                 }
             }
             walkJSON(definition, (val, _, path) => {
-                if (val.$id && path) {
+                if (val.$schema && path) {
                     return true
                 }
                 if (val.properties && !path?.endsWith('properties')) {

packages/dialog/test/commands/dialog/schemas/app.schema

@@ -1,6 +1,6 @@
 {
 	"$schema": "https://raw.githubusercontent.com/microsoft/botframework-sdk/master/schemas/component/component.schema",
-	"$id": "C:\\Users\\chrim\\source\\repos\\botframework-cli\\packages\\dialog\\test\\commands\\dialog\\schemas\\app.schema",
+	"$id": "file:///C:/Users/chrim/source/repos/botframework-cli/packages/dialog/test/commands/dialog/schemas/app.schema#",
 	"type": "object",
 	"title": "Component kinds",
 	"description": "These are all of the kinds that can be created by the loader.",