Initial check-in of dialog generation. (#353)

* Intermediate check-in before inverting
template/prop loops.

* Working generator with incomplete templates.

* Basic .lu files working.

* Update to generate single .lu file.

* Fix cli issues.

* Fix CLI issues with template resources.

* Clean up lint warnings.

* Fix lint errors.

* Generate composite .lu

* Add dialog and multi-language support.

* shortcut to debug

* Add per-locale .lg file.

* Update debug config to use temp directory.
Generate multilingual .lg file.

* Update to include schema prefix on names.

* Fix bugs in .lg genertion.

* try help

* help and cancel

* Copy schema to output directory.

* help and cancel version 1

* separate help.lu

* several fixes

* remove help enum

* reset launch.json

* remove empty line

* Add the ability to copy library dialog files.

* Add ability to specify schema and multiple
templates.

* Rename resources.
Support multiple templates.
Add *PROPN** pattern.

* Fix bug in striingify call.

* Generate schema.main.dialog and update config.

* Update templates to newest format.

* Update templates to match names.

* add .dialogs for several basic rules

* Update templates to use $ and add begindialog as a hack to work around ada
ptive issue.

* add .dialogs for several basic rules

* fix comments

* merge from origin

* small changes

* Add formCopy cmd file.

* Adjust for bug in templates.

* fix path

* Version 1 of contact.form.dialog

* Same info is modeled for both Home and Work.
* List of state abbreviations and their spelled-out names are specified in the separate state.en-us.lu, and used in the $mappings specifications.
* Latitude and Longitude should be auto-filled if the corresponding address are filled.
* Validation patterns are specified for zip codes and phone numbers.

* Fixed contact.form.dialog and added activity.form.dialog

* contact.form.dialog
  - Fixed spelling
  - Use phonenumber prebuilt entity type
* activity.form.dialog: the second scenario from Darren "An employee working for a customer wants to add a quick update to a customer record after a call. This update will then be available for other people working with that customer to see".

* Added opportunity.form.dialog

For Darren's scenario "An employee working with a customer has had a meeting and needs to create a new opportunity."

* change names

* Revert "change names"

This reverts commit 64ffaff.

* change names

* basic work done for ask, set and help

* cancel confirmation

* cancel confirmation done, unexpected property set confirmation in progress

* improve unexpected property test

* Version 1 of contact.form.dialog

* Same info is modeled for both Home and Work.
* List of state abbreviations and their spelled-out names are specified in the separate state.en-us.lu, and used in the $mappings specifications.
* Latitude and Longitude should be auto-filled if the corresponding address are filled.
* Validation patterns are specified for zip codes and phone numbers.

* Fixed contact.form.dialog and added activity.form.dialog

* contact.form.dialog
  - Fixed spelling
  - Use phonenumber prebuilt entity type
* activity.form.dialog: the second scenario from Darren "An employee working for a customer wants to add a quick update to a customer record after a call. This update will then be available for other people working with that customer to see".

* Added opportunity.form.dialog

For Darren's scenario "An employee working with a customer has had a meeting and needs to create a new opportunity."

* rename state.lu

* prebuilt lu templates

* remove choose entity

we only have chooseProperty and clarifyEntity

* Switch to event expected.

* Update templates for latest runtime and add ask for
multiple properties.

* Add **NAMES/**NAME** and generate property names.

* fix unexpected property confirmation usage

* change unexpected property action from ask to sendactivity

* all done confirmation

* Update names to support chooseProperty.

* Check-in bread value hack.

* Fix template bug around clarify.

* Update templates to use names and start numbers.

* complete the new forms

* contextual help

* activity schema working fine

* add meta info to opportunity schema

* opportunity semi-working

cannot use phonenumber;
properties not in required also got asked;
in stringset, default value is not expression, cannot get the value for @email

* three schemas working

sandwich, activity, opportunity

* contact schema working

Adding welcome message in beginDialog,
Add help message for non-enum properties,
All three CDS schemas working

* Sync changes for demo.

* Before adding mapping templates

* Getting close to E2e.

* Largely working with some template debugging.

* Update the readme.

* Update to latest expression and lg packages.
Support app schema.

* Update standard schema.

* Finish merge.

* Show metaschema.

* Fix readme typo.

* several changes (#299)

* several changes

(1) remove unused old files;
(2) change incorrect file names;
(3) add string ask templates;
(4) remove meta properties from schemas;
(5) fix an error in calculating optional properties

* fix help property bug

* Add string templates.

* a few changes (#318)

(1) fix the cancel.lu bug
(2) add datetime.lu
(2) stringSetdatetime.dialog.lg

* Add template for not understood.

* Connect to bf cli.

* Add basic role support.

* Nigao/newtemplate (#321)

* a few changes

(1) fix the cancel.lu bug
(2) add datetime.lu
(2) stringSetdatetime.dialog.lg

* autohelp for new templates

* Update to use @{} everywhere and remove auto-help
since it required post-ask action.

* Nigao/newtemplate (#323)

* a few changes

(1) fix the cancel.lu bug
(2) add datetime.lu
(2) stringSetdatetime.dialog.lg

* autohelp for new templates

* add $requires, and datetime entity sets

* add all stringSet for prebuilt entities explicitly

* change prebuilt-entity.lu.lg files

* Switch to using a triggerIntent instead of none for entities.
Cleanup filename expressions.

* Fix bug in status.

* Switch from dialog:form to dialog:generate

* Add top-level documentation.

* Update tests to use dialog:generate.

* Rename tests to generator.

* Fixing linting errors

* Generate confirmation card.

* Fix reference regex ot not match structured lg.

Author: Chris McConnell

.gitignore

@@ -76,3 +76,10 @@ tsconfig.tsbuildinfo
 packages/dialog/test/commands/dialog/projects/obj/Debug/netcoreapp2.1/LUISGen.csproj.CoreCompileInputs.cache
 packages/dialog/test/commands/dialog/projects/obj/Debug/netcoreapp2.1/LUISGen.AssemblyInfo.cs
 packages/dialog/test/commands/dialog/projects/obj/Debug/netcoreapp2.1/LUISGen.AssemblyInfoInputs.cache
+
+#generated antlr files
+.antlr
+packages/dialog/tsconfig.tsbuildinfo
+
+# BuildInfo
+tsconfig.tsbuildinfo

packages/cli/package.json

@@ -10,6 +10,7 @@
   "dependencies": {
     "@microsoft/bf-chatdown": "1.0.0",
     "@microsoft/bf-cli-config": "1.0.0",
+    "@microsoft/bf-dialog": "1.0.0",
     "@microsoft/bf-lu": "1.0.0",
     "@microsoft/bf-qnamaker": "1.0.0",
     "@oclif/command": "~1.5.13",
@@ -62,6 +63,7 @@
       "@oclif/plugin-help",
       "@microsoft/bf-chatdown",
       "@microsoft/bf-cli-config",
+      "@microsoft/bf-dialog",
       "@microsoft/bf-qnamaker",
       "@microsoft/bf-lu"
     ],

packages/dialog/.vscode/launch.json

@@ -4,12 +4,20 @@
     // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
     "version": "0.2.0",
     "configurations": [
+        {
+            "type": "antlr-debug",
+            "request": "launch",
+            "name": "Debug Current Grammar",
+            "input": "test/commands/dialog/forms/input.txt",
+            "startRule": "doc",
+            "visualParseTree": true
+        },
         {
             "type": "node",
             "request": "launch",
             "name": "Dialog Merge Tests",
             "program": "${workspaceFolder}/node_modules/mocha/bin/_mocha",
-            "cwd":"${workspaceFolder}",
+            "cwd": "${workspaceFolder}",
             "args": [
                 "-u",
                 "tdd",
@@ -25,7 +33,7 @@
             "request": "launch",
             "name": "Dialog Verify Tests",
             "program": "${workspaceFolder}/node_modules/mocha/bin/_mocha",
-            "cwd":"${workspaceFolder}",
+            "cwd": "${workspaceFolder}",
             "sourceMaps": false,
             "args": [
                 "-u",
@@ -37,6 +45,22 @@
             ],
             "internalConsoleOptions": "openOnSessionStart"
         },
+        {
+            "type": "node",
+            "request": "launch",
+            "name": "Dialog Generator Tests",
+            "program": "${workspaceFolder}/node_modules/mocha/bin/_mocha",
+            "preLaunchTask": "tsc: build - tsconfig.json",
+            "args": [
+                "-u",
+                "tdd",
+                "--timeout",
+                "999999",
+                "--colors",
+                "${workspaceFolder}/test/commands/dialog/generator.test.ts"
+            ],
+            "internalConsoleOptions": "openOnSessionStart"
+        },
         {
             "type": "node",
             "request": "launch",
@@ -51,6 +75,90 @@
                 "-f",
                 "c:/source/github/botbuilder-dotnet/**/*.schema"
             ]
+        },
+        {
+            "type": "node",
+            "request": "launch",
+            "name": "Generate Sandwich Form",
+            "program": "${workspaceFolder}/bin/run",
+            "preLaunchTask": "tsc: build - tsconfig.json",
+            "outFiles": [
+                "${workspaceFolder}/lib/**/*.js"
+            ],
+            "args": [
+                "dialog:generate",
+                "${workspaceFolder}/test/commands/dialog/forms/sandwich.form.dialog",
+                "-o",
+                "${env:TEMP}/sandwich.out",
+                "--force",
+                "--verbose",
+                "--schema",
+                "https://raw.githubusercontent.com/microsoft/botbuilder-dotnet/chrimc/map/schemas/sdk.schema"
+            ],
+            "internalConsoleOptions": "openOnSessionStart"
+        },
+        {
+            "type": "node",
+            "request": "launch",
+            "name": "Generate Activity Form",
+            "program": "${workspaceFolder}/bin/run",
+            "preLaunchTask": "tsc: build - tsconfig.json",
+            "outFiles": [
+                "${workspaceFolder}/lib/**/*.js"
+            ],
+            "args": [
+                "dialog:form",
+                "${workspaceFolder}/test/commands/dialog/forms/activity.form.dialog",
+                "-o",
+                "${env:TEMP}/activity.out",
+                "--force",
+                "--verbose",
+                "--schema",
+                "https://raw.githubusercontent.com/microsoft/botbuilder-dotnet/chrimc/map/schemas/sdk.schema"
+            ],
+            "internalConsoleOptions": "openOnSessionStart"
+        },
+        {
+            "type": "node",
+            "request": "launch",
+            "name": "Generate Contact Form",
+            "program": "${workspaceFolder}/bin/run",
+            "preLaunchTask": "tsc: build - tsconfig.json",
+            "outFiles": [
+                "${workspaceFolder}/lib/**/*.js"
+            ],
+            "args": [
+                "dialog:form",
+                "${workspaceFolder}/test/commands/dialog/forms/contact.form.dialog",
+                "-o",
+                "${env:TEMP}/contact.out",
+                "--force",
+                "--verbose",
+                "--schema",
+                "https://raw.githubusercontent.com/microsoft/botbuilder-dotnet/chrimc/map/schemas/sdk.schema"
+            ],
+            "internalConsoleOptions": "openOnSessionStart"
+        },
+        {
+            "type": "node",
+            "request": "launch",
+            "name": "Generate Opportunity Form",
+            "program": "${workspaceFolder}/bin/run",
+            "preLaunchTask": "tsc: build - tsconfig.json",
+            "outFiles": [
+                "${workspaceFolder}/lib/**/*.js"
+            ],
+            "args": [
+                "dialog:form",
+                "${workspaceFolder}/test/commands/dialog/forms/opportunity.form.dialog",
+                "-o",
+                "${env:TEMP}/opportunity.out",
+                "--force",
+                "--verbose",
+                "--schema",
+                "https://raw.githubusercontent.com/microsoft/botbuilder-dotnet/chrimc/map/schemas/sdk.schema"
+            ],
+            "internalConsoleOptions": "openOnSessionStart"
         }
     ]
 }

packages/dialog/bad-schema.form.dialog

@@ -0,0 +1 @@
+undefined

packages/dialog/package.json

@@ -5,18 +5,22 @@
   "bugs": "https://github.com/microsoft/botframework-cli/issues",
   "dependencies": {
     "@microsoft/bf-cli-command": "~1.0.0",
+    "@types/lru-cache": "^5.1.0",
     "@types/semver": "^6.0.1",
     "@types/xml2js": "^0.4.4",
     "ajv": "^6.9.1",
+    "botframework-expressions": "https://botbuilder.myget.org/F/botbuilder-declarative/npm/botframework-expressions/-/4.7.0-preview0.tgz",
+    "botbuilder-lg": "https://botbuilder.myget.org/F/botbuilder-declarative/npm/botbuilder-lg/-/4.7.0-preview0.tgz",
     "chalk": "^2.4.2",
+    "clone": "^2.1.2",
     "fs-extra": "^8.1.0",
     "get-json": "^1.0.1",
     "globby": "^9.2.0",
     "json-schema-merge-allof": "^0.6.0",
     "json-schema-ref-parser": "^7.1.0",
-    "tslib": "^1.10.0",
-    "clone": "^2.1.2",
+    "path": "^0.12.7",
     "semver": "^5.7.0",
+    "tslib": "^1.10.0",
     "xml2js": "^0.4.19"
   },
   "devDependencies": {

packages/dialog/src/commands/dialog/generate.ts

@@ -0,0 +1,60 @@
+/*!
+ * Copyright (c) Microsoft Corporation. All rights reserved.
+ * Licensed under the MIT License.
+ */
+
+import { Command, flags } from '@microsoft/bf-cli-command';
+import * as gen from '../../library/dialogGenerator'
+import * as ppath from 'path'
+
+export default class GenerateDialog extends Command {
+    static description = '[PREVIEW] Generate localized .lu, .lg, .qna and .dialog assets to define a bot based on a schema using templates.'
+
+    static examples = [`
+      $ bf dialog:generate sandwich.schema --output c:/tmp
+    `]
+
+    static args = [
+        { name: 'schema', required: true, description: 'JSON Schema file used to drive generation.' }
+    ]
+
+    static flags: flags.Input<any> = {
+        force: flags.boolean({ char: 'f', description: 'Force overwriting generated files.' }),
+        help: flags.help({ char: 'h' }),
+        locale: flags.string({ char: 'l', description: 'Locales to generate. [default: en-us]', multiple: true }),
+        output: flags.string({ char: 'o', description: 'Output path for where to put generated .lu, .lg, .qna and .dialog files. [default: ./<form>-resources]', default: '.', required: false }),
+        schema: flags.string({ char: 's', description: 'Path to your app.schema file.', required: false }),
+        templates: flags.string({ char: 't', description: 'Directory with templates to use for generating form assets.', multiple: true }),
+        verbose: flags.boolean({ description: 'Output verbose logging of files as they are processed', default: false }),
+    }
+
+    async run() {
+        const { args, flags } = this.parse(GenerateDialog)
+        try {
+            let formName = ppath.basename(args.schema, '.schema.dialog')
+            let outDir = flags.output
+            if (!outDir) {
+                outDir = ppath.join(formName + '-resources')
+            }
+            await gen.generate(args.schema, outDir, flags.schema, flags.locale, flags.templates, flags.force,
+                (type, msg) => {
+                    if (type === gen.FeedbackType.message
+                        || type === gen.FeedbackType.error
+                        || (type === gen.FeedbackType.info && flags.verbose)) {
+                        this.progress(msg)
+                    }
+                })
+            return true;
+        } catch (e) {
+            this.thrownError(e)
+        }
+    }
+
+    thrownError(err: Error): void {
+        this.error(err.message)
+    }
+
+    progress(msg: string): void {
+        this.error(msg)
+    }
+}

packages/dialog/src/commands/dialog/merge.ts

@@ -6,14 +6,14 @@
 import { Command, flags } from '@microsoft/bf-cli-command';
 import * as Validator from 'ajv';
 import * as fs from 'fs-extra';
-import * as getJson from 'get-json';
 import * as glob from 'globby';
 import * as os from 'os';
 import * as ppath from 'path';
 import * as semver from 'semver';
 import * as xp from 'xml2js';
 let allof: any = require('json-schema-merge-allof')
 let clone = require('clone')
+let getJson = require('get-json')
 let parser: any = require('json-schema-ref-parser')
 let util: any = require('util')
 let exec: any = util.promisify(require('child_process').exec)
@@ -40,7 +40,7 @@ export default class DialogMerge extends Command {
         verbose: flags.boolean({ description: 'output verbose logging of files as they are processed', default: false }),
     }
 
-    private verbose = false
+    private verbose? = false
     private failed = false
     private missingTypes = new Set()
     private currentFile = ''

packages/dialog/src/commands/dialog/readme.md

@@ -0,0 +1,59 @@
+# Dialog Commands
+
+# Merge
+
+# Verify
+
+# Form
+The form command generates .lu, .lg, .qna and .dialog assets from a schema defined using JSON Schema.  The parameters to the command are:
+* **--force, -f** Force overwriting generated files.
+* **--help, -h** Generate help.
+* **--locale, -l** Locales to generate.  By default en-us.
+* **--output, -o** Output directory.
+* **--schema, -s** Path to your app.schema file. By default is the standard SDK app.schema.
+* **--templates, -t** Directories with templates to use for generating form assets.
+* **--verbose, -v** Verbose logging of generated files.
+
+## Schema
+Schemas are specified using JSON Schema.  You can use the normal JSON Schema mechanisms including $ref and allOf which will be resolved into a single schema.  In addition there are a few extra keywords including:
+* **$mappings** List of entity names that can map to a property. The order of the entities also defines the precedence to use when resolving entities.  By default the mappings are based on the type:
+  * **enum**
+  * **number**, **string**
+* **$templates** The template names to use for generating assets. $templates can be defined at the top-level in a schema or per-property which by default are based on the type:
+  * **enum**
+  * **number**, **string**
+* **\$expectedOnly** A list of properties that are only possible if they are expected.
+* **\$requires** A list of JSON Schema to use for internal mechanisms.  You can use either actual paths or just the name of the schema to use if found in one of the template directories.  The standard schema is `standard.schema.dialog`.  The form schema and all of the required schemas will have the top-level `properties`, `definitions`, `required`, `$expectedOnly` and `$templates` merged.
+* **\$triggerIntent** Name of the trigger intent or by default the name of the form.
+
+`<form>.form.dialog` will be generated with the form schema in it.  `<form>.schema.dialog` will have the whole schema defined.
+
+## Templates
+Each entity or property can have associated .lu, .lg, .qna and .dialog files that are generated by 
+copying or instantiating templates found in the template directories.  If a template name matches exactly it is
+just copied.  If the template ends with .lg then it is analyzed to see if it has a template named 'template' and optionally one named 'filename'.  If 'filename' is specified, then the filename will be the result of generating generated file is the result of evaluating that template, otherwise it defaults to `<formName>-<templateName>[.<locale>].<extension>`.  When evaluating templates there are a number of variables defined in the scope including:
+* **formName** The name of the form being generated.
+* **appSchema** The path to the app.schema to use. 
+* **form** The JSON Schema defining the form.
+* **schema** The JSON Schema of the form + internal properties.
+* **locales** The list of all locales being generated.
+* **properties** All of the form property names.
+* **entities** All of the types of schema entities being used.
+* **triggerIntent** $triggerIntent or the form name by default.
+* **locale** The locale being generated or empty if no locale.
+* **property** For per-property templates this the property name being generated.
+* **templates** Object with generated templates per lu, lg, qna, json and dialog.  The object contains:
+  * **name** Base name of the template without final extension.
+  * **fullName** The name of the template including the extension.
+  * **relative** Path relative to the output directory of where template is.
+
+Templates are generated in the following order:
+* Generate per-locale language resources
+  * Per-entity generate .lg, .lu, .qna files
+  * Per-property
+    * Per-template in `<property>.$templates` generate .lg, .lu, .qna files.
+  * Per-template in `$templates` generate .lg, .lu, .qna files.
+* Generate non language resources
+  * Per-property
+    * Per-template in `<property>.$templates` generate .dialog and .json files.
+  * Per-template in `$templates` generate .dialog and .json files.

packages/dialog/src/commands/dialog/verify.ts

@@ -28,7 +28,7 @@ export default class DialogVerify extends Command {
         verbose: flags.boolean({ description: 'Show verbose output', default: false }),
     }
 
-    private currentFile: string
+    private currentFile = ''
     private files = 0
     private errors = 0
     private warnings = 0