perf(cloud9): "Upload Lambda" is slow since Toolkit 2.x #4443

Problem:
In Cloud9, using the "Upload Lambda" action on even a small directory,
takes 5+ seconds. Example trace:

    2024-02-16 15:45:44 [DEBUG]: createRegionPrompter: selected { id: 'us-west-2', name: 'US West (Oregon)' }
    2024-02-16 15:45:44 [DEBUG]: findApplicationJsonFile (cloud9: true)
    2024-02-16 15:45:44 [DEBUG]: cloud9Findfile: /home/ec2-user/environment/
    2024-02-16 15:45:46 [DEBUG]: cloud9Findfile: /home/ec2-user/environment/.c9
    2024-02-16 15:45:49 [DEBUG]: cloud9Findfile: /home/ec2-user/environment/.c9/amazonwebservices.aws-toolkit-vscode
    2024-02-16 15:45:50 [DEBUG]: cloud9Findfile: /home/ec2-user/environment/.c9/metadata
    2024-02-16 15:45:52 [DEBUG]: cloud9Findfile: /home/ec2-user/environment/.c9/test 4
    2024-02-16 15:45:52 [DEBUG]: uploadLambda: .application.json not found in: "/home/ec2-user/environment/"

Solution:
The vscode `stat()` API is very slow, especially on the Cloud9 VFS.
Avoid it by special-casing the implementation of `fsCommon.exists()` on
Cloud9.

    2024-02-16 16:12:14 [DEBUG]: createRegionPrompter: selected { id: 'us-west-2', name: 'US West (Oregon)' }
    2024-02-16 16:12:14 [DEBUG]: findApplicationJsonFile (cloud9: true)
    2024-02-16 16:12:14 [DEBUG]: cloud9Findfile: /home/ec2-user/environment/
    2024-02-16 16:12:14 [DEBUG]: cloud9Findfile: /home/ec2-user/environment/.c9
    2024-02-16 16:12:14 [DEBUG]: cloud9Findfile: /home/ec2-user/environment/.c9/amazonwebservices.aws-toolkit-vscode
    2024-02-16 16:12:14 [DEBUG]: cloud9Findfile: /home/ec2-user/environment/.c9/metadata
    2024-02-16 16:12:14 [DEBUG]: cloud9Findfile: /home/ec2-user/environment/.c9/test 4
    2024-02-16 16:12:14 [DEBUG]: uploadLambda: .application.json not found in: "/home/ec2-user/environment/"

TODO: the following functions which loop through `readdir` results are
also potentially slow:

    getDirSize
    cleanLogFiles

Author: justinmk3

docs/ARCHITECTURE.md

@@ -337,7 +337,14 @@ Abstractly, a 'wizard' is a collection of discrete, linear steps (subroutines),
 
 ### Creating a Wizard (Quick Picks)
 
-A new wizard can be created by extending off the base `Wizard` class, using the template type to specify the shape of the wizard state. All wizards have an internal 'form' property that is used to assign steps. We can assign UI elements (namely, quick picks) using the `bindPrompter` method on form elements. This method accepts a callback that should return a `Prompter` given the current state. For this example, we will use `createQuickPick` and `createInputBox` for our prompters:
+Create a new wizard by extending the base `Wizard` class, using the template type to specify the
+shape of the wizard state. All wizards have an internal `form` property that is used to assign
+steps. You can assign UI elements (namely, quickpicks) using the `bindPrompter` method on form
+elements. This method accepts a callback that should return a `Prompter` given the current state.
+For this example, we will use `createQuickPick` and `createInputBox` for our prompters:
+
+If you need to call async functions to construct your `Wizard` subclass, define your init logic in
+the `init()` method instead of the constructor.
 
 ```ts
 interface ExampleState {
@@ -382,7 +389,7 @@ Note that all wizards can potentially return `undefined` if the workflow was can
 Use `createWizardTester` on an instance of a wizard. Tests can then be constructed by asserting both the user-defined and internal state. Using the above `ExampleWizard`:
 
 ```ts
-const tester = createWizardTester(new ExampleWizard())
+const tester = await createWizardTester(new ExampleWizard())
 tester.foo.assertShowFirst() // Fails if `foo` is not shown (or not shown first)
 tester.bar.assertDoesNotShow() // True since `foo` is not assigned an explicit value
 tester.foo.applyInput('Hello, world!') // Manipulate 'user' state

packages/toolkit/src/lambda/commands/uploadLambda.ts

@@ -9,15 +9,10 @@ import * as nls from 'vscode-nls'
 const localize = nls.loadMessageBundle()
 
 import AdmZip from 'adm-zip'
-import * as fs from 'fs-extra'
 import * as path from 'path'
+import { fsCommon } from '../../srcShared/fs'
 import { showConfirmationMessage, showViewLogsMessage } from '../../shared/utilities/messages'
-import {
-    fileExists,
-    cloud9Findfile,
-    makeTemporaryToolkitFolder,
-    tryRemoveFolder,
-} from '../../shared/filesystemUtilities'
+import { cloud9Findfile, makeTemporaryToolkitFolder, tryRemoveFolder } from '../../shared/filesystemUtilities'
 import * as localizedText from '../../shared/localizedText'
 import { getLogger } from '../../shared/logger'
 import { SamCliBuildInvocation } from '../../shared/sam/cli/samCliBuild'
@@ -209,16 +204,15 @@ export interface UploadLambdaWizardState {
 }
 
 export class UploadLambdaWizard extends Wizard<UploadLambdaWizardState> {
-    constructor(lambda?: LambdaFunction, invokePath?: vscode.Uri) {
-        super({ initState: { lambda } })
+    public override async init(): Promise<this> {
         this.form.lambda.region.bindPrompter(() => createRegionPrompter().transform(region => region.id))
 
-        if (invokePath) {
+        if (this.invokePath) {
             this.form.uploadType.setDefault('directory')
-            if (fs.statSync(invokePath.fsPath).isFile()) {
-                this.form.targetUri.setDefault(vscode.Uri.file(path.dirname(invokePath.fsPath)))
+            if (await fsCommon.existsFile(this.invokePath.fsPath)) {
+                this.form.targetUri.setDefault(vscode.Uri.file(path.dirname(this.invokePath.fsPath)))
             } else {
-                this.form.targetUri.setDefault(invokePath)
+                this.form.targetUri.setDefault(this.invokePath)
             }
         } else {
             this.form.uploadType.bindPrompter(() => createUploadTypePrompter())
@@ -242,7 +236,7 @@ export class UploadLambdaWizard extends Wizard<UploadLambdaWizardState> {
 
         this.form.lambda.name.bindPrompter(state => {
             // invoking from the command palette passes no arguments
-            if (!invokePath) {
+            if (!this.invokePath) {
                 if (state.uploadType === 'directory') {
                     return createFunctionNamePrompter(state.lambda!.region!, state.targetUri)
                 } else {
@@ -253,10 +247,10 @@ export class UploadLambdaWizard extends Wizard<UploadLambdaWizardState> {
                     )
                 }
             } else {
-                return createFunctionNamePrompter(state.lambda!.region!, invokePath)
+                return createFunctionNamePrompter(state.lambda!.region!, this.invokePath)
             }
         })
-        if (lambda) {
+        if (this.lambda) {
             this.form.directoryBuildType.bindPrompter(() => createBuildPrompter(), {
                 showWhen: ({ uploadType }) => uploadType === 'directory',
             })
@@ -265,6 +259,12 @@ export class UploadLambdaWizard extends Wizard<UploadLambdaWizardState> {
         }
 
         this.form.confirmedDeploy.bindPrompter(state => createConfirmDeploymentPrompter(state.lambda!))
+
+        return this
+    }
+
+    constructor(readonly lambda?: LambdaFunction, readonly invokePath?: vscode.Uri) {
+        super({ initState: { lambda } })
     }
 }
 
@@ -304,7 +304,7 @@ async function runUploadLambdaWithSamBuild(lambda: Required<LambdaFunction>, par
     // Detect if handler is present and provide strong guidance against proceeding if not.
     try {
         const handlerFile = path.join(parentDir.fsPath, getLambdaDetails(lambda.configuration).fileName)
-        if (!(await fileExists(handlerFile))) {
+        if (!(await fsCommon.exists(handlerFile))) {
             const isConfirmed = await showConfirmationMessage({
                 prompt: localize(
                     'AWS.lambda.upload.handlerNotFound',
@@ -416,7 +416,7 @@ async function runUploadLambdaZipFile(lambda: LambdaFunction, zipFileUri: vscode
             cancellable: false,
         },
         async progress => {
-            const zipFile = await fs.readFile(zipFileUri.fsPath).catch(err => {
+            const zipFile = await fsCommon.readFile(zipFileUri.fsPath).catch(err => {
                 throw new ToolkitError('Failed to read zip', { cause: err })
             })
             return await uploadZipBuffer(lambda, zipFile, progress)
@@ -459,7 +459,7 @@ async function zipAndUploadDirectory(
  */
 async function uploadZipBuffer(
     lambda: LambdaFunction,
-    zip: Buffer,
+    zip: Uint8Array,
     progress: vscode.Progress<{
         message?: string | undefined
         increment?: number | undefined
@@ -482,14 +482,14 @@ export async function findApplicationJsonFile(
     startPath: vscode.Uri,
     cloud9 = isCloud9()
 ): Promise<vscode.Uri | undefined> {
-    if (!(await fileExists(startPath.fsPath))) {
+    if (!(await fsCommon.exists(startPath.fsPath))) {
         getLogger().error(
             'findApplicationJsonFile() invalid path (not accessible or does not exist): "%s"',
             startPath.fsPath
         )
         return undefined
     }
-    const isdir = fs.statSync(startPath.fsPath).isDirectory()
+    const isdir = await fsCommon.existsDir(startPath.fsPath)
     const parentDir = isdir ? startPath.fsPath : path.dirname(startPath.fsPath)
     const found = cloud9
         ? await cloud9Findfile(parentDir, '.application.json')
@@ -508,10 +508,10 @@ export async function findApplicationJsonFile(
     return found[0]
 }
 
-export function getFunctionNames(file: vscode.Uri, region: string): string[] | undefined {
+export async function getFunctionNames(file: vscode.Uri, region: string): Promise<string[] | undefined> {
     try {
         const names: string[] = []
-        const appData = JSON.parse(fs.readFileSync(file.fsPath, { encoding: 'utf-8' }).toString())
+        const appData = JSON.parse(await fsCommon.readFileAsString(file.fsPath))
         if (appData['Functions']) {
             const functions = Object.keys(appData['Functions'])
             if (functions) {
@@ -536,7 +536,7 @@ async function listAllLambdaNames(region: string, path?: vscode.Uri) {
     // Get Lambda functions from .application.json #2588
     if (path) {
         const appFile = await findApplicationJsonFile(path)
-        const namesFromAppFile = appFile ? getFunctionNames(appFile, region) : undefined
+        const namesFromAppFile = appFile ? await getFunctionNames(appFile, region) : undefined
         if (!appFile) {
             getLogger().debug('lambda: .application.json not found')
         } else if (!namesFromAppFile) {

packages/toolkit/src/shared/clients/lambdaClient.ts

@@ -98,7 +98,7 @@ export class DefaultLambdaClient {
         }
     }
 
-    public async updateFunctionCode(name: string, zipFile: Buffer): Promise<Lambda.FunctionConfiguration> {
+    public async updateFunctionCode(name: string, zipFile: Uint8Array): Promise<Lambda.FunctionConfiguration> {
         getLogger().debug(`updateFunctionCode called for function: ${name}`)
         const client = await this.createSdkClient()
 

packages/toolkit/src/shared/filesystemUtilities.ts

@@ -42,6 +42,7 @@ export async function getDirSize(
             return getDirSize(filePath, startTime, duration, fileExt)
         }
         if (type === vscode.FileType.File && fileName.endsWith(fileExt)) {
+            // TODO: This is SLOW on Cloud9.
             const stat = await fsCommon.stat(filePath)
             return stat.size
         }
@@ -271,6 +272,7 @@ export async function hasFileWithSuffix(dir: string, suffix: string, exclude?: v
  * @returns  List of one or zero Uris (for compat with vscode.workspace.findFiles())
  */
 export async function cloud9Findfile(dir: string, fileName: string): Promise<vscode.Uri[]> {
+    getLogger().debug('cloud9Findfile: %s', dir)
     const files = await fsCommon.readdir(dir)
     const subDirs: vscode.Uri[] = []
     for (const file of files) {

packages/toolkit/src/shared/wizards/wizard.ts

@@ -64,6 +64,7 @@ type StepCache = { picked?: any; stepOffset?: [number, number] } & { [key: strin
 export type StateWithCache<TState, TProp> = TState & { stepCache: StepCache; estimator: StepEstimator<TProp> }
 
 export interface WizardOptions<TState> {
+    /** Optional {@link WizardForm} provided by the caller instead of the default. */
     readonly initForm?: WizardForm<TState>
     readonly initState?: Partial<TState>
     /** Provides a way to apply inputs to Prompters as if the user has already responded */
@@ -75,13 +76,26 @@ export interface WizardOptions<TState> {
  * A generic wizard that consumes data from a series of {@link Prompter prompters}. The 'form' public property
  * exposes functionality to add prompters to the wizard with optional context, utilizing the {@link WizardForm}
  * class. Wizards will modify a single property of their internal state with each prompt.
+ *
+ * Subclasses can override {@link init} if an "async constructor" is needed.
  */
 export class Wizard<TState extends Partial<Record<keyof TState, unknown>>> {
-    protected readonly _form: WizardForm<TState>
+    protected readonly __form: WizardForm<TState>
     protected readonly boundSteps: Map<string, StepFunction<TState>> = new Map()
     protected stateController: StateMachineController<TState>
     private _stepOffset: [number, number] = [0, 0]
     private _exitStep?: StepFunction<TState>
+    /** Guards against accidental use of the Wizard before `init()`. */
+    private _ready: boolean
+
+    /** Checks that `init()` was performed (if it was defined). */
+    private assertReady() {
+        // Check for `false` explicity so that the base-class constructor can access `this._form`.
+        // We want to guard against confusion when implementing a subclass, not this base-class.
+        if (this._ready === false && this.init) {
+            throw Error('run() (or init()) must be called immediately after creating the Wizard')
+        }
+    }
 
     /**
      * The offset is applied to both the current step and total number of steps. Useful if the wizard is
@@ -97,6 +111,11 @@ export class Wizard<TState extends Partial<Record<keyof TState, unknown>>> {
         return this._stepOffset[1] + this.stateController.totalSteps
     }
 
+    public get _form() {
+        this.assertReady()
+        return this.__form
+    }
+
     public get form() {
         return this._form.body
     }
@@ -117,11 +136,21 @@ export class Wizard<TState extends Partial<Record<keyof TState, unknown>>> {
 
     public constructor(private readonly options: WizardOptions<TState> = {}) {
         this.stateController = new StateMachineController(options.initState as TState)
-        this._form = options.initForm ?? new WizardForm()
+        this.__form = options.initForm ?? new WizardForm()
         this._exitStep =
             options.exitPrompterProvider !== undefined ? this.createExitStep(options.exitPrompterProvider) : undefined
+
+        // Subclass constructor logic should live in `init()`, if it exists.
+        this._ready = !this.init
     }
 
+    /**
+     * Optional, one-time, asynchronous setup for subclasses that need an "async constructor". The
+     * subclass must put all of its constructor logic here, instead of its normal constructor.
+     * Called by `run()` exactly once.
+     */
+    public async init?(): Promise<this>
+
     private assignSteps(): void {
         this._form.properties.forEach(prop => {
             const provider = this._form.getPrompterProvider(prop)
@@ -132,6 +161,12 @@ export class Wizard<TState extends Partial<Record<keyof TState, unknown>>> {
     }
 
     public async run(): Promise<TState | undefined> {
+        if (!this._ready && this.init) {
+            this._ready = true // Let init() use `this._form`.
+            await this.init()
+            delete this.init
+        }
+
         this.assignSteps()
         this.resolveNextSteps((this.options.initState ?? {}) as TState).forEach(step =>
             this.stateController.addStep(step)

packages/toolkit/src/srcShared/fs.ts

@@ -82,6 +82,25 @@ export class FileSystemCommon {
 
     async exists(path: Uri | string, fileType?: vscode.FileType): Promise<boolean> {
         path = FileSystemCommon.getUri(path)
+
+        if (isCloud9()) {
+            // vscode.workspace.fs.stat() is SLOW. Avoid it on Cloud9.
+            try {
+                const stat = await fsPromises.stat(path.fsPath)
+                // Note: comparison is bitwise (&) because `FileType` enum is bitwise.
+                // See vscode.FileType docstring.
+                if (fileType === undefined || fileType & vscode.FileType.Unknown) {
+                    return true
+                } else if (fileType & vscode.FileType.Directory) {
+                    return stat.isDirectory()
+                } else if (fileType & vscode.FileType.File) {
+                    return stat.isFile()
+                }
+            } catch {
+                return false
+            }
+        }
+
         try {
             const stat = await this.stat(path)
             // check filetype if it was given

packages/toolkit/src/test/apprunner/wizards/codeRepositoryWizard.test.ts

@@ -22,10 +22,10 @@ describe('AppRunnerCodeRepositoryWizard', function () {
     let tester: WizardTester<AppRunner.SourceConfiguration>
     let repoTester: WizardTester<AppRunner.CodeRepository>
 
-    beforeEach(function () {
+    beforeEach(async function () {
         // apprunner client and git api will never be called
         const wizard = new AppRunnerCodeRepositoryWizard({} as any, {} as any)
-        tester = createWizardTester(wizard)
+        tester = await createWizardTester(wizard)
         repoTester = tester.CodeRepository
     })
 

packages/toolkit/src/test/apprunner/wizards/createServiceWizard.test.ts

@@ -14,7 +14,7 @@ import { DefaultAppRunnerClient } from '../../../shared/clients/apprunnerClient'
 describe('CreateServiceWizard', function () {
     let tester: WizardTester<AppRunner.CreateServiceRequest>
 
-    beforeEach(function () {
+    beforeEach(async function () {
         const regionCode = 'us-east-1'
         const wizard = new CreateAppRunnerServiceWizard(
             regionCode,
@@ -27,7 +27,7 @@ describe('CreateServiceWizard', function () {
             }
         )
 
-        tester = createWizardTester(wizard)
+        tester = await createWizardTester(wizard)
     })
 
     describe('CreateAppRunnerServiceWizard', function () {

packages/toolkit/src/test/apprunner/wizards/imageRepositoryWizard.test.ts

@@ -15,9 +15,9 @@ describe('AppRunnerImageRepositoryWizard', function () {
     let tester: WizardTester<AppRunner.SourceConfiguration>
     let repoTester: WizardTester<AppRunner.ImageRepository>
 
-    beforeEach(function () {
+    beforeEach(async function () {
         const wizard = new AppRunnerImageRepositoryWizard({} as any, {} as any) // the clients will never be called
-        tester = createWizardTester(wizard)
+        tester = await createWizardTester(wizard)
         repoTester = tester.ImageRepository
     })
 
@@ -52,9 +52,9 @@ describe('AppRunnerImageRepositoryWizard', function () {
 describe('ImageIdentifierForm', function () {
     let tester: WizardTester<{ repo: TaggedEcrRepository }>
 
-    beforeEach(function () {
+    beforeEach(async function () {
         const form = new ImageIdentifierForm({} as any) // ecr will never be called
-        tester = createWizardTester(form)
+        tester = await createWizardTester(form)
     })
 
     it('asks for tag if not provided', function () {

packages/toolkit/src/test/cloudWatchLogs/commands/searchLogGroup.test.ts

@@ -13,15 +13,15 @@ import { createWizardTester } from '../../shared/wizards/wizardTestUtils'
 
 describe('searchLogGroup', async function () {
     describe('Wizard', async function () {
-        it('shows steps in correct order', function () {
-            const testWizard = createWizardTester(new SearchLogGroupWizard())
+        it('shows steps in correct order', async function () {
+            const testWizard = await createWizardTester(new SearchLogGroupWizard())
             testWizard.submenuResponse.assertShowFirst()
             testWizard.timeRange.assertShowSecond()
             testWizard.filterPattern.assertShowThird()
         })
 
         it('skips steps if parameters are given', async function () {
-            const nodeTestWizard = createWizardTester(
+            const nodeTestWizard = await createWizardTester(
                 new SearchLogGroupWizard({
                     groupName: 'group-test',
                     regionName: 'region-test',