inital commit of computer use

Author: jawhnycooke

computer-use/.gitignore

@@ -0,0 +1,34 @@
+*.swp
+package-lock.json
+.pytest_cache
+*.egg-info
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# CDK Context & Staging files
+.cdk.staging/
+cdk.out/
+cdk*.json
+cdk.*.json
+cdk.context.json
+# IDE specific files
+.idea/
+.vscode/
+*.sublime-workspace
+*.sublime-project
+
+# OS specific files
+.DS_Store
+Thumbs.db

computer-use/README.md

@@ -0,0 +1,176 @@
+# Computer Use AWS Infrastructure
+
+This project contains the AWS CDK infrastructure code for deploying the Computer Use AWS application in the us-west-2 (Oregon) region. The infrastructure includes ECS Fargate services, ECR repositories, and all necessary networking components.
+
+## Project Structure
+
+```
+ComputerUseAWS/
+├── README.md
+├── app.py
+├── cdk.json
+├── computer_use_aws_stack.py
+├── requirements.txt
+├── scripts/
+│   └── get_urls.sh
+├── computer_use_aws/
+│   ├── environment_image/
+│   │   ├── computer_use_demo/
+│   │   ├── image/
+│   │   ├── Dockerfile
+│   │   └── requirements.txt
+│   └── orchestration_image/
+│       ├── computer_use_demo/
+│       ├── Dockerfile
+│       └── requirements.txt
+└── tests/
+    ├── integration/
+    └── unit/
+```
+
+## Prerequisites
+
+1. AWS CLI installed and configured with us-west-2 region
+2. Python 3.7 or later
+3. Node.js 14.x or later (required for CDK)
+4. Docker installed and running
+5. AWS CDK CLI installed (`npm install -g aws-cdk`)
+
+## Quick Start
+1. Clone the repository:
+```bash
+git clone git@ssh.gitlab.aws.dev:jonaevau/ComputerUseAWS.git
+cd ComputerUseAWS
+```
+
+2. Make the gets_urls.sh cript executable:
+```bash
+chmod +x scripts/gets_urls.sh
+```
+
+3. Configure AWS CLI for us-west-2 (if not already configured):
+```bash
+aws configure set default.region us-west-2
+```
+
+4. Create virtual environment:
+```bash
+python -m venv .venv
+source .venv/bin/activate  # On Windows use: .venv\Scripts\activate
+```
+
+5. Install the required dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+6. Bootstrap CDK in us-west-2 (if you haven't already):
+```bash
+cdk bootstrap aws://ACCOUNT-NUMBER/us-west-2
+```
+
+7. Deploy the solution in **Fail-Secure Mode** (Default). In this mode, if no IP address is provided, the security groups will default to a highly restrictive setting (255.255.255.255/32) that effectively blocks all access. This is the recommended setting for the sandbox environment.
+
+```bash
+# Deploy with your current IP (Fail-Secure)
+cdk deploy --context deployer_ip=$(curl -s https://api.ipify.org)
+
+# Deploy with manual IP (Fail-Secure)
+cdk deploy --context deployer_ip=203.0.113.1
+# This will automatically be converted to 203.0.113.1/32
+
+# Deploy with IP address range (Fail-Secure)
+cdk deploy --context deployer_ip=203.0.113.0/24 
+# Allows 203.0.113.0 through 203.0.113.255
+```
+
+**Note**: This stack takes ~10-15 minutes to deploy. After the deployment it may take a few additional minutes for the Environment/Virtual Machine to come online
+
+8. After the deploy has completed, you can cet the URLs of the services:
+```bash
+./scripts/get_urls.sh
+```
+
+**Note**:If you are using aws profiles append --profile <profile name> to the command line if not provided "default" is assumed
+```bash
+./scripts/get_urls.sh --profile <your profile name>
+```
+
+
+## Usage
+
+Navigate to the links provided in the output of the `get_urls.sh` script to access the services, the Orchestration Service URL and Environment Service URL.
+
+The DCV **username** is `computeruse` and the **password** is `admin`. DCV is used to connect to the environment container for remote desktop access, for activities such as resetting the state of the environment, or elliciting the state of the environment prior to a new task.
+
+The Streamlit interface is used to configure the API provider, model, and other parameters for the environment container. Then ultimately instruct the model via a chat interface to perform tasks.
+
+## GUI Access
+
+After navigating to the Orchestration Service URL (Streamlit interface), you'll need to log in:
+
+1. Default credentials:
+   - **Username:** `admin`
+   - **Password:** `computeruse`
+
+2. After successful login, you'll see:
+   - Configuration panel in the sidebar
+   - Chat interface in the main area
+   - HTTP Exchange Logs tab for debugging
+
+Note: The session will timeout after 60 minutes of inactivity, requiring you to log in again.
+
+## Infrastructure Components
+
+- **VPC**: Configured with public and private subnets across 2 AZs in us-west-2
+- **ECR Repository**: Single repository for both environment and orchestration images
+- **ECS Cluster**: Fargate cluster for running containers
+- **Task Definition**: Includes both containers with appropriate port mappings
+- **Security Groups**: 
+  - Environment container: Accepts traffic only from orchestration container
+  - Orchestration container: Accepts public traffic on port 8501
+- **IAM Roles**: Task execution role with minimal permissions
+- **CloudWatch Logs**: Configured for container logging
+- **KMS**: Encryption key for secure storage
+
+## Container Ports
+
+- **Environment Container**:
+  - 8443: DCV
+  - 5000: Flask Control API
+- **Orchestration Container**:
+  - 8501: Streamlit interface
+
+## Monitoring
+
+- Container insights enabled for the ECS cluster
+- CloudWatch logs configured with KMS encryption
+- VPC flow logs enabled for network monitoring
+- All logs retained for one month
+
+## Troubleshooting
+
+1. If deployment fails:
+```bash
+# Check AWS credentials
+aws sts get-caller-identity
+
+# Check CloudFormation events
+aws cloudformation describe-stack-events --stack-name ComputerUseAwsStack
+```
+
+2. If containers fail to start:
+```bash
+# Check ECS service events
+aws ecs describe-services --cluster computer-use-aws-cluster --services computer-use-aws-service-computeruseawsstack
+
+# Check container logs
+aws logs get-log-events --log-group-name /ecs/computer-use-aws-computeruseawsstack
+```
+
+## Clean Up
+
+To destroy the infrastructure:
+```bash
+cdk destroy
+```

computer-use/app.py

@@ -0,0 +1,14 @@
+#!/usr/bin/env python3
+import os
+import aws_cdk as cdk
+from computer_use_aws_stack import ComputerUseAwsStack
+
+app = cdk.App()
+ComputerUseAwsStack(app, "ComputerUseAwsStack",
+    env=cdk.Environment(
+        account=os.getenv('CDK_DEFAULT_ACCOUNT'),
+        region='us-west-2'
+    ),
+)
+
+app.synth()

computer-use/computer_use_aws/__init__.py

@@ -0,0 +1,134 @@
+FROM public.ecr.aws/ubuntu/ubuntu:22.04
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV DEBIAN_PRIORITY=high
+
+RUN apt-get update && \
+    apt-get -y upgrade && \
+    apt-get -y install \
+    build-essential \
+    # UI Requirements
+    xvfb \
+    xterm \
+    dbus \
+    dbus-x11 \
+    x11-xserver-utils \
+    xdotool \
+    scrot \
+    imagemagick \
+    sudo \
+    mutter \
+    x11vnc \
+    # Python/pyenv reqs
+    build-essential \
+    libssl-dev  \
+    zlib1g-dev \
+    libbz2-dev \
+    libreadline-dev \
+    libsqlite3-dev \
+    curl \
+    git \
+    libncursesw5-dev \
+    xz-utils \
+    tk-dev \
+    libxml2-dev \
+    libxmlsec1-dev \
+    libffi-dev \
+    liblzma-dev \
+    # Network tools
+    net-tools \
+    netcat \
+    # PPA req
+    software-properties-common && \
+    # Userland apps
+    sudo add-apt-repository ppa:mozillateam/ppa && \
+    sudo apt-get install -y --no-install-recommends \
+    libreoffice \
+    firefox-esr \
+    x11-apps \
+    xpdf \
+    gedit \
+    xpaint \
+    tint2 \
+    galculator \
+    pcmanfm \
+    unzip && \
+    apt-get clean
+
+RUN apt-get install -y wget
+
+# Install Amazon DCV
+RUN wget https://d1uj6qtbmh3dt5.cloudfront.net/NICE-GPG-KEY && \
+    gpg --import NICE-GPG-KEY && \
+    wget https://d1uj6qtbmh3dt5.cloudfront.net/2024.0/Servers/nice-dcv-2024.0-17979-ubuntu2204-aarch64.tgz && \
+    tar -xvzf nice-dcv-2024.0-17979-ubuntu2204-aarch64.tgz && \
+    cd nice-dcv-2024.0-17979-ubuntu2204-aarch64 && \
+    apt install -y ./nice-dcv-server_2024.0.17979-1_arm64.ubuntu2204.deb && \
+    apt install -y ./nice-dcv-web-viewer_2024.0.17979-1_arm64.ubuntu2204.deb && \
+    cd .. && \
+    rm -rf nice-dcv-2024.0-17979-ubuntu2204-aarch64*
+
+# setup user
+ENV USERNAME=computeruse
+ENV HOME=/home/$USERNAME
+RUN useradd -m -s /bin/bash -d $HOME $USERNAME
+RUN echo "${USERNAME} ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
+RUN usermod -aG video $USERNAME
+
+# Add after DCV installation and before switching to non-root user
+RUN mkdir -p /var/run/dcv && \
+    mkdir -p /var/log/dcv && \
+    chown -R $USERNAME:$USERNAME /var/run/dcv && \
+    chown -R $USERNAME:$USERNAME /var/log/dcv && \
+    chmod 755 /var/run/dcv && \
+    chmod 755 /var/log/dcv && \
+    mkdir -p /etc/dcv && \
+    echo -e "[security]\nauthentication=none\n\n[display/linux]\ndisable-local-console=false" > /etc/dcv/dcv.conf && \
+    echo "computeruse:admin" | chpasswd
+
+# Copy the dbus config file to allow non-root users
+COPY --chown=root:root ./image/dbus.conf /etc/dbus-1/system.d/dcv-server.conf
+
+USER computeruse
+WORKDIR $HOME
+
+# setup python
+RUN git clone https://github.com/pyenv/pyenv.git ~/.pyenv && \
+    cd ~/.pyenv && src/configure && make -C src && cd .. && \
+    echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc && \
+    echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc && \
+    echo 'eval "$(pyenv init -)"' >> ~/.bashrc
+ENV PYENV_ROOT="$HOME/.pyenv"
+ENV PATH="$PYENV_ROOT/bin:$PATH"
+ENV PYENV_VERSION_MAJOR=3
+ENV PYENV_VERSION_MINOR=11
+ENV PYENV_VERSION_PATCH=6
+ENV PYENV_VERSION=$PYENV_VERSION_MAJOR.$PYENV_VERSION_MINOR.$PYENV_VERSION_PATCH
+RUN eval "$(pyenv init -)" && \
+    pyenv install $PYENV_VERSION && \
+    pyenv global $PYENV_VERSION && \
+    pyenv rehash
+ENV PATH="$HOME/.pyenv/shims:$HOME/.pyenv/bin:$PATH"
+RUN python -m pip install --upgrade pip==23.1.2 setuptools==58.0.4 wheel==0.40.0 && \
+    python -m pip config set global.disable-pip-version-check true
+
+# only reinstall if requirements.txt changes
+COPY --chown=$USERNAME:$USERNAME requirements.txt $HOME/computer_use_demo/requirements.txt
+RUN python -m pip install -r $HOME/computer_use_demo/requirements.txt
+
+# setup desktop env & app
+COPY --chown=$USERNAME:$USERNAME image/ $HOME
+RUN chmod +x $HOME/entrypoint.sh $HOME/dvc_startup.sh
+COPY --chown=$USERNAME:$USERNAME computer_use_demo/ $HOME/computer_use_demo/
+ARG DISPLAY_NUM=1
+ARG HEIGHT=768
+ARG WIDTH=1024
+ENV DISPLAY_NUM=$DISPLAY_NUM
+ENV HEIGHT=$HEIGHT
+ENV WIDTH=$WIDTH
+
+# Expose ports
+EXPOSE 5000
+EXPOSE 8443
+
+ENTRYPOINT [ "./entrypoint.sh" ]

computer-use/computer_use_aws/environment_image/Dockerfile

@@ -0,0 +1,29 @@
+from flask import Flask, request, jsonify
+from tools import BashTool, ComputerTool, EditTool, ToolCollection
+
+app = Flask(__name__)
+
+tool_collection = ToolCollection(
+    ComputerTool(),
+    BashTool(),
+    EditTool(),
+)
+
+@app.route('/execute', methods=['POST'])
+async def execute_tool():
+    data = request.json
+    tool_name = data.get('tool')
+    tool_input = data.get('input')
+
+    if not tool_name or not tool_input:
+        return jsonify({"error": "Missing tool name or input"}), 400
+
+    try:
+        result = await tool_collection.run(name=tool_name, tool_input=tool_input)
+        return jsonify({"result": result.to_dict()})
+    except Exception as e:
+        print(e)
+        return jsonify({"error": str(e)}), 500
+
+if __name__ == '__main__':
+    app.run(host='0.0.0.0', port=5000)