Skip to content

Commit 8ec154f

Browse files
geroplgeropl
committed
[memory-bank] Documented docker-up, image-builder-bob, node-labeler
Tool: gitpod/catfood.gitpod.cloud
1 parent b17b3c1 commit 8ec154f

File tree

5 files changed

+460
-1
lines changed

5 files changed

+460
-1
lines changed

‎memory-bank/activeContext.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,9 @@ Key areas of focus include:
4343
- spicedb: Provides authorization and permission management
4444
- scrubber: Removes or masks sensitive information from data
4545
- service-waiter: Waits for services to become available
46+
- docker-up: Sets up and manages Docker within workspace containers
47+
- image-builder-bob: Builds and pushes workspace images during workspace startup
48+
- node-labeler: Manages node labels and annotations for workspace scheduling
4649

4750
As work progresses, this section will continue to be updated to reflect:
4851
- Additional component documentation

‎memory-bank/components/docker-up.md‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
# Docker-Up Component
2+
3+
## Overview
4+
5+
The Docker-Up component in Gitpod is responsible for setting up and managing Docker within workspace containers. It provides a way to run Docker in a rootless mode inside Gitpod workspaces, enabling users to build and run containers without requiring privileged access. The component handles the installation, configuration, and startup of Docker daemon and related tools, ensuring they work correctly within the constraints of a workspace environment.
6+
7+
## Purpose
8+
9+
The primary purposes of the Docker-Up component are:
10+
- Enable Docker functionality within Gitpod workspaces
11+
- Provide rootless Docker execution for security
12+
- Automatically install Docker and its dependencies when needed
13+
- Configure Docker daemon with appropriate settings for workspace environments
14+
- Handle network namespace and user namespace setup
15+
- Provide Docker Compose functionality
16+
- Ensure proper permissions for Docker socket access
17+
- Manage container runtime (runc) with appropriate configuration
18+
- Support custom Docker daemon arguments
19+
- Facilitate container port binding through capability management
20+
21+
## Architecture
22+
23+
The Docker-Up component consists of several key parts:
24+
25+
1. **Docker-Up**: The main executable that sets up and starts the Docker daemon
26+
2. **Runc-Facade**: A wrapper around runc (the OCI container runtime) that handles rootless container execution
27+
3. **Dockerd**: Configuration and argument parsing for the Docker daemon
28+
4. **Embedded Binaries**: Docker, Docker Compose, and runc binaries embedded in the component
29+
30+
The component is designed to run as a service within a Gitpod workspace, automatically starting when Docker functionality is requested and configuring the environment appropriately.
31+
32+
## Key Features
33+
34+
### Docker Daemon Management
35+
36+
- Starts and configures the Docker daemon with appropriate settings
37+
- Sets up the Docker socket with proper permissions
38+
- Configures data storage location within the workspace
39+
- Handles MTU configuration based on the container network interface
40+
- Supports custom Docker daemon arguments through environment variables
41+
- Manages Docker daemon lifecycle
42+
43+
### Automatic Installation
44+
45+
- Detects missing prerequisites (Docker, Docker Compose, iptables, etc.)
46+
- Automatically installs required components when needed
47+
- Embeds Docker and Docker Compose binaries for reliable installation
48+
- Ensures correct versions of dependencies are installed
49+
- Handles different Linux distribution package managers
50+
51+
### Rootless Container Execution
52+
53+
- Configures Docker to run in rootless mode for security
54+
- Uses runc-facade to handle rootless container execution
55+
- Manages user namespace mappings
56+
- Configures capabilities for container processes
57+
- Handles OOM score adjustment for container stability
58+
59+
### Network Configuration
60+
61+
- Configures Docker network with appropriate MTU settings
62+
- Handles network namespace setup
63+
- Enables binding to privileged ports (below 1024) without root
64+
- Configures iptables for container networking
65+
66+
## Configuration
67+
68+
The Docker-Up component can be configured through command-line flags and environment variables:
69+
70+
### Command-line Flags
71+
- `--verbose`, `-v`: Enables verbose logging
72+
- `--runc-facade`: Enables the runc-facade to handle rootless idiosyncrasies
73+
- `--bin-dir`: Directory where runc-facade is found
74+
- `--auto-install`: Auto-install prerequisites (Docker)
75+
- `--user-accessible-socket`: Make the Docker socket user accessible
76+
- `--dont-wrap-netns`: Control network namespace wrapping
77+
- `--auto-login`: Use content of GITPOD_IMAGE_AUTH to automatically login with the Docker daemon
78+
79+
### Environment Variables
80+
- `DOCKERD_ARGS`: JSON-formatted custom arguments for the Docker daemon
81+
- `LISTEN_FDS`: Used for socket activation
82+
- `WORKSPACEKIT_WRAP_NETNS`: Controls network namespace wrapping
83+
- `GITPOD_IMAGE_AUTH`: Docker registry authentication information
84+
85+
## Integration Points
86+
87+
The Docker-Up component integrates with:
88+
1. **Workspace Container**: Runs within the workspace container
89+
2. **Supervisor**: Started by the supervisor when Docker functionality is requested
90+
3. **Workspacekit**: Interacts with workspacekit for namespace management
91+
4. **Container Registry**: For pulling and pushing container images
92+
5. **User Code**: Provides Docker CLI and API for user code
93+
94+
## Usage Patterns
95+
96+
### Starting Docker Daemon
97+
```bash
98+
docker-up
99+
```
100+
101+
### Using Custom Docker Daemon Arguments
102+
```bash
103+
DOCKERD_ARGS='{"remap-user":"1000"}' docker-up
104+
```
105+
106+
### Using Docker Compose
107+
```bash
108+
docker-compose up -d
109+
```
110+
111+
## Dependencies
112+
113+
### Internal Dependencies
114+
- `components/common-go`: Common Go utilities
115+
116+
### External Dependencies
117+
- Docker daemon
118+
- Docker CLI
119+
- Docker Compose
120+
- runc (OCI container runtime)
121+
- iptables
122+
- uidmap (for user namespace mapping)
123+
124+
## Security Considerations
125+
126+
The component implements several security measures:
127+
128+
1. **Rootless Execution**: Runs Docker without root privileges
129+
2. **User Namespace Mapping**: Isolates container user IDs from host
130+
3. **Capability Management**: Provides minimal required capabilities
131+
4. **Socket Permissions**: Controls access to the Docker socket
132+
5. **OOM Score Adjustment**: Prevents container processes from being killed under memory pressure
133+
6. **Network Isolation**: Configures network namespaces for isolation
134+
135+
## Implementation Details
136+
137+
### Runc-Facade
138+
139+
The runc-facade is a wrapper around the standard runc container runtime that:
140+
- Modifies container configurations for rootless execution
141+
- Adds the CAP_NET_BIND_SERVICE capability to allow binding to privileged ports
142+
- Sets OOM score adjustment to prevent container processes from being killed
143+
- Removes problematic sysctl settings that don't work in rootless mode
144+
- Implements retry logic for container creation to handle timing issues
145+
146+
### Docker Daemon Arguments
147+
148+
The component supports custom Docker daemon arguments through the `DOCKERD_ARGS` environment variable, which accepts a JSON object with configuration options:
149+
- `remap-user`: Configure user namespace remapping
150+
- `proxies`: HTTP/HTTPS proxy settings
151+
- `http-proxy`: HTTP proxy configuration
152+
- `https-proxy`: HTTPS proxy configuration
153+
154+
## Related Components
155+
156+
- **Supervisor**: Manages the lifecycle of Docker-Up
157+
- **Workspacekit**: Provides namespace management
158+
- **Registry-Facade**: Interacts with Docker for image management
159+
- **Image-Builder**: Uses Docker for building workspace images

‎memory-bank/components/image-builder-bob.md‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
# Image-Builder-Bob Component
2+
3+
## Overview
4+
5+
The Image-Builder-Bob component in Gitpod is a CLI tool responsible for building and pushing workspace images during workspace startup. It operates within a headless workspace created by the image-builder-mk3 component and handles the building of custom Docker images based on user-defined Dockerfiles in `.gitpod.yml`. The component consists of two main parts: a build process that creates the images and a proxy that handles authentication for pushing the images to registries.
6+
7+
## Purpose
8+
9+
The primary purposes of the Image-Builder-Bob component are:
10+
- Build custom workspace images from user-defined Dockerfiles
11+
- Create base layers for workspace images
12+
- Push built images to container registries
13+
- Handle authentication for private container registries
14+
- Provide a secure way to build images without exposing registry credentials
15+
- Support the workspace prebuild process
16+
- Enable customization of development environments
17+
- Optimize image building with caching
18+
- Facilitate the use of custom Docker configurations in workspaces
19+
20+
## Architecture
21+
22+
The Image-Builder-Bob component consists of several key parts:
23+
24+
1. **Bob Build**: A command that builds the base layer and workspace image
25+
2. **Bob Proxy**: A proxy that authenticates image pushes to registries
26+
3. **Builder Package**: Core functionality for building images
27+
4. **Proxy Package**: Handles registry authentication and proxying
28+
5. **Runc-Facade**: A wrapper for the runc container runtime
29+
30+
The component operates in a headless workspace where:
31+
- `bob proxy` runs in ring1 (started by workspacekit) and receives credentials for pushing images
32+
- `bob build` runs as a workspace task and builds the images, pushing them to `bob proxy`
33+
34+
## Key Features
35+
36+
### Image Building
37+
38+
- **Base Layer Building**: Creates a base image from a custom Dockerfile specified in `.gitpod.yml`
39+
- **Workspace Image Building**: Uses the base layer to create a workspace image
40+
- **Caching**: Reuses previously built base images when available
41+
- **Buildkit Integration**: Uses Buildkit for efficient image building
42+
- **Context Management**: Handles the build context for Docker images
43+
- **Error Handling**: Provides detailed error messages for build failures
44+
- **Logging**: Writes build logs to `/workspace/.gitpod/bob.log`
45+
46+
### Registry Authentication
47+
48+
- **Secure Credential Handling**: Manages registry credentials securely
49+
- **Authentication Proxy**: Proxies and authenticates image pushes
50+
- **Encryption**: Supports encrypted authentication tokens
51+
- **Multiple Registry Support**: Can authenticate with different registries
52+
- **Cloud Provider Integration**: Supports authentication with cloud provider registries (AWS ECR, etc.)
53+
54+
## Configuration
55+
56+
The Image-Builder-Bob component is configured through environment variables:
57+
58+
### Build Configuration
59+
- `BOB_TARGET_REF`: Reference for the target image
60+
- `BOB_BASE_REF`: Reference for the base image
61+
- `BOB_BUILD_BASE`: Whether to build the base image
62+
- `BOB_DOCKERFILE_PATH`: Path to the Dockerfile
63+
- `BOB_CONTEXT_DIR`: Directory to use as build context
64+
- `BOB_EXTERNAL_BUILDKITD`: External Buildkit daemon to use
65+
- `BOB_LOCAL_CACHE_IMPORT`: Local cache import configuration
66+
- `THEIA_WORKSPACE_ROOT`: Workspace root directory
67+
68+
### Authentication Configuration
69+
- `BOB_BASELAYER_AUTH`: Authentication for the base layer registry
70+
- `BOB_WSLAYER_AUTH`: Authentication for the workspace layer registry
71+
- `BOB_AUTH_KEY`: Key for decrypting authentication tokens
72+
73+
### Proxy Configuration
74+
- `WORKSPACEKIT_BOBPROXY_BASEREF`: Base image reference for the proxy
75+
- `WORKSPACEKIT_BOBPROXY_TARGETREF`: Target image reference for the proxy
76+
- `WORKSPACEKIT_BOBPROXY_AUTH`: Authentication for the proxy
77+
- `WORKSPACEKIT_BOBPROXY_ADDITIONALAUTH`: Additional authentication for the proxy
78+
79+
## Usage Patterns
80+
81+
### Building an Image
82+
```bash
83+
BOB_BASE_REF=localhost:5000/source:latest BOB_TARGET_REF=localhost:5000/target:83 bob build
84+
```
85+
86+
### Running the Proxy
87+
```bash
88+
bob proxy --base-ref=localhost:5000/source:latest --target-ref=localhost:5000/target:83 --auth='{"username":"user","password":"pass"}'
89+
```
90+
91+
### Typical Workflow
92+
1. `image-builder-mk3` creates a headless workspace
93+
2. `bob proxy` starts in ring1 with registry credentials
94+
3. `bob build` runs as a workspace task
95+
4. Base layer is built if needed (custom Dockerfile)
96+
5. Workspace image is built using the base layer
97+
6. Images are pushed through `bob proxy` to the registry
98+
7. Workspace starts using the built image
99+
100+
## Integration Points
101+
102+
The Image-Builder-Bob component integrates with:
103+
1. **Image-Builder-MK3**: Creates the headless workspace where Bob runs
104+
2. **Workspacekit**: Starts `bob proxy` in ring1
105+
3. **Registry-Facade**: The built images are later modified by registry-facade
106+
4. **Container Registries**: For pushing and pulling images
107+
5. **Buildkit**: For efficient image building
108+
109+
## Dependencies
110+
111+
### Internal Dependencies
112+
- `components/common-go`: Common Go utilities
113+
114+
### External Dependencies
115+
- Buildkit: For building container images
116+
- Docker Registry: For storing built images
117+
- Containerd: For container operations
118+
- OCI Tools: For working with OCI images
119+
120+
## Security Considerations
121+
122+
The component implements several security measures:
123+
124+
1. **Credential Isolation**: Registry credentials are only available to `bob proxy`, not to user code
125+
2. **Encryption**: Authentication tokens can be encrypted
126+
3. **Proxy Authentication**: All image pushes are authenticated through the proxy
127+
4. **Rootless Building**: Images are built without requiring root privileges
128+
5. **Isolated Workspaces**: Building happens in isolated headless workspaces
129+
130+
## Implementation Details
131+
132+
### Build Process
133+
134+
The build process consists of two main steps:
135+
1. **Base Layer Building**: If a custom Dockerfile is specified, a base image is built
136+
2. **Workspace Image Building**: Using crane to copy the image from the base layer
137+
138+
The base layer can be either a previously built custom Dockerfile or a public image. The built images do not include components like `supervisor` or the IDE, as these layers are added by `registry-facade` during image pull.
139+
140+
### Proxy Implementation
141+
142+
The proxy acts as an intermediary between `bob build` and the actual container registry:
143+
1. Receives image pushes from `bob build` on localhost
144+
2. Authenticates with the target registry using provided credentials
145+
3. Forwards the image to the target registry
146+
4. Handles authentication for both base and target images
147+
148+
## Related Components
149+
150+
- **Image-Builder-MK3**: Orchestrates the image building process
151+
- **Registry-Facade**: Adds additional layers to the built images
152+
- **Supervisor**: Manages the workspace environment
153+
- **Workspacekit**: Starts `bob proxy` in ring1

0 commit comments

Comments
 (0)