Skip to content

Commit 42c7549

Browse files
JuArceJuArce
authored
infra(aggregation_mode): add setup files (#1866)
1 parent 2e8796f commit 42c7549

14 files changed

+334
-0
lines changed

‎Makefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,9 @@ start_proof_aggregator: is_aggregator_set ## Starts proof aggregator with provin
167167
start_proof_aggregator_gpu: is_aggregator_set ## Starts proof aggregator with proving + GPU acceleration (CUDA)
168168
AGGREGATOR=$(AGGREGATOR) SP1_PROVER=cuda cargo run --manifest-path ./aggregation_mode/Cargo.toml --release --features prove,gpu -- config-files/config-proof-aggregator.yaml
169169

170+
install_aggregation_mode: ## Install the aggregation mode with proving enabled
171+
cargo install --path aggregation_mode --features prove
172+
170173
_AGGREGATOR_:
171174

172175
build_aggregator:

‎infra/aggregation_mode/README.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Aggregation Mode Setup
2+
3+
## Setup on Server with GPU
4+
5+
To setup the server with GPU, follow the steps in [aggregation_mode.sh](aggregation_mode.sh).
6+
7+
After running all the steps, `aggregation_mode.timer` will execute every 24hs the `aggregation_mode.service`
8+
9+
## Check Service Status
10+
11+
To check the status of the timer, run:
12+
13+
```bash
14+
systemctl status aggregation_mode.timer --user
15+
```
16+
17+
To check the status of the service, run:
18+
19+
```bash
20+
systemctl status aggregation_mode.service --user
21+
```
22+
23+
## Start Service manually
24+
25+
If you need to start the service manually, without waiting for the timer, run:
26+
27+
```bash
28+
systemctl start aggregation_mode.service --user
29+
```
30+
31+
## Check Logs
32+
33+
To check the logs of the service, run:
34+
35+
```bash
36+
journalctl -xfeu aggregation_mode.service --user
37+
```
38+
39+
Note: You can add `-n <n_of_lines>` to limit the number of lines to show.

‎infra/aggregation_mode/aggregation_mode.service‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
[Unit]
2+
Description=AggregationMode
3+
After=network.target
4+
5+
[Service]
6+
Type=oneshot
7+
WorkingDirectory=/home/user
8+
ExecStartPre=sleep 60
9+
ExecStart=/home/user/.cargo/bin/proof_aggregator /home/user/config/config-proof-aggregator.yaml
10+
Environment="SP1_PROVER=cuda"
11+
12+
[Install]
13+
WantedBy=multi-user.target

‎infra/aggregation_mode/aggregation_mode.sh‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
#!/bin/bash
2+
3+
# This guide assumes you already clone the github repository
4+
# You have to cd to the repository
5+
6+
# Set new server name
7+
while :; do
8+
echo -e "\nEnter new server name:"
9+
read -p "> " new_server_name
10+
11+
if [[ ! "$new_server_name" =~ ^[a-zA-Z0-9-]+$ ]]; then
12+
echo "Invalid characters used in the server name. Please use only alphanumeric characters and hyphens (-)."
13+
else
14+
echo -e "\nSetting new server name to '$new_server_name'..."
15+
echo "Old server name: $old_server_name"
16+
sudo hostnamectl set-hostname "$new_server_name"
17+
sudo sed -i "s/$old_server_name/$new_server_name/g" /etc/hosts
18+
echo "Please reconnect to the server to see the name change."
19+
break
20+
fi
21+
done
22+
23+
# Enable linger
24+
sudo loginctl enable-linger user
25+
26+
# Install other dependencies
27+
sudo apt install -y gcc pkg-config libssl-dev build-essential apt-transport-https ca-certificates curl software-properties-common nvtop
28+
29+
# Install docker
30+
sudo apt-get update
31+
sudo apt-get install ca-certificates curl
32+
sudo install -m 0755 -d /etc/apt/keyrings
33+
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
34+
sudo chmod a+r /etc/apt/keyrings/docker.asc
35+
echo \
36+
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
37+
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
38+
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
39+
sudo apt-get update
40+
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
41+
sudo groupadd docker
42+
sudo usermod -aG docker $USER
43+
newgrp docker
44+
45+
# Install tailscale
46+
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null
47+
curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/noble.tailscale-keyring.list | sudo tee /etc/apt/sources.list.d/tailscale.list
48+
sudo apt-get update
49+
sudo apt-get install tailscale
50+
sudo tailscale up --ssh --advertise-tags=tag:server && sudo tailscale set --auto-update
51+
52+
# Install CUDA
53+
sudo add-apt-repository ppa:graphics-drivers/ppa
54+
sudo apt update
55+
sudo apt install nvidia-driver-570
56+
57+
# If see errors
58+
sudo apt-mark unhold cuda-drivers cuda-toolkit-12-6 nvidia-dkms-565-server nvidia-fabricmanager-565 nvidia-headless-565-server nvidia-utils-565-server
59+
sudo apt update
60+
sudo apt install nvidia-driver-570
61+
sudo apt autoremove
62+
sudo apt autoclean
63+
sudo reboot
64+
nvidia-smi # To check if the driver is installed correctly
65+
66+
# Setup Docker and CUDA
67+
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
68+
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
69+
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
70+
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
71+
sudo apt-get update
72+
sudo apt-get install -y nvidia-container-toolkit
73+
sudo nvidia-ctk runtime configure --runtime=docker
74+
sudo systemctl restart docker
75+
76+
# Install Rust
77+
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
78+
. "$HOME/.cargo/env"
79+
80+
# Install SP1
81+
curl -L https://sp1.succinct.xyz | bash
82+
source $HOME/.bashrc
83+
sp1up
84+
85+
# Install cast
86+
curl -L https://foundry.paradigm.xyz | bash
87+
source $HOME/.bashrc
88+
foundryup
89+
90+
# Create directories
91+
mkdir -p ~/config
92+
mkdir -p ~/.config/systemd/user
93+
mkdir -p ~/.keystores
94+
95+
# Create keystore
96+
cast wallet import proof_aggregation.keystore -k $HOME/.keystores -i
97+
98+
# Create config file interactively
99+
./infra/aggregation_mode/config_file.sh ./infra/aggregation_mode/config-proof-aggregator.template.yaml
100+
touch $HOME/config/proof-aggregator.last_aggregated_block.json
101+
read -p "Enter a number (last_aggregated_block): " num && echo "{\"last_aggregated_block\":$num}" > $HOME/config/proof-aggregator.last_aggregated_block.json
102+
103+
# Build the proof_aggregator
104+
make install_aggregation_mode
105+
106+
# Setup systemd service
107+
cp ./infra/aggregation_mode/aggregation_mode.service $HOME/.config/systemd/user/aggregation_mode.service
108+
cp ./infra/aggregation_mode/aggregation_mode.timer $HOME/.config/systemd/user/aggregation_mode.timer
109+
110+
#sudo systemctl enable aggregation_mode.service
111+
systemctl --user enable aggregation_mode.timer
112+
systemctl --user start aggregation_mode.timer
113+
114+
# Run the proof_aggregator manually if you want
115+
systemctl --user start aggregation_mode.service
116+
117+
# Check timer status
118+
systemctl --user status aggregation_mode.timer
119+
120+
# Check logs
121+
journalctl -xfeu aggregation_mode.service --user -n10

‎infra/aggregation_mode/aggregation_mode.timer‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
[Unit]
2+
Description=AggregationMode Timer
3+
Requires=aggregation_mode.service
4+
5+
[Timer]
6+
Unit=aggregation_mode.service
7+
AccuracySec=1s
8+
# Run at 15:00 UTC (12:00 UTC-3) every day
9+
OnCalendar=15:00
10+
11+
[Install]
12+
WantedBy=timers.target

‎infra/aggregation_mode/config-proof-aggregator.template.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
aligned_service_manager_address: <aligned_service_manager_address>
2+
proof_aggregation_service_address: <proof_aggregation_service_address>
3+
eth_rpc_url: <eth_rpc_url>
4+
eth_ws_url: <eth_ws_url>
5+
max_proofs_in_queue: 1000
6+
last_aggregated_block_filepath: ~/config/proof-aggregator.last_aggregated_block.json
7+
8+
ecdsa:
9+
private_key_store_path: <private_key_store_path>
10+
private_key_store_password: <private_key_store_password>

‎infra/aggregation_mode/config_file.sh‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
#!/bin/bash
2+
3+
# Check if template file path is provided as argument
4+
if [ $# -ne 1 ]; then
5+
echo "Usage: $0 <template_file_path>"
6+
exit 1
7+
fi
8+
9+
TEMPLATE_FILE="$1"
10+
11+
# Verify template file exists
12+
if [ ! -f "$TEMPLATE_FILE" ]; then
13+
echo "Error: Template file '$TEMPLATE_FILE' not found"
14+
exit 1
15+
fi
16+
17+
# Create temporary file by copying the template
18+
TEMP_FILE=$(mktemp)
19+
cp "$TEMPLATE_FILE" "$TEMP_FILE"
20+
21+
# Function to prompt for input and replace placeholder
22+
prompt_and_replace() {
23+
local placeholder=$1
24+
local description=$2
25+
26+
read -p "Enter $description: " value
27+
sed -i "s|$placeholder|$value|g" "$TEMP_FILE"
28+
}
29+
30+
# Prompt for each placeholder found in the template
31+
prompt_and_replace "<aligned_service_manager_address>" "Aligned Service Manager Address"
32+
prompt_and_replace "<proof_aggregation_service_address>" "Proof Aggregation Service Address"
33+
prompt_and_replace "<eth_rpc_url>" "Ethereum RPC URL"
34+
prompt_and_replace "<eth_ws_url>" "Ethereum WebSocket URL"
35+
prompt_and_replace "<private_key_store_path>" "ECDSA Private Key Store Path (~/.keystores/proof_aggregation.keystore)"
36+
prompt_and_replace "<private_key_store_password>" "ECDSA Private Key Store Password"
37+
38+
# Create destination directory if it doesn't exist
39+
mkdir -p /home/user/config
40+
41+
# Copy the completed file to destination
42+
cp "$TEMP_FILE" $HOME/config/config-proof-aggregator.yaml
43+
44+
# Clean up temporary file
45+
rm "$TEMP_FILE"
46+
47+
echo "Configuration file has been created and copied to $HOME/config/config-proof-aggregator.yaml"

‎infra/aggregation_mode/remote_management/.env.aggregation_mode_start‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
API_KEY=<TensorDock_API_KEY>
2+
INSTANCE_ID=<INSTANCE_ID>

‎infra/aggregation_mode/remote_management/README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
This files are supposed to be used in case we want to manage the process using a second server who starts the GPU server using a timer.
2+
3+
It is useful to avoid being charged for the time the GPU server is not used.
4+
5+
As the GPU stock is limited we are keeping the GPU server running all the time, but in case we want to use it only when needed, we can use this scripts to start and stop the GPU server using a timer.

‎infra/aggregation_mode/remote_management/aggregation_mode_shutdown.sh‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/bin/bash
2+
3+
BASE_URL=https://dashboard.tensordock.com/api/v2
4+
5+
echo "Stopping GPU Server..."
6+
7+
sleep 120
8+
9+
curl -X POST "$BASE_URL/instances/$INSTANCE_ID/stop" \
10+
-H "Authorization: Bearer $API_KEY" \
11+
-H "Content-Type: application/json" \
12+
-H "Accept: application/json" \
13+
--data-urlencode 'disassociate_resources=true' \
14+
--max-time 60
15+
16+
echo "GPU Server stopped successfully."

0 commit comments

Comments
 (0)