Skip to content

Commit b3bb54d

Browse files
marconaemarconae
authored
Merge pull request #3 from exasol-labs/bucketfs-skills
Added bucketfs skills
2 parents e9d745b + 6e188a7 commit b3bb54d

File tree

3 files changed

+257
-1
lines changed

3 files changed

+257
-1
lines changed

‎plugins/exasol/.claude-plugin/plugin.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "exasol",
3-
"description": "Interact with Exasol databases using the exapump CLI tool. Upload CSV/Parquet files, run SQL queries, export data, leverage Exasol-specific SQL features, design tables with DISTRIBUTE BY/PARTITION BY, write UDFs in Python/Java/Lua/R, and build custom Script Language Containers.",
3+
"description": "Interact with Exasol databases using the exapump CLI tool. Upload CSV/Parquet files, run SQL queries, export data, leverage Exasol-specific SQL features, design tables with DISTRIBUTE BY/PARTITION BY, write UDFs in Python/Java/Lua/R, build custom Script Language Containers, and manage BucketFS files.",
44
"version": "0.7.0",
55
"author": {
66
"name": "exasol-labs"

‎plugins/exasol/commands/bucketfs.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# /bucketfs Command
2+
3+
Manage files in Exasol BucketFS using the `exapump` CLI tool.
4+
5+
## Usage
6+
7+
```
8+
/bucketfs <task description or exapump command>
9+
```
10+
11+
## Arguments
12+
13+
The argument can be either:
14+
- A **direct exapump command**: `/bucketfs exapump bucketfs ls -r`
15+
- A **task description**: `/bucketfs upload model.pkl to the models directory`
16+
- A **question about BucketFS**: `/bucketfs what files are in the jars directory?`
17+
18+
## Behavior
19+
20+
When invoked:
21+
22+
0. **Verify connection configuration** before doing anything else:
23+
- Check if `~/.exapump/config.toml` exists and contains a default profile.
24+
- If configured, proceed.
25+
- If not configured, **ask the user** for the required connection details (host, port, bucket, passwords). Do not guess or assume any defaults. Help the user create a profile in `~/.exapump/config.toml` before continuing.
26+
27+
1. **Identify the task type**:
28+
- If the argument contains or implies an `exapump bucketfs` command (`ls`, `cp`, `rm`) — build and execute it directly.
29+
- If it is a natural language task description — translate it to the appropriate `exapump bucketfs` command(s) and execute.
30+
31+
2. **List files or directories** — use `exapump bucketfs ls`:
32+
- Use `exapump bucketfs ls <path>` to list a specific directory.
33+
- Use `exapump bucketfs ls -r <path>` for a recursive listing.
34+
- Use `exapump bucketfs ls` to list the bucket root.
35+
36+
3. **Upload a file or directory** — use `exapump bucketfs cp`:
37+
- Single file: `exapump bucketfs cp <local-file> <bucket-path>`
38+
- Preserve filename: `exapump bucketfs cp <local-file> <bucket-dir>/`
39+
- Direction is auto-detected from the source type.
40+
41+
4. **Download a file** — use `exapump bucketfs cp` in reverse:
42+
- `exapump bucketfs cp <bucket-path> <local-destination>`
43+
44+
5. **Delete a file** — use `exapump bucketfs rm`:
45+
- `exapump bucketfs rm <path-in-bucket>`
46+
- **Always confirm with the user before executing a delete operation.**
47+
48+
6. **On errors**:
49+
- Check that the path exists with `exapump bucketfs ls <path>`.
50+
- For permission errors, verify that `bfs_write_password` / `bfs_read_password` are set correctly in `~/.exapump/config.toml`.
51+
- For TLS errors, check `bfs_tls` and `bfs_validate_certificate` in the profile.
52+
- Connection settings can be overridden per command via `--bfs-host`, `--bfs-port`, `--bfs-bucket`, `--profile`, etc.
53+
54+
## Examples
55+
56+
```
57+
/bucketfs list all files in the bucket
58+
/bucketfs exapump bucketfs ls -r models/
59+
/bucketfs upload my_model.pkl to models/my_model.pkl
60+
/bucketfs download jars/library.jar to ./lib/
61+
/bucketfs delete models/old_model.pkl
62+
/bucketfs what files are under the jars folder?
63+
```

‎plugins/exasol/skills/exasol-bucketfs/SKILL.md‎

Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
---
2+
name: exasol-bucketfs
3+
description: "Exasol BucketFS file system management via exapump CLI. Covers listing, uploading, downloading, and deleting files and directories in BucketFS, BucketFS configuration, bucket structure, and use with UDFs."
4+
---
5+
6+
# Exasol BucketFS Skill
7+
8+
Trigger when the user mentions **BucketFS**, **exapump**, **bucket**, **bfsdefault**, **upload to BucketFS**, **download from BucketFS**, **delete from BucketFS**, **BucketFS path**, **BucketFS file**, or any BucketFS file management task.
9+
10+
## BucketFS Concepts
11+
12+
**BucketFS** is a synchronous distributed file system available on all nodes of an Exasol cluster. Files stored in BucketFS are automatically replicated to every cluster node.
13+
14+
Key concepts:
15+
- **Service**: A named BucketFS instance. The default service is `bfsdefault`.
16+
- **Bucket**: A storage container within a service. The default bucket is `default`.
17+
- **Path inside BucketFS**: Files are referenced by the path within the bucket (e.g., `models/my_model.pkl`).
18+
- **Local path inside UDFs**: Files are accessible at `/buckets/<service>/<bucket>/<path>` (e.g., `/buckets/bfsdefault/default/models/my_model.pkl`).
19+
20+
Important characteristics:
21+
- Writes are atomic — a file is either fully written or not at all.
22+
- No transactions and no file locks; the latest write wins.
23+
- All nodes see identical content after synchronisation.
24+
- BucketFS is not included in database backups — manage backups separately.
25+
- Not suited for very large datasets due to replication overhead.
26+
27+
## exapump CLI
28+
29+
The `exapump` command is the CLI tool for managing BucketFS. All BucketFS operations use the `exapump bucketfs` subcommand.
30+
31+
### Connection Configuration
32+
33+
Connection settings are stored in `~/.exapump/config.toml` as named profiles. Example:
34+
35+
```toml
36+
[production]
37+
host = "exasol-prod.example.com"
38+
user = "admin"
39+
password = "s3cret"
40+
default = true
41+
bfs_write_password = "bucketpw"
42+
bfs_read_password = "bucketpw"
43+
```
44+
45+
Key profile fields:
46+
47+
| Field | Default | Purpose |
48+
|-------|---------|---------|
49+
| `bfs_host` | Falls back to `host` | BucketFS hostname |
50+
| `bfs_port` | `2581` | BucketFS port |
51+
| `bfs_bucket` | `default` | Bucket name |
52+
| `bfs_write_password` | Required | Write authentication |
53+
| `bfs_read_password` | Falls back to write password | Read authentication |
54+
| `bfs_tls` | Falls back to `tls` | Enable TLS |
55+
| `bfs_validate_certificate` | Falls back to `validate_certificate` | Certificate validation |
56+
57+
Connection parameters can also be overridden per command via CLI flags (highest priority):
58+
59+
| Flag | Purpose |
60+
|------|---------|
61+
| `--profile` | Select a named profile |
62+
| `--bfs-host` | Override hostname |
63+
| `--bfs-port` | Override port |
64+
| `--bfs-bucket` | Override bucket name |
65+
| `--bfs-write-password` | Override write password |
66+
| `--bfs-read-password` | Override read password |
67+
| `--bfs-tls` | Override TLS setting |
68+
| `--bfs-validate-certificate` | Override certificate validation |
69+
70+
**Parameter resolution order:** CLI flags → profile values → smart defaults.
71+
72+
### Configuration Protocol
73+
74+
**Before any BucketFS operation**, verify the connection is configured:
75+
76+
1. Check if `~/.exapump/config.toml` exists and contains a default profile.
77+
2. If configured, proceed with the operation.
78+
3. If not configured, **ask the user** for the required connection details (host, port, bucket, passwords). Do not guess or assume any defaults. Help the user create the profile in `~/.exapump/config.toml`.
79+
80+
---
81+
82+
## Commands
83+
84+
### `ls` — List Contents
85+
86+
```bash
87+
exapump bucketfs ls [PATH] [OPTIONS]
88+
exapump bucketfs ls -r [PATH] # Recursive listing
89+
exapump bucketfs ls --recursive [PATH]
90+
```
91+
92+
**Examples:**
93+
```bash
94+
exapump bucketfs ls # List bucket root
95+
exapump bucketfs ls models/ # List a directory
96+
exapump bucketfs ls -r models/ # Recursively list all files under models/
97+
```
98+
99+
---
100+
101+
### `cp` — Copy / Upload / Download
102+
103+
Direction is automatically determined by the source type (local file vs. BucketFS path).
104+
105+
Upload a local file to BucketFS:
106+
```bash
107+
exapump bucketfs cp <local-file> <bucket-path>
108+
exapump bucketfs cp <local-file> <bucket-dir>/ # Preserve filename
109+
```
110+
111+
Download a file from BucketFS to local:
112+
```bash
113+
exapump bucketfs cp <bucket-path> <local-path>
114+
```
115+
116+
**Examples:**
117+
```bash
118+
exapump bucketfs cp my_model.pkl models/my_model.pkl # Upload with explicit name
119+
exapump bucketfs cp my_model.pkl models/ # Upload, preserve filename
120+
exapump bucketfs cp library.jar jars/library.jar # Upload JAR for UDF
121+
exapump bucketfs cp models/my_model.pkl . # Download to current dir
122+
exapump bucketfs cp models/my_model.pkl ./local-copy.pkl # Download with rename
123+
```
124+
125+
---
126+
127+
### `rm` — Remove a File
128+
129+
```bash
130+
exapump bucketfs rm <path-in-bucket>
131+
```
132+
133+
**Examples:**
134+
```bash
135+
exapump bucketfs rm models/old_model.pkl # Delete a single file
136+
```
137+
138+
---
139+
140+
## Typical Use Cases
141+
142+
### Upload a JAR for a Java UDF
143+
144+
```bash
145+
exapump bucketfs cp my_library.jar jars/my_library.jar
146+
```
147+
148+
Reference in UDF SQL:
149+
```sql
150+
CREATE OR REPLACE JAVA SCALAR SCRIPT my_schema.my_func(input VARCHAR(2000))
151+
RETURNS VARCHAR(2000) AS
152+
%scriptclass com.example.MyClass;
153+
%jar /buckets/bfsdefault/default/jars/my_library.jar;
154+
/
155+
```
156+
157+
### Upload an ML Model for a Python UDF
158+
159+
```bash
160+
exapump bucketfs cp model.pkl models/model.pkl
161+
```
162+
163+
Load in Python UDF:
164+
```python
165+
import pickle
166+
with open('/buckets/bfsdefault/default/models/model.pkl', 'rb') as f:
167+
model = pickle.load(f)
168+
```
169+
170+
### Upload a Custom Script Language Container (SLC)
171+
172+
```bash
173+
exapump bucketfs cp my_slc.tar.gz slc/my_slc.tar.gz
174+
```
175+
176+
Then activate via SQL:
177+
```sql
178+
ALTER SESSION SET SCRIPT_LANGUAGES='PYTHON3=localzmq+protobuf:///bfsdefault/default/slc/my_slc?lang=python#buckets/bfsdefault/default/slc/my_slc/exaudf/exaudfclient_py3';
179+
```
180+
181+
### Browse and Clean Up BucketFS
182+
183+
```bash
184+
exapump bucketfs ls -r # See all files
185+
exapump bucketfs rm old_model.pkl # Remove an outdated file
186+
```
187+
188+
---
189+
190+
## Related Skills
191+
192+
- **exasol-udfs**: For creating UDF scripts that reference files stored in BucketFS.
193+
- **exasol-database**: For SQL-level operations and connecting to Exasol.

0 commit comments

Comments
 (0)