Skip to content

Commit 8ddee96

Browse files
lloeschelloesche
authored
Update README (#20)
1 parent bc06de6 commit 8ddee96

File tree

2 files changed

+136
-61
lines changed

2 files changed

+136
-61
lines changed

‎README.md‎

Lines changed: 136 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -1,121 +1,194 @@
1-
# cloud2sql
2-
Read infrastructure data from your cloud and export it to a SQL database.
1+
# Cloud2SQL 🤩
2+
Read infrastructure data from your cloud ☁️ and export it to a SQL database 📋.
33

4+
![Cloud2SQL](misc/cloud2sql.gif)
45

56
## Installation
67

78
Python 3.9 or higher is required.
8-
It is recommended to use a separate virtual environment for this project. You can create one with the following command:
99

10-
```bash
11-
python3 -m venv venv --prompt "cloud2sql"
12-
source venv/bin/activate
13-
```
14-
15-
Once the virtual environment is activated, you can install cloud2sql:
10+
You can install Cloud2SQL using:
1611

1712
```bash
18-
pip install cloud2sql[all]
13+
pip install --user cloud2sql[all]
1914
```
2015

2116
If you only require support for a specific database, instead of `cloud2sql[all]` you can choose between `cloud2sql[snowflake]`, `cloud2sql[parquet]`, `cloud2sql[postgresql]`, `cloud2sql[mysql]`.
2217

2318
## Usage
2419

2520
The sources and destinations for `cloud2sql` are configured via a configuration file. Create your own configuration by adjusting the [config template file](./config-template.yaml).
21+
2622
You can safely delete the sections that are not relevant to you (e.g. if you do not use AWS, you can delete the `aws` section).
2723
All sections refer to cloud providers and are enabled if a configuration section is provided.
2824

29-
The following databases are currently supported:
25+
In the next section you will create a YAML configuration file. Once you have created your configuration file, you can run `cloud2sql` with the following command:
26+
27+
```bash
28+
cloud2sql --config myconfig.yaml
29+
```
30+
31+
## Configuration
32+
33+
Cloud2SQL uses a YAML configuration file to define the `sources` and `destinations`.
34+
35+
### Sources
36+
37+
#### AWS
38+
39+
```yaml
40+
sources:
41+
aws:
42+
# AWS Access Key ID (null to load from env - recommended)
43+
access_key_id: null
44+
# AWS Secret Access Key (null to load from env - recommended)
45+
secret_access_key: null
46+
# IAM role name to assume
47+
role: null
48+
# List of AWS profiles to collect
49+
profiles: null
50+
# List of AWS Regions to collect (null for all)
51+
region: null
52+
# Scrape the entire AWS organization
53+
scrape_org: false
54+
# Assume given role in current account
55+
assume_current: false
56+
# Do not scrape current account
57+
do_not_scrape_current: false
58+
```
59+
60+
#### Google Cloud
61+
62+
```yaml
63+
sources:
64+
gcp:
65+
# GCP service account file(s)
66+
service_account: []
67+
# GCP project(s)
68+
project: []
69+
```
70+
71+
#### Kubernetes
72+
73+
```yaml
74+
sources:
75+
k8s:
76+
# Configure access via kubeconfig files.
77+
# Structure:
78+
# - path: "/path/to/kubeconfig"
79+
# all_contexts: false
80+
# contexts: ["context1", "context2"]
81+
config_files: []
82+
# Alternative: configure access to k8s clusters directly in the config.
83+
# Structure:
84+
# - name: 'k8s-cluster-name'
85+
# certificate_authority_data: 'CERT'
86+
# server: 'https://k8s-cluster-server.example.com'
87+
# token: 'TOKEN'
88+
configs: []
89+
```
90+
91+
#### DigitalOcean
92+
93+
```yaml
94+
sources:
95+
digitalocean:
96+
# DigitalOcean API tokens for the teams to be collected
97+
api_tokens: []
98+
# DigitalOcean Spaces access keys for the teams to be collected, separated by colons
99+
spaces_access_keys: []
100+
```
101+
102+
### Destinations
30103
31104
#### SQLite
32105
33-
```
106+
```yaml
34107
destinations:
35-
sqlite:
36-
database: /path/to/database.db
108+
sqlite:
109+
database: /path/to/database.db
37110
```
38111
39112
#### PostgreSQL
40113
41-
```
114+
```yaml
42115
destinations:
43-
postgresql:
44-
host: 127.0.0.1
45-
port: 5432
46-
user: cloud2sql
47-
password: changeme
48-
database: cloud2sql
49-
args:
50-
key: value
116+
postgresql:
117+
host: 127.0.0.1
118+
port: 5432
119+
user: cloud2sql
120+
password: changeme
121+
database: cloud2sql
122+
args:
123+
key: value
51124
```
52125
53126
#### MySQL
54127
55-
```
128+
```yaml
56129
destinations:
57-
mysql:
58-
host: 127.0.0.1
59-
port: 3306
60-
user: cloud2sql
61-
password: changeme
62-
database: cloud2sql
63-
args:
64-
key: value
130+
mysql:
131+
host: 127.0.0.1
132+
port: 3306
133+
user: cloud2sql
134+
password: changeme
135+
database: cloud2sql
136+
args:
137+
key: value
65138
```
66139
67140
#### MariaDB
68141
69-
```
142+
```yaml
70143
destinations:
71-
mariadb:
72-
host: 127.0.0.1
73-
port: 3306
74-
user: cloud2sql
75-
password: changeme
76-
database: cloud2sql
77-
args:
78-
key: value
144+
mariadb:
145+
host: 127.0.0.1
146+
port: 3306
147+
user: cloud2sql
148+
password: changeme
149+
database: cloud2sql
150+
args:
151+
key: value
79152
```
80153
81154
#### Snowflake
82155
83-
```
156+
```yaml
84157
destinations:
85-
snowflake:
86-
host: myorg-myaccount
87-
user: cloud2sql
88-
password: changeme
89-
database: cloud2sql/public
90-
args:
91-
warehouse: compute_wh
92-
role: accountadmin
158+
snowflake:
159+
host: myorg-myaccount
160+
user: cloud2sql
161+
password: changeme
162+
database: cloud2sql/public
163+
args:
164+
warehouse: compute_wh
165+
role: accountadmin
93166
```
94167
95168
#### Apache Parquet
96169
97-
```
170+
```yaml
98171
destinations:
99-
file:
100-
path: /where/to/write/parquet/files/
101-
format: parquet
102-
batch_size: 100_000
172+
file:
173+
path: /where/to/write/parquet/files/
174+
format: parquet
175+
batch_size: 100_000
103176
```
104177
105178
#### CSV
106179
107-
```
180+
```yaml
108181
destinations:
109-
file:
110-
path: /where/to/write/to/csv/files/
111-
format: csv
112-
batch_size: 100_000
182+
file:
183+
path: /where/to/write/to/csv/files/
184+
format: csv
185+
batch_size: 100_000
113186
```
114187
115188
116189
#### My database is not listed here
117190
118-
cloud2sql uses SQLAlchemy to connect to the database. If your database is not listed here, you can check if it is supported in [SQLAlchemy Dialects](https://docs.sqlalchemy.org/en/20/dialects/index.html).
191+
Cloud2SQL uses SQLAlchemy to connect to the database. If your database is not listed here, you can check if it is supported in [SQLAlchemy Dialects](https://docs.sqlalchemy.org/en/20/dialects/index.html).
119192
Install the relevant driver and use the connection string from the documentation.
120193
121194
#### Example
@@ -127,6 +200,8 @@ The example uses our AWS default credentials and the default kubernetes config.
127200
cloud2sql --config config-example.yaml
128201
```
129202

203+
For a more in-depth example, check out our [blog post](https://resoto.com/blog/2022/12/21/installing-cloud2sql).
204+
130205
## Local Development
131206

132207
Create a local development environment with the following command:

‎misc/cloud2sql.gif‎

598 KB
Loading

0 commit comments

Comments
 (0)