You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+55-25Lines changed: 55 additions & 25 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,8 +14,7 @@ pip install local-storage-utils
14
14
15
15
This installs the `lsu` CLI.
16
16
17
-
Installing from TestPyPI (for dry-runs)
18
-
-------------------------------------
17
+
## Installing from TestPyPI (for dry-runs)
19
18
20
19
If you want to test publishing to TestPyPI and install the package from the test index, prefer doing that inside a virtual environment. This avoids the "externally-managed-environment" / PEP 668 error you saw when trying to install system-wide on Debian/Ubuntu.
21
20
@@ -39,21 +38,48 @@ Notes:
39
38
40
39
## Install (from source)
41
40
41
+
```bash
42
42
git clone <this-repo-url>
43
43
cd local-storage-utils
44
44
python3 -m venv .venv
45
45
source .venv/bin/activate
46
46
python -m pip install -U pip
47
47
pip install -e .
48
+
```
48
49
49
50
### Troubleshooting local installs
51
+
50
52
- If you see "Command 'python' not found", use `python3 -m venv .venv` (above). Inside the venv, `python` will point to Python 3.
51
53
- If you see "externally-managed-environment", you are attempting a system-wide install. Always install into a virtual environment:
The keys above map directly to CLI flags (CLI flags override values in `config.yaml`). Omit any option to use sensible defaults.
94
120
95
-
# local-storage-utils — Quickstart
121
+
## Quickstart
122
+
123
+
Lightweight utilities for analyzing and cleaning Datastore (Firestore in Datastore mode). Works with the Datastore emulator for local integration testing or GCP when using Application Default Credentials.
96
124
97
-
Lightweight utilities for analyzing and cleaning Datastore (Firestore in Datastore mode). Works with the
98
-
Datastore emulator for local integration testing or GCP when using Application Default Credentials.
125
+
### Quick overview
99
126
100
-
Quick overview
101
-
- CLI: run commands via `python3 cli.py <command>` (or install the package and use the entrypoint).
102
-
- Makefile: convenience targets are provided to create a venv, install deps, and run tests locally.
127
+
- CLI: run commands via `python3 cli.py <command>` (or install the package and use the entrypoint).
128
+
- Makefile: convenience targets are provided to create a venv, install deps, and run tests locally.
103
129
104
-
Quickstart (from source)
130
+
### Quickstart (from source)
105
131
```bash
106
132
git clone <this-repo-url>
107
133
cd local-storage-utils
@@ -112,14 +138,15 @@ pip install -U pip
112
138
pip install -e .
113
139
```
114
140
115
-
Makefile shortcuts
116
-
- `make venv`— create `.venv` and install package in editable mode
117
-
- `make unit`— run fast unit tests
118
-
- `make integration`— run integration tests (starts/seeds emulator when configured)
141
+
### Makefile shortcuts
142
+
143
+
- `make venv`— create `.venv` and install package in editable mode
144
+
- `make unit`— run fast unit tests
145
+
- `make integration`— run integration tests (starts/seeds emulator when configured)
119
146
120
147
Use these targets to get a working dev environment quickly.
- The seeder accepts `SEED_COUNT` and `SEED_NS_COUNT` env vars to increase dataset size for perf tests.
143
172
144
173
Run integration tests:
174
+
145
175
```bash
146
176
# create venv and install deps (see Quickstart), then:
147
177
make integration
148
178
```
149
179
150
-
Development & tests
180
+
### Development & tests
181
+
151
182
- Run unit tests:
152
183
- `make unit`(fast)
153
184
- Run full test suite locally:
154
185
- `make integration`
155
186
156
-
Publishing
157
-
-------
187
+
## Publishing
158
188
159
189
This project uses the `release` workflow to publish releases to PyPI. Follow the packaging tutorial for a complete guide on packaging and publishing: https://packaging.python.org/en/latest/tutorials/packaging-projects/
160
190
161
191
We support publishing to either TestPyPI (for dry runs) or the real PyPI. The workflow can be triggered automatically on pushes to `main` or manually via the Actions UI (use the "Run workflow" button). When you run it manually you can set the `publish_target` input to `testpypi` to publish to TestPyPI instead of PyPI.
162
192
163
-
Secrets and tokens
193
+
### Secrets and tokens
164
194
- For production publishing to the real PyPI, set the repository secret named `PYPI_API_TOKEN` with a PyPI API token.
165
195
- For test publishing to TestPyPI, set the repository secret named `TEST_PYPI_API_TOKEN` with a TestPyPI API token.
166
196
167
197
The release workflow selects the appropriate token based on the `publish_target` input. Use TestPyPI first to validate packaging and metadata before publishing to the real index.
168
198
169
-
Notes
199
+
## Notes
200
+
170
201
- `sample_size`bounds per-kind/group analysis to avoid scanning entire datasets. Set to 0 or `null` in config to disable sampling.
171
202
- `enable_parallel`(default true) enables multi-threaded processing during analysis and deletion; set to false to force single-threaded behavior.
172
203
173
204
If you'd like a short walkthrough or to change the default Makefile targets, tell me what you'd prefer and I can adjust the README or Makefile.
0 commit comments