You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+33-25Lines changed: 33 additions & 25 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,8 +14,7 @@ pip install local-storage-utils
14
14
15
15
This installs the `lsu` CLI.
16
16
17
-
Installing from TestPyPI (for dry-runs)
18
-
-------------------------------------
17
+
## Installing from TestPyPI (for dry-runs)
19
18
20
19
If you want to test publishing to TestPyPI and install the package from the test index, prefer doing that inside a virtual environment. This avoids the "externally-managed-environment" / PEP 668 error you saw when trying to install system-wide on Debian/Ubuntu.
21
20
@@ -39,21 +38,26 @@ Notes:
39
38
40
39
## Install (from source)
41
40
41
+
```bash
42
42
git clone <this-repo-url>
43
43
cd local-storage-utils
44
44
python3 -m venv .venv
45
45
source .venv/bin/activate
46
46
python -m pip install -U pip
47
47
pip install -e .
48
+
```
48
49
49
50
### Troubleshooting local installs
51
+
50
52
- If you see "Command 'python' not found", use `python3 -m venv .venv` (above). Inside the venv, `python` will point to Python 3.
51
53
- If you see "externally-managed-environment", you are attempting a system-wide install. Always install into a virtual environment:
The keys above map directly to CLI flags (CLI flags override values in `config.yaml`). Omit any option to use sensible defaults.
94
98
95
-
# local-storage-utils — Quickstart
99
+
## Quickstart
100
+
101
+
Lightweight utilities for analyzing and cleaning Datastore (Firestore in Datastore mode). Works with the Datastore emulator for local integration testing or GCP when using Application Default Credentials.
96
102
97
-
Lightweight utilities for analyzing and cleaning Datastore (Firestore in Datastore mode). Works with the
98
-
Datastore emulator for local integration testing or GCP when using Application Default Credentials.
103
+
### Quick overview
99
104
100
-
Quick overview
101
-
- CLI: run commands via `python3 cli.py <command>` (or install the package and use the entrypoint).
102
-
- Makefile: convenience targets are provided to create a venv, install deps, and run tests locally.
105
+
- CLI: run commands via `python3 cli.py <command>` (or install the package and use the entrypoint).
106
+
- Makefile: convenience targets are provided to create a venv, install deps, and run tests locally.
103
107
104
-
Quickstart (from source)
108
+
### Quickstart (from source)
105
109
```bash
106
110
git clone <this-repo-url>
107
111
cd local-storage-utils
@@ -112,14 +116,15 @@ pip install -U pip
112
116
pip install -e .
113
117
```
114
118
115
-
Makefile shortcuts
116
-
- `make venv`— create `.venv` and install package in editable mode
117
-
- `make unit`— run fast unit tests
118
-
- `make integration`— run integration tests (starts/seeds emulator when configured)
119
+
### Makefile shortcuts
120
+
121
+
- `make venv`— create `.venv` and install package in editable mode
122
+
- `make unit`— run fast unit tests
123
+
- `make integration`— run integration tests (starts/seeds emulator when configured)
119
124
120
125
Use these targets to get a working dev environment quickly.
- The seeder accepts `SEED_COUNT` and `SEED_NS_COUNT` env vars to increase dataset size for perf tests.
143
150
144
151
Run integration tests:
152
+
145
153
```bash
146
154
# create venv and install deps (see Quickstart), then:
147
155
make integration
148
156
```
149
157
150
-
Development & tests
158
+
### Development & tests
159
+
151
160
- Run unit tests:
152
161
- `make unit`(fast)
153
162
- Run full test suite locally:
154
163
- `make integration`
155
164
156
-
Publishing
157
-
-------
165
+
## Publishing
158
166
159
167
This project uses the `release` workflow to publish releases to PyPI. Follow the packaging tutorial for a complete guide on packaging and publishing: https://packaging.python.org/en/latest/tutorials/packaging-projects/
160
168
161
169
We support publishing to either TestPyPI (for dry runs) or the real PyPI. The workflow can be triggered automatically on pushes to `main` or manually via the Actions UI (use the "Run workflow" button). When you run it manually you can set the `publish_target` input to `testpypi` to publish to TestPyPI instead of PyPI.
162
170
163
-
Secrets and tokens
171
+
### Secrets and tokens
164
172
- For production publishing to the real PyPI, set the repository secret named `PYPI_API_TOKEN` with a PyPI API token.
165
173
- For test publishing to TestPyPI, set the repository secret named `TEST_PYPI_API_TOKEN` with a TestPyPI API token.
166
174
167
175
The release workflow selects the appropriate token based on the `publish_target` input. Use TestPyPI first to validate packaging and metadata before publishing to the real index.
168
176
169
-
Notes
177
+
## Notes
178
+
170
179
- `sample_size`bounds per-kind/group analysis to avoid scanning entire datasets. Set to 0 or `null` in config to disable sampling.
171
180
- `enable_parallel`(default true) enables multi-threaded processing during analysis and deletion; set to false to force single-threaded behavior.
172
181
173
182
If you'd like a short walkthrough or to change the default Makefile targets, tell me what you'd prefer and I can adjust the README or Makefile.
0 commit comments