Skip to content
Open

Rc1 #1765

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
265 commits
Select commit Hold shift + click to select a range
706343b
Not tracking .env since this file is volatile
Oct 9, 2024
f0dd620
Adding deepfreeze_elements.py
Oct 14, 2024
a65e5c7
Updated Schema
Oct 18, 2024
88b66a8
Fixing some syntactical things.
wortmanb Oct 28, 2024
3bf04c9
Reorganize imports
Oct 28, 2024
2b2eeaa
Implement as group and subcommand
wortmanb Oct 28, 2024
9e391d1
Merge branch 'master' of github.com:wortmanb/curator
Oct 28, 2024
594c406
Align options
Oct 28, 2024
6465751
Fixing mistakes
Oct 28, 2024
7c18262
Singleton setup updates
wortmanb Nov 1, 2024
85100da
Click & Doco updates
wortmanb Nov 1, 2024
b89cbc6
More plumbing
wortmanb Nov 1, 2024
dc6037b
Formatting, mostly
wortmanb Nov 1, 2024
f469020
Added more feedback on --setup
wortmanb Nov 1, 2024
61a8421
Added a TODO
wortmanb Nov 6, 2024
54ae0a9
Update README.rst
wortmanb Nov 13, 2024
3a0af36
Merge branch 'elastic:master' into master
wortmanb Nov 13, 2024
d247c38
Testing updates
Nov 14, 2024
38220c6
Merge branch 'master' of github.com:wortmanb/curator
Nov 14, 2024
3fc9ab9
Adding boto3 to dependencies. Sorry, Aaron.
Jan 15, 2025
33ecbfb
Adding deepfreeze group and multiple actions
Jan 15, 2025
532adb9
Refactored deepfreeze
Jan 15, 2025
5a5e366
Updates & fixes to click plumbing
Jan 15, 2025
6396899
Added save_settings
Jan 16, 2025
be61cc3
Still trying to get the plumbing right
Jan 17, 2025
7fb4934
Fix extra paren
Jan 24, 2025
891ab82
Fix action dispatching
Jan 24, 2025
dab6f38
Setup works
Jan 26, 2025
22ac9a3
Update README.rst
wortmanb Jan 27, 2025
5612c51
Today's big update
Jan 27, 2025
e987730
Merge branch 'master' of github.com:wortmanb/curator
Jan 27, 2025
d33c035
Purging old repos
Jan 27, 2025
9218ef0
Fixed selection of the latest repo
Jan 28, 2025
a3961da
Code cleanup
Jan 28, 2025
a7e68fe
Trying to make Flake happy
Jan 28, 2025
ebfaf19
Doco & formatting fixes
Jan 28, 2025
1732905
Added two more dataclasses
Jan 28, 2025
fb350c3
First unit tests
Jan 29, 2025
cfaa6d3
More tests
Jan 29, 2025
08ce133
Beginning to notate types
Jan 29, 2025
f803c75
Test suite for DF Setup
Jan 29, 2025
c5c6ca0
Muzzling the linter
Jan 29, 2025
b9cb94e
test suite for Rotate
Jan 29, 2025
14b14c1
Early work on thawing
Jan 30, 2025
9bb6317
Adding ruff.toml
Jan 30, 2025
02bd510
Making Ruff happy
Jan 30, 2025
3da6be7
Mostly added type hints
Jan 30, 2025
63a1cdb
Added s3client.py
Jan 31, 2025
df861e9
Refactored to use new S3Client class
Jan 31, 2025
a9c8727
Unit test for S3Client
Feb 1, 2025
41bbded
Removing unit test for obsolete method
Feb 3, 2025
5cca7c8
Fixed some unit tests
Feb 3, 2025
4800010
Cleaned up imports
Feb 3, 2025
21a3779
Updates to ThawedRepo
Feb 3, 2025
1ca7aab
Test code to determine earliest/latest @timestamp in repo
Feb 4, 2025
4acdb9d
Added get_timestamp_range
Feb 5, 2025
73abe95
Expanded use of new get_timestamp_range
Feb 5, 2025
9f14b33
Re-templating deepfreeze integration testing
Feb 5, 2025
e2ad76d
rough outline of setup integration test
Feb 5, 2025
bc74a84
Removing repo_time_tester.py
Feb 5, 2025
87db906
Templated new methods in s3client.py
Feb 5, 2025
f1e4e78
Fix unused import in integration test
Feb 5, 2025
45a6004
Implement index thawing
Feb 5, 2025
7818075
Doco fixes
Feb 5, 2025
eca85a4
Updates to latest methods
Feb 6, 2025
7273d53
New tests
Feb 6, 2025
cd1ea0a
Merge branch 'master' of github.com:elastic/curator
Oct 10, 2025
7e5ebb1
Status action
Feb 7, 2025
613ec4f
Refactored
Feb 7, 2025
186c348
Add rich to the dependency list
Feb 7, 2025
85654ce
Switching to rich for interactive output
Feb 8, 2025
7477988
Adding to .gitignore
Feb 8, 2025
2abeddd
Another script of mine that doesn't need sharing
Feb 8, 2025
86f0629
Type hints
Feb 9, 2025
ec54d59
Repository record-keeping
Feb 8, 2025
e508062
Starting to flesh out thawing
Feb 8, 2025
aeb48cd
Adding cluster name
Feb 10, 2025
14eb009
Another script of mine that doesn't need sharing
Feb 8, 2025
4f42301
Remove stale method
Feb 8, 2025
267a15e
Removing ancient version of thaw.py
Feb 10, 2025
a57d1af
Merge branch 'master' of github.com:wortmanb/curator
Feb 8, 2025
a5ce331
FIxing options, testing setup
Feb 10, 2025
471c509
Adding push to glacier
Feb 10, 2025
6c826c2
Refafctored a couple utility methods
Feb 10, 2025
fc76805
Add check_restore_status method
Feb 11, 2025
b6e0b21
This is worth keeping
Feb 11, 2025
f3725fb
Adding Remount
Feb 11, 2025
dd446a5
Refactored object name
Feb 11, 2025
05badf0
Fleshing out new actions
Feb 11, 2025
81034e0
Updated unit tests for utility functions
Feb 11, 2025
d898eec
Tests now work
Feb 11, 2025
bf5c5e1
Switching back to dictionary-like interface
Feb 11, 2025
e6f2f1d
Added list_objects method
Feb 13, 2025
ee9682b
Loads of updates
Feb 13, 2025
249650d
Update Repository to construct from name or dictionary
Feb 16, 2025
f5beefd
Add methods for use in testing, mostly
Feb 16, 2025
325612d
Added method to update mounted repo date ranges
Feb 18, 2025
8bf440b
Changes to methods with better signatures
Feb 18, 2025
5d211f4
Docstring updates
Feb 18, 2025
a439d57
Merging change to modularize deepfreee
Feb 24, 2025
664fce2
Removing unneeded import
Feb 25, 2025
7548c69
Minor tweaks
Feb 25, 2025
8c65d69
Adding exceptions specific to Deepfreeze
Feb 25, 2025
dc6bd02
Expand ensure_settings_index definition
Feb 25, 2025
35b2251
Flesh out integration tests for rotate
Feb 25, 2025
c017f74
Adding more exceptions
Feb 25, 2025
adff5cd
Adding readme to track work still to be done
Feb 27, 2025
6b095ad
Move the geneeration of new Repository objects outside the object
Feb 27, 2025
a566227
Fixed issues with rotate testing
Feb 28, 2025
b4a1fa2
Increase interval to remove race conditions
Feb 28, 2025
0f92edf
Updates from expanded integration testing
Feb 28, 2025
b6dbaa2
New test cases to better cover repo/bucket issues
Feb 28, 2025
f7fad72
Removing integration test for status
Feb 28, 2025
fe6b8e9
Templates which can give success right now
Feb 28, 2025
0679423
Spelling matters
Mar 3, 2025
1234a27
Add snapshot count to repository status
Mar 3, 2025
85e4871
Update repos after creating the latest, not before
Mar 4, 2025
006e5eb
Various testing updates
Mar 4, 2025
eb84d9c
Updates to Repository utilities
Mar 4, 2025
fae0d47
Type hints
Feb 9, 2025
565271b
Merge branch 'deepfreeze' of github.com:wortmanb/curator into deepfreeze
Mar 31, 2025
4fc0bff
Removing unused imports & assignments
Apr 7, 2025
3516d6a
Restored setup functionality with boundaries
Apr 20, 2025
e5c3468
Fixing isoformat error
Apr 21, 2025
f902b9a
Fix repo searching
Apr 22, 2025
4dd1165
Fixing some issues with repo rotation
Apr 22, 2025
4c9fddc
Added single-letter options to al actions
Apr 22, 2025
1899b27
Adding testing setup scripts
Apr 22, 2025
217cb77
Documentation update
Apr 25, 2025
3e43c15
Fixes for date comparisons
Apr 25, 2025
203f815
Fixed AWS code, repo removal
Apr 25, 2025
35f40f9
Fixing date updating
Apr 25, 2025
5bf7b99
Do this query using name, not id
Apr 25, 2025
cd15ac3
Remove unnecessary import
Apr 25, 2025
fc3701c
Fix missing base_path and bucket
Apr 26, 2025
dc4b1bb
Fixed repo with null start/end
Apr 26, 2025
42accb0
Clean up
Apr 26, 2025
5920a5f
Facilitate screening repos by mounted status
Apr 28, 2025
6b1024a
Removing all but the options I've been working on most. We'll add cap…
wortmanb Apr 29, 2025
cecef8c
Bugfix; change approach to unmounting failures
Apr 28, 2025
252b61b
Merge branch 'dev' of github.com:wortmanb/curator into dev
Apr 29, 2025
25daf1d
Update to ILM policy for faster testing
May 3, 2025
1b2af2c
Bugfixes
May 4, 2025
4856fb4
Updates to reflect renaming of this function
May 4, 2025
c52d10e
Removing actions
May 4, 2025
d1085d5
More removals
May 4, 2025
89e70c5
Cleaning up
May 4, 2025
f50793a
More aggressive ILM policy
May 4, 2025
94e9950
Change these warnings to infos
May 4, 2025
38cfe68
Removing unneeded import
May 4, 2025
4b21fc0
More aggressive ILM policy for testing & demo
May 4, 2025
52c454d
Don't limit ourselves needlessly
May 4, 2025
4dd7b0a
Removing FIXMEs and TODOs that have been handled
May 4, 2025
1ca6bc9
More code removal
May 4, 2025
91d652e
Implement S3Client as an ABC
May 5, 2025
862765b
WIP on Glacier
May 14, 2025
e750c05
Updating to fix problem with bucket/path contents not aging to Glacier.
May 14, 2025
6bc926f
Removing an unnecessary import
May 14, 2025
6fafcff
Updated to work with nginx reverse proxy
May 14, 2025
99f4e98
Fixing misspellings of "Deepfreeze"
Oct 10, 2025
ebb8cdf
Removing files used for local testing
Oct 10, 2025
9979909
Fixing missing commas
Oct 10, 2025
aff2afe
Unit testing
Oct 10, 2025
04e85b7
Remove warnings, re-add actions to those who don't use filters
Oct 12, 2025
19fa35a
Fix repo date range calculation & storage
Oct 13, 2025
6cc4363
First pass at Thaw action
Oct 13, 2025
b476802
Adding thaw id management and listing
Oct 13, 2025
627df7c
Post-rebase updates
Oct 13, 2025
296125e
Date logic updates
Oct 13, 2025
125a1ea
Add cleanup action
Oct 13, 2025
a541e84
Add --limit to status (show only _n_ repositories)
Oct 13, 2025
8d7f226
Fixup cleanup CLI and tweak help messages
Oct 13, 2025
b006304
Cleanup indices as well as repositories
Oct 13, 2025
e57f482
Added refreeze
Oct 13, 2025
4687957
Added refreeze by thaw_id, and get confirmation before destructive ac…
Oct 13, 2025
00e1186
How to manage the various git branches
Oct 13, 2025
e3671a8
Merge branch 'RC1' of github.com:wortmanb/curator into RC1
Oct 13, 2025
b617705
Improve error messages during deepfreeze setup
Oct 13, 2025
8eb289c
Improve error messages during deepfreeze setup
Oct 13, 2025
86c46b0
[BF] Handle multiple policies during rotation
Oct 14, 2025
01e70d4
Update unit tests
Oct 14, 2025
0bb97ad
Merge branch 'RC2' of github.com:wortmanb/curator into RC2
Oct 15, 2025
bf767ae
Fixed detection of initial repo(s)
Oct 15, 2025
d9942fa
Merge branch 'RC2' of github.com:wortmanb/curator into RC2
Oct 15, 2025
d79ea7d
Type hints
Oct 15, 2025
02bda0c
Escape bucket paths and other fixes
Oct 15, 2025
6f61d10
First pass at Thaw action
Oct 13, 2025
788b0fc
Adding thaw id management and listing
Oct 13, 2025
7a44dcb
Post-rebase updates
Oct 13, 2025
52851ac
Date logic updates
Oct 13, 2025
946dc15
Add cleanup action
Oct 13, 2025
9a2fdcc
Add --limit to status (show only _n_ repositories)
Oct 13, 2025
462fad4
Fixup cleanup CLI and tweak help messages
Oct 13, 2025
947a687
Cleanup indices as well as repositories
Oct 13, 2025
a6c6fab
Added refreeze
Oct 13, 2025
964dad3
Improve error messages during deepfreeze setup
Oct 13, 2025
a1fbc38
Added refreeze by thaw_id, and get confirmation before destructive ac…
Oct 13, 2025
658cf14
[BF] Handle multiple policies during rotation
Oct 14, 2025
a5df42e
Improve error messages during deepfreeze setup
Oct 13, 2025
26ef48b
Update unit tests
Oct 14, 2025
83ba522
Merge branch 'RC2' of github.com:wortmanb/curator into RC2
Oct 15, 2025
9670c9b
Fixed detection of initial repo(s)
Oct 15, 2025
7792dd2
Merge branch 'RC2' of github.com:wortmanb/curator into RC2
Oct 15, 2025
af5e282
Type hints
Oct 15, 2025
a0f7b8a
Escape bucket paths and other fixes
Oct 15, 2025
d4711c4
Merge branch 'RC2' of github.com:wortmanb/curator into RC2
Oct 15, 2025
9427287
Added doc to help explain the process
Oct 15, 2025
b5e42cc
Fail quickly if no ILM policies reference the repo
Oct 15, 2025
51e8cc7
Improve messaging during thaw operations
Oct 16, 2025
b5d3932
Fix display of thawing and thawed repos in status output
Oct 16, 2025
50ad269
Add section selections & porcelain output
Oct 16, 2025
bee6f82
Show full Request ID
Oct 16, 2025
5cef7ca
Print thaw ID after async creation
Oct 17, 2025
43aece5
Cleanup old thaw IDs
Oct 17, 2025
bfcbe76
Fix repo thaw check
Oct 17, 2025
6a1f5cb
Restore snapshots and indices, too
Oct 17, 2025
fdfe60b
Speed improvements
Oct 17, 2025
5f8f805
Fix thawed repo detection in status
Oct 17, 2025
e207590
Move thawed to their own section in status
Oct 17, 2025
04585af
Protect thawed repos during rotation
Oct 17, 2025
39e70b9
Add date range to thaw --list
Oct 17, 2025
57e3b05
thaw --check-status without an ID checks all
Oct 17, 2025
88f1cae
Critical Bugfix
Oct 17, 2025
c3a11b9
Fixing logic for updating date ranges when thawed indices are mounted
Oct 18, 2025
3d7f7bd
Fixup script for bad repo dates
Oct 18, 2025
b204b5f
New state machine for thawed repos
Oct 18, 2025
46c2a59
Implement new state machine
Oct 18, 2025
824eae8
Fix repo state issue
Oct 18, 2025
37c5f47
Adding "active" state to state machine
Oct 18, 2025
0481b62
First cut at new refreeze operation
Oct 18, 2025
3da0254
Add default of refreeze all, with confirmation
Oct 18, 2025
8eeb59d
Fix mounting repos in --check-status
Oct 20, 2025
6ab4bce
Fixing index selection for mounting after thawing
Oct 20, 2025
0aa8504
--porcelain added throughout
Oct 20, 2025
6d6b136
Output consistency
Oct 20, 2025
70f531a
Fix date range update logic in rotate
Oct 20, 2025
b79e27f
Start on new integration tests for thaw
Oct 20, 2025
3ef79b1
Fold & wrap text in Rich output
Oct 21, 2025
cf6ef8f
Add short option for porcelain
Oct 21, 2025
07fbb93
Improved data stream detection during thaw operations
Oct 21, 2025
935f19a
Various pre-tests and other improvements
Oct 21, 2025
587eb26
Check for plugin
Oct 21, 2025
7bb9ef5
Update progress display during --check-status
Oct 23, 2025
20eb496
Change default thaw duration from 7d to 30d
Oct 23, 2025
695238b
Updated thaw logic
Oct 23, 2025
f99c8fa
Updates to refreeze
Oct 23, 2025
1b2eb15
Updating ILM processing for thawed indices
Oct 23, 2025
b88f7ba
Default thaw list to only open thaws
Oct 23, 2025
cf60a93
Differentiate completed & refrozen
Oct 23, 2025
9519ad2
Fixes to cleanup
Oct 27, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -176,3 +176,13 @@ cython_debug/
#.idea/

.vscode

# pyenv version
.python-version

# vim backup files
*~
repo_time_tester.py
reset.sh
seed_data_to_ds.py
docker_test/scripts/license.json
78 changes: 78 additions & 0 deletions GIT_WORKFLOW.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Git Workflow for RC Development

## Context
After submitting PR from RC1 branch to upstream, continuing development that builds upon RC1 changes.

## Recommended Approach: Branch from RC1

### Create new branch from RC1
```bash
git checkout -b RC2 RC1
```

### Benefits
- Continue developing immediately with the RC1 foundation
- Keep RC1 frozen for the PR review process
- Maintain flexibility for PR changes

### Workflow

1. **Develop new features on RC2**
```bash
git checkout RC2
# Make changes, commit as normal
```

2. **If RC1 needs changes from PR review:**
```bash
# Switch back to RC1
git checkout RC1

# Make requested changes
# Commit changes

# Push updates to PR
git push origin RC1
```

3. **Sync RC2 with updated RC1:**
```bash
git checkout RC2
git rebase RC1
```

4. **After RC1 is merged upstream:**
```bash
# Sync master with upstream
git checkout master
git fetch upstream
git merge upstream/master
git push origin master

# Rebase RC2 onto master
git checkout RC2
git rebase master

# Clean up merged RC1 branch
git branch -d RC1
git push origin --delete RC1
```

## Alternative Naming
Instead of RC2, consider more descriptive names:
- `deepfreeze-phase2`
- `feature/deepfreeze-enhancements`
- `RC2-deepfreeze-completion`

## Other Approaches Considered

### New branch from master (for independent work)
```bash
git checkout master
git checkout -b feature/new-feature
```
**Use when:** Next work is independent of RC1 changes

### Wait for PR merge (most conservative)
Wait until PR is accepted, sync with upstream, then branch
**Use when:** No urgency and want clean linear history
3 changes: 3 additions & 0 deletions README.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
.. _readme:

**THIS FORK OF ELASTIC/CURATOR REPRESENTS A WORK-IN-PROGRESS AND SHOULD NOT BE CONSIDERED "RUNNABLE". IT IS STILL IN DEVELOPMENT.**

**HERE THERE BE TYGERS.**

Curator
=======
Expand Down
45 changes: 27 additions & 18 deletions curator/actions/__init__.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
"""Use __init__ to make these not need to be nested under lowercase.Capital"""

from curator.actions.alias import Alias
from curator.actions.allocation import Allocation
from curator.actions.close import Close
from curator.actions.cluster_routing import ClusterRouting
from curator.actions.cold2frozen import Cold2Frozen
from curator.actions.create_index import CreateIndex
from curator.actions.deepfreeze import Cleanup, Deepfreeze, Refreeze, Rotate, Setup, Status, Thaw
from curator.actions.delete_indices import DeleteIndices
from curator.actions.forcemerge import ForceMerge
from curator.actions.index_settings import IndexSettings
Expand All @@ -13,24 +15,31 @@
from curator.actions.replicas import Replicas
from curator.actions.rollover import Rollover
from curator.actions.shrink import Shrink
from curator.actions.snapshot import Snapshot, DeleteSnapshots, Restore
from curator.actions.snapshot import DeleteSnapshots, Restore, Snapshot

CLASS_MAP = {
'alias' : Alias,
'allocation' : Allocation,
'close' : Close,
'cluster_routing' : ClusterRouting,
'cold2frozen': Cold2Frozen,
'create_index' : CreateIndex,
'delete_indices' : DeleteIndices,
'delete_snapshots' : DeleteSnapshots,
'forcemerge' : ForceMerge,
'index_settings' : IndexSettings,
'open' : Open,
'reindex' : Reindex,
'replicas' : Replicas,
'restore' : Restore,
'rollover' : Rollover,
'snapshot' : Snapshot,
'shrink' : Shrink,
"alias": Alias,
"allocation": Allocation,
"cleanup": Cleanup,
"close": Close,
"cluster_routing": ClusterRouting,
"cold2frozen": Cold2Frozen,
"create_index": CreateIndex,
"deepfreeze": Deepfreeze,
"delete_indices": DeleteIndices,
"delete_snapshots": DeleteSnapshots,
"forcemerge": ForceMerge,
"index_settings": IndexSettings,
"open": Open,
"refreeze": Refreeze,
"reindex": Reindex,
"replicas": Replicas,
"restore": Restore,
"rollover": Rollover,
"shrink": Shrink,
"snapshot": Snapshot,
"setup": Setup,
"rotate": Rotate,
"status": Status,
"thaw": Thaw,
}
15 changes: 15 additions & 0 deletions curator/actions/deepfreeze/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Deepfreeze Module

## To Do
- [ ] Fix generation of Repository using utility method instead of constructor
- [ ] Ensure dry_run is respected throughout
- [ ] Ensure Repository updates in the STATUS_INDEX are happening properly and reliably


## To Fix


## Author

Deepfreeze was written by Bret Wortman ([email protected]) but it's built on
the foundation of Curator, which is the work of Aaron Mildenstein and many others.
51 changes: 51 additions & 0 deletions curator/actions/deepfreeze/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
"""Deepfreeze actions module"""

from .constants import PROVIDERS, SETTINGS_ID, STATUS_INDEX
from .exceptions import (
ActionException,
DeepfreezeException,
MissingIndexError,
MissingSettingsError,
)
from .cleanup import Cleanup
from .helpers import Deepfreeze, Repository, Settings
from .refreeze import Refreeze
from .rotate import Rotate
from .setup import Setup
from .status import Status
from .thaw import Thaw
from .utilities import (
check_restore_status,
create_repo,
decode_date,
ensure_settings_index,
find_repos_by_date_range,
get_all_indices_in_repo,
get_all_repos,
get_matching_repo_names,
get_matching_repos,
get_next_suffix,
get_repositories_by_names,
get_settings,
get_thaw_request,
get_timestamp_range,
list_thaw_requests,
mount_repo,
push_to_glacier,
save_settings,
save_thaw_request,
unmount_repo,
update_repository_date_range,
)

CLASS_MAP = {
"cleanup": Cleanup,
"deepfreeze": Deepfreeze,
"refreeze": Refreeze,
"repository": Repository,
"settings": Settings,
"setup": Setup,
"rotate": Rotate,
"status": Status,
"thaw": Thaw,
}
Loading