Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
4d5c199
models update (version with commented has_many for deleted_*)
karoltravis Mar 14, 2022
71dfd6f
logs and abuses files removed
karoltravis Mar 15, 2022
a931abe
models.rb
karoltravis Mar 15, 2022
572da02
non-crutial features removed
karoltravis Mar 15, 2022
17e0e53
new schema.sql and few fixes
karoltravis Mar 15, 2022
68ddf99
further tests accomodation
karoltravis Mar 15, 2022
4eee27a
further tests accomodation, typo with EmailUnsubscribe fixed
karoltravis Mar 17, 2022
d890ddd
proper ids in model_spec
karoltravis Mar 17, 2022
3094d6e
further tests accomodation, expected_dependency_trees
karoltravis Mar 17, 2022
020727b
further tests accomodation, problem with RepoCount solved
karoltravis Mar 17, 2022
99a16b7
new expected files for remove_repo_requests
karoltravis Mar 17, 2022
14ac9a3
remove_repo_builds -> remove_heavy_data_for_repo in backup_spec
karoltravis Mar 18, 2022
e4fcd3a
context when threshold is given
karoltravis Mar 18, 2022
f65c639
better description of heavy data removing in tests
karoltravis Mar 21, 2022
5c5c4c9
further temporarily ignored tests commented
karoltravis Mar 21, 2022
722a61a
tests improved
karoltravis Mar 21, 2022
6d10551
new expected files for remove_repo_builds
karoltravis Mar 21, 2022
fc92401
create_for_repo in repository factory
karoltravis Mar 21, 2022
488c100
new factories wip
karoltravis Mar 23, 2022
a3fc9f7
wip
karoltravis Mar 23, 2022
92133ab
wip
karoltravis Mar 23, 2022
0aabbac
wip
karoltravis Mar 23, 2022
ac4dc99
wip
karoltravis Mar 24, 2022
7d709f5
new factories - all tests pass
karoltravis Mar 24, 2022
a3346fd
README updated
karoltravis Mar 24, 2022
941d092
gemspec updated
karoltravis Mar 24, 2022
2502354
fix
karoltravis Mar 25, 2022
4a8a76a
abort message updated
karoltravis Mar 25, 2022
11adb6f
v0.1.0
karoltravis Mar 25, 2022
d80db67
abort message updated
karoltravis Mar 25, 2022
707b1a3
abort message updated again
karoltravis Mar 25, 2022
eefe679
fix
karoltravis Mar 25, 2022
3709187
tests fixed
karoltravis Mar 28, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
215 changes: 215 additions & 0 deletions Gemfile.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
PATH
remote: .
specs:
travis-backup-for-v3 (0.1.0)
activerecord
bootsnap
pg
pry
rails
tzinfo-data

GEM
remote: https://rubygems.org/
specs:
actioncable (6.1.4.1)
actionpack (= 6.1.4.1)
activesupport (= 6.1.4.1)
nio4r (~> 2.0)
websocket-driver (>= 0.6.1)
actionmailbox (6.1.4.1)
actionpack (= 6.1.4.1)
activejob (= 6.1.4.1)
activerecord (= 6.1.4.1)
activestorage (= 6.1.4.1)
activesupport (= 6.1.4.1)
mail (>= 2.7.1)
actionmailer (6.1.4.1)
actionpack (= 6.1.4.1)
actionview (= 6.1.4.1)
activejob (= 6.1.4.1)
activesupport (= 6.1.4.1)
mail (~> 2.5, >= 2.5.4)
rails-dom-testing (~> 2.0)
actionpack (6.1.4.1)
actionview (= 6.1.4.1)
activesupport (= 6.1.4.1)
rack (~> 2.0, >= 2.0.9)
rack-test (>= 0.6.3)
rails-dom-testing (~> 2.0)
rails-html-sanitizer (~> 1.0, >= 1.2.0)
actiontext (6.1.4.1)
actionpack (= 6.1.4.1)
activerecord (= 6.1.4.1)
activestorage (= 6.1.4.1)
activesupport (= 6.1.4.1)
nokogiri (>= 1.8.5)
actionview (6.1.4.1)
activesupport (= 6.1.4.1)
builder (~> 3.1)
erubi (~> 1.4)
rails-dom-testing (~> 2.0)
rails-html-sanitizer (~> 1.1, >= 1.2.0)
activejob (6.1.4.1)
activesupport (= 6.1.4.1)
globalid (>= 0.3.6)
activemodel (6.1.4.1)
activesupport (= 6.1.4.1)
activerecord (6.1.4.1)
activemodel (= 6.1.4.1)
activesupport (= 6.1.4.1)
activestorage (6.1.4.1)
actionpack (= 6.1.4.1)
activejob (= 6.1.4.1)
activerecord (= 6.1.4.1)
activesupport (= 6.1.4.1)
marcel (~> 1.0.0)
mini_mime (>= 1.1.0)
activesupport (6.1.4.1)
concurrent-ruby (~> 1.0, >= 1.0.2)
i18n (>= 1.6, < 2)
minitest (>= 5.1)
tzinfo (~> 2.0)
zeitwerk (~> 2.3)
ast (2.4.2)
bootsnap (1.9.1)
msgpack (~> 1.0)
brakeman (5.1.2)
builder (3.2.4)
byebug (11.1.3)
coderay (1.1.3)
concurrent-ruby (1.1.9)
crass (1.0.6)
database_cleaner-active_record (2.0.1)
activerecord (>= 5.a)
database_cleaner-core (~> 2.0.0)
database_cleaner-core (2.0.1)
diff-lcs (1.4.4)
erubi (1.10.0)
factory_bot (6.2.0)
activesupport (>= 5.0.0)
ffi (1.15.4)
globalid (0.5.2)
activesupport (>= 5.0)
i18n (1.8.11)
concurrent-ruby (~> 1.0)
jaro_winkler (1.5.4)
listen (3.7.0)
rb-fsevent (~> 0.10, >= 0.10.3)
rb-inotify (~> 0.9, >= 0.9.10)
loofah (2.12.0)
crass (~> 1.0.2)
nokogiri (>= 1.5.9)
mail (2.7.1)
mini_mime (>= 0.1.1)
marcel (1.0.2)
method_source (1.0.0)
mini_mime (1.1.2)
minitest (5.14.4)
msgpack (1.4.2)
nio4r (2.5.8)
nokogiri (1.12.5-x86_64-linux)
racc (~> 1.4)
parallel (1.21.0)
parser (3.0.2.0)
ast (~> 2.4.1)
pg (1.2.3)
pry (0.14.1)
coderay (~> 1.1)
method_source (~> 1.0)
racc (1.6.0)
rack (2.2.3)
rack-test (1.1.0)
rack (>= 1.0, < 3)
rails (6.1.4.1)
actioncable (= 6.1.4.1)
actionmailbox (= 6.1.4.1)
actionmailer (= 6.1.4.1)
actionpack (= 6.1.4.1)
actiontext (= 6.1.4.1)
actionview (= 6.1.4.1)
activejob (= 6.1.4.1)
activemodel (= 6.1.4.1)
activerecord (= 6.1.4.1)
activestorage (= 6.1.4.1)
activesupport (= 6.1.4.1)
bundler (>= 1.15.0)
railties (= 6.1.4.1)
sprockets-rails (>= 2.0.0)
rails-dom-testing (2.0.3)
activesupport (>= 4.2.0)
nokogiri (>= 1.6)
rails-html-sanitizer (1.4.2)
loofah (~> 2.3)
railties (6.1.4.1)
actionpack (= 6.1.4.1)
activesupport (= 6.1.4.1)
method_source
rake (>= 0.13)
thor (~> 1.0)
rainbow (3.0.0)
rake (13.0.6)
rb-fsevent (0.11.0)
rb-inotify (0.10.1)
ffi (~> 1.0)
rspec-core (3.10.1)
rspec-support (~> 3.10.0)
rspec-expectations (3.10.1)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.10.0)
rspec-mocks (3.10.2)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.10.0)
rspec-rails (5.0.2)
actionpack (>= 5.2)
activesupport (>= 5.2)
railties (>= 5.2)
rspec-core (~> 3.10)
rspec-expectations (~> 3.10)
rspec-mocks (~> 3.10)
rspec-support (~> 3.10)
rspec-support (3.10.3)
rubocop (0.75.1)
jaro_winkler (~> 1.5.1)
parallel (~> 1.10)
parser (>= 2.6)
rainbow (>= 2.2.2, < 4.0)
ruby-progressbar (~> 1.7)
unicode-display_width (>= 1.4.0, < 1.7)
rubocop-rspec (1.41.0)
rubocop (>= 0.68.1)
ruby-progressbar (1.11.0)
sprockets (4.0.2)
concurrent-ruby (~> 1.0)
rack (> 1, < 3)
sprockets-rails (3.2.2)
actionpack (>= 4.0)
activesupport (>= 4.0)
sprockets (>= 3.0.0)
thor (1.1.0)
tzinfo (2.0.4)
concurrent-ruby (~> 1.0)
tzinfo-data (1.2021.5)
tzinfo (>= 1.0.0)
unicode-display_width (1.6.1)
websocket-driver (0.7.5)
websocket-extensions (>= 0.1.0)
websocket-extensions (0.1.5)
zeitwerk (2.5.1)

PLATFORMS
x86_64-linux

DEPENDENCIES
brakeman
byebug
database_cleaner-active_record
factory_bot
listen
rspec-rails
rubocop (~> 0.75.1)
rubocop-rspec
travis-backup-for-v3!

BUNDLED WITH
2.3.10
35 changes: 7 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
# README

*travis-backup* is an application that helps with housekeeping and backup for Travis CI database v2.2 and with migration to v3.0 database. By default it removes requests and builds with their corresponding jobs and logs, as long as they are older than given threshold says (and backups them in files, if this option is active). Although it can be also run in special modes to perform other specific tasks.
*travis-backup-for-v3* is an application that helps with housekeeping and backup for Travis CI database v3.0. It removes requests and builds with their dependencies, as long as they are older than given threshold says.

### Installation and run

You can install the gem using

`gem install travis-backup`
`gem install travis-backup-for-v3`

Next you can run it like:

```
travis_backup 'postgres://user:pass@localhost:5432/my_db' --threshold 6
travis_backup_for_v3 'postgres://user:pass@localhost:5432/my_db' --threshold 6
```

All arguments:
Expand All @@ -26,12 +26,6 @@ All arguments:
-u, --user_id ID # run only for given user
-o, --org_id ID # run only for given organization
-r, --repo_id ID # run only for given repository
--move_logs # run in move logs mode - move all logs to database at destination_db_url URL
--destination_db_url URL # URL for moving logs to
--remove_orphans # run in remove orphans mode
--orphans_table # name of the table we will remove orphans from (if not defined, all tables are considered)
--load_from_files # loads files stored in files_location to the database
--id_gap # concerns file loading - the gap between the biggest id in database and the lowest one that will be set to loaded data (that's for data inserted by other users during the load being performed; equals 1000 by default)
```

Or inside your app:
Expand Down Expand Up @@ -61,14 +55,6 @@ backup.run(repo_id: 1)

#### Special modes

Using `--move_logs` flag you can move all logs to database at `destination_db_url` URL (which is required in this case). When you run gem in this mode no files are created and no other tables are being touched.

Using `--remove_orphans` flag you can remove all orphaned data from the tables. You can pick a specific table using `--orphans_table` flag or, by leaving it undefined, let all tables to be processed in the removing orphans procedure. It can be combined with `--backup` flag in order to save removed data in files.

Using `--user_id`, `--org_id` or `--repo_id` flag without setting `--threshold` results in removing the specified user/organization/repository with all its dependencies. It can be combined with `--backup` flag in order to save removed data in files.

Using `--load_from_files` flag you can restore dumped data from files located at path given by `--files_location`. The distance defined by `--id_gap` is going to be kept between biggest ids in the database and the lowest ones from the data loaded from files (and it equals 1000 by default).

Using `--dry_run` flag you can check which data would be removed by gem, but without removing them actually. Instead of that reports will be printed on standard output. This flag can be also combined with special modes.

### Configuration options
Expand All @@ -85,18 +71,11 @@ backup:
user_id: 1 # run only for given user
org_id: 1 # run only for given organization
repo_id: 1 # run only for given repository
move_logs: false # run in move logs mode - move all logs to database at destination_db_url URL
remove_orphans: false # run in remove orphans mode
orphans_table: 'builds' # name of the table we will remove orphans from (if not defined, all tables are considered)
load_from_files: false # loads files stored in files_location to the database
id_gap: 1500 # concerns file loading - the gap between the biggest id in database and the lowest one that will be set to loaded data (that's for data inserted by other users during the load being performed; equals 1000 by default)
```

You can also set these properties using env vars corresponding to them: `IF_BACKUP`, `BACKUP_DRY_RUN`, `BACKUP_LIMIT`, `BACKUP_THRESHOLD`, `BACKUP_FILES_LOCATION`, `BACKUP_USER_ID`, `BACKUP_ORG_ID`, `BACKUP_REPO_ID`, `BACKUP_MOVE_LOGS`, `BACKUP_REMOVE_ORPHANS`, `BACKUP_ORPHANS_TABLE`, `BACKUP_LOAD_FROM_FILES`, `BACKUP_ID_GAP`.

You should also specify your database url. You can do this the standard way in `config/database.yml` file, setting the `database_url` hash argument while creating `Backup` instance or using the `DATABASE_URL` env var. Your database should be consistent with the Travis 2.2 database schema.
You can also set these properties using env vars corresponding to them: `IF_BACKUP`, `BACKUP_DRY_RUN`, `BACKUP_LIMIT`, `BACKUP_THRESHOLD`, `BACKUP_FILES_LOCATION`, `BACKUP_USER_ID`, `BACKUP_ORG_ID`, `BACKUP_REPO_ID`.

For `move_logs` mode you need also to specify a destination database. You can set it also in `config/database.yml` file, in `destination` subsection, setting the `destination_db_url` hash argument while creating `Backup` instance or using the `BACKUP_DESTINATION_DB_URL` env var. Your destination database should be consistent with the Travis 3.0 database schema.
You should also specify your database url. You can do this the standard way in `config/database.yml` file, setting the `database_url` hash argument while creating `Backup` instance or using the `DATABASE_URL` env var. Your database should be consistent with the Travis 3.0 database schema.

### How to run the test suite

Expand All @@ -112,9 +91,9 @@ and
bundle exec rspec
```

To make tests working properly you should also ensure database connection strings for empty test databases. You can set them as `DATABASE_URL` and `BACKUP_DESTINATION_DB_URL` environment variables or in `config/database.yml`.
To make tests working properly you should also ensure database connection string for empty test database. You can set it as `DATABASE_URL` environment variable or in `config/database.yml`.

**Warning: these databases will be cleaned during tests, so ensure that they include no important data.**
**Warning: this database will be cleaned during tests, so ensure that it includes no important data.**

### Ruby version

Expand Down
File renamed without changes.
Loading