Skip to content

Not save partial #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
uguese wants to merge 4 commits into
base: initial
Choose a base branch
from
Open

Not save partial #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
03e2c20
Merge pull request #1 from travis-ci/initial
weban-pl Mar 2, 2021
156cc43
Do not save partial file (not filled with builds == limit)
piotrmilcarz Mar 29, 2021
d488516
Rails bump
piotrmilcarz Mar 29, 2021
0c5b53e
Rails bump
piotrmilcarz Mar 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ gem 'activerecord'
gem 'google-cloud-storage', '~> 1.8', require: false
gem 'pg'
gem 'pry'
gem 'rails', '~> 6.1.2.1'
gem 'rails', '~> 6.1.3.1'
gem 'redis'

gem 'bootsnap', require: false
Expand Down
164 changes: 81 additions & 83 deletions Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,60 +1,60 @@
GEM
remote: https://rubygems.org/
specs:
actioncable (6.1.2.1)
actionpack (= 6.1.2.1)
activesupport (= 6.1.2.1)
actioncable (6.1.3.1)
actionpack (= 6.1.3.1)
activesupport (= 6.1.3.1)
nio4r (~> 2.0)
websocket-driver (>= 0.6.1)
actionmailbox (6.1.2.1)
actionpack (= 6.1.2.1)
activejob (= 6.1.2.1)
activerecord (= 6.1.2.1)
activestorage (= 6.1.2.1)
activesupport (= 6.1.2.1)
actionmailbox (6.1.3.1)
actionpack (= 6.1.3.1)
activejob (= 6.1.3.1)
activerecord (= 6.1.3.1)
activestorage (= 6.1.3.1)
activesupport (= 6.1.3.1)
mail (>= 2.7.1)
actionmailer (6.1.2.1)
actionpack (= 6.1.2.1)
actionview (= 6.1.2.1)
activejob (= 6.1.2.1)
activesupport (= 6.1.2.1)
actionmailer (6.1.3.1)
actionpack (= 6.1.3.1)
actionview (= 6.1.3.1)
activejob (= 6.1.3.1)
activesupport (= 6.1.3.1)
mail (~> 2.5, >= 2.5.4)
rails-dom-testing (~> 2.0)
actionpack (6.1.2.1)
actionview (= 6.1.2.1)
activesupport (= 6.1.2.1)
actionpack (6.1.3.1)
actionview (= 6.1.3.1)
activesupport (= 6.1.3.1)
rack (~> 2.0, >= 2.0.9)
rack-test (>= 0.6.3)
rails-dom-testing (~> 2.0)
rails-html-sanitizer (~> 1.0, >= 1.2.0)
actiontext (6.1.2.1)
actionpack (= 6.1.2.1)
activerecord (= 6.1.2.1)
activestorage (= 6.1.2.1)
activesupport (= 6.1.2.1)
actiontext (6.1.3.1)
actionpack (= 6.1.3.1)
activerecord (= 6.1.3.1)
activestorage (= 6.1.3.1)
activesupport (= 6.1.3.1)
nokogiri (>= 1.8.5)
actionview (6.1.2.1)
activesupport (= 6.1.2.1)
actionview (6.1.3.1)
activesupport (= 6.1.3.1)
builder (~> 3.1)
erubi (~> 1.4)
rails-dom-testing (~> 2.0)
rails-html-sanitizer (~> 1.1, >= 1.2.0)
activejob (6.1.2.1)
activesupport (= 6.1.2.1)
activejob (6.1.3.1)
activesupport (= 6.1.3.1)
globalid (>= 0.3.6)
activemodel (6.1.2.1)
activesupport (= 6.1.2.1)
activerecord (6.1.2.1)
activemodel (= 6.1.2.1)
activesupport (= 6.1.2.1)
activestorage (6.1.2.1)
actionpack (= 6.1.2.1)
activejob (= 6.1.2.1)
activerecord (= 6.1.2.1)
activesupport (= 6.1.2.1)
marcel (~> 0.3.1)
mimemagic (~> 0.3.2)
activesupport (6.1.2.1)
activemodel (6.1.3.1)
activesupport (= 6.1.3.1)
activerecord (6.1.3.1)
activemodel (= 6.1.3.1)
activesupport (= 6.1.3.1)
activestorage (6.1.3.1)
actionpack (= 6.1.3.1)
activejob (= 6.1.3.1)
activerecord (= 6.1.3.1)
activesupport (= 6.1.3.1)
marcel (~> 1.0.0)
mini_mime (~> 1.0.2)
activesupport (6.1.3.1)
concurrent-ruby (~> 1.0, >= 1.0.2)
i18n (>= 1.6, < 2)
minitest (>= 5.1)
Expand All @@ -63,12 +63,12 @@ GEM
addressable (2.7.0)
public_suffix (>= 2.0.2, < 5.0)
ast (2.4.2)
bootsnap (1.7.2)
bootsnap (1.7.3)
msgpack (~> 1.0)
brakeman (5.0.0)
builder (3.2.4)
byebug (11.1.3)
coderay (1.1.2)
coderay (1.1.3)
concurrent-ruby (1.1.8)
crass (1.0.6)
declarative (0.0.20)
Expand All @@ -84,10 +84,10 @@ GEM
multipart-post (>= 1.2, < 3)
ruby2_keywords
faraday-net_http (1.0.1)
ffi (1.14.2)
ffi (1.15.0)
globalid (0.4.2)
activesupport (>= 4.2.0)
google-apis-core (0.2.1)
google-apis-core (0.3.0)
addressable (~> 2.5, >= 2.5.1)
googleauth (~> 0.14)
httpclient (>= 2.8.1, < 3.0)
Expand All @@ -97,97 +97,95 @@ GEM
rexml
signet (~> 0.14)
webrick
google-apis-iamcredentials_v1 (0.1.0)
google-apis-iamcredentials_v1 (0.2.0)
google-apis-core (~> 0.1)
google-apis-storage_v1 (0.1.0)
google-apis-storage_v1 (0.3.0)
google-apis-core (~> 0.1)
google-cloud-core (1.5.0)
google-cloud-core (1.6.0)
google-cloud-env (~> 1.0)
google-cloud-errors (~> 1.0)
google-cloud-env (1.4.0)
google-cloud-env (1.5.0)
faraday (>= 0.17.3, < 2.0)
google-cloud-errors (1.0.1)
google-cloud-storage (1.30.0)
google-cloud-errors (1.1.0)
google-cloud-storage (1.31.0)
addressable (~> 2.5)
digest-crc (~> 0.4)
google-apis-iamcredentials_v1 (~> 0.1)
google-apis-storage_v1 (~> 0.1)
google-cloud-core (~> 1.2)
googleauth (~> 0.9)
mini_mime (~> 1.0)
googleauth (0.15.0)
googleauth (0.16.0)
faraday (>= 0.17.3, < 2.0)
jwt (>= 1.4, < 3.0)
memoist (~> 0.16)
multi_json (~> 1.11)
os (>= 0.9, < 2.0)
signet (~> 0.14)
httpclient (2.8.3)
i18n (1.8.8)
i18n (1.8.9)
concurrent-ruby (~> 1.0)
jaro_winkler (1.5.4)
jwt (2.2.2)
listen (3.4.1)
listen (3.5.0)
rb-fsevent (~> 0.10, >= 0.10.3)
rb-inotify (~> 0.9, >= 0.9.10)
loofah (2.9.0)
crass (~> 1.0.2)
nokogiri (>= 1.5.9)
mail (2.7.1)
Copy link
Author

@uguese uguese Sep 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi how are you doing today?

mini_mime (>= 0.1.1)
marcel (0.3.3)
mimemagic (~> 0.3.2)
marcel (1.0.0)
memoist (0.16.2)
method_source (1.0.0)
mimemagic (0.3.5)
mini_mime (1.0.2)
mini_mime (1.0.3)
mini_portile2 (2.5.0)
minitest (5.14.3)
minitest (5.14.4)
msgpack (1.4.2)
multi_json (1.15.0)
multipart-post (2.1.1)
nio4r (2.5.5)
nokogiri (1.11.1)
nio4r (2.5.7)
nokogiri (1.11.2)
mini_portile2 (~> 2.5.0)
racc (~> 1.4)
nokogiri (1.11.1-x86_64-darwin)
nokogiri (1.11.2-x86_64-darwin)
racc (~> 1.4)
os (1.1.1)
parallel (1.20.1)
parser (3.0.0.0)
ast (~> 2.4.1)
pg (1.2.3)
pry (0.13.0)
pry (0.14.0)
coderay (~> 1.1)
method_source (~> 1.0)
public_suffix (4.0.6)
racc (1.5.2)
rack (2.2.3)
rack-test (1.1.0)
rack (>= 1.0, < 3)
rails (6.1.2.1)
actioncable (= 6.1.2.1)
actionmailbox (= 6.1.2.1)
actionmailer (= 6.1.2.1)
actionpack (= 6.1.2.1)
actiontext (= 6.1.2.1)
actionview (= 6.1.2.1)
activejob (= 6.1.2.1)
activemodel (= 6.1.2.1)
activerecord (= 6.1.2.1)
activestorage (= 6.1.2.1)
activesupport (= 6.1.2.1)
rails (6.1.3.1)
actioncable (= 6.1.3.1)
actionmailbox (= 6.1.3.1)
actionmailer (= 6.1.3.1)
actionpack (= 6.1.3.1)
actiontext (= 6.1.3.1)
actionview (= 6.1.3.1)
activejob (= 6.1.3.1)
activemodel (= 6.1.3.1)
activerecord (= 6.1.3.1)
activestorage (= 6.1.3.1)
activesupport (= 6.1.3.1)
bundler (>= 1.15.0)
railties (= 6.1.2.1)
railties (= 6.1.3.1)
sprockets-rails (>= 2.0.0)
rails-dom-testing (2.0.3)
activesupport (>= 4.2.0)
nokogiri (>= 1.6)
rails-html-sanitizer (1.3.0)
loofah (~> 2.3)
railties (6.1.2.1)
actionpack (= 6.1.2.1)
activesupport (= 6.1.2.1)
railties (6.1.3.1)
actionpack (= 6.1.3.1)
activesupport (= 6.1.3.1)
method_source
rake (>= 0.8.7)
thor (~> 1.0)
Expand All @@ -211,10 +209,10 @@ GEM
rspec-mocks (3.10.2)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.10.0)
rspec-rails (4.0.2)
actionpack (>= 4.2)
activesupport (>= 4.2)
railties (>= 4.2)
rspec-rails (5.0.1)
actionpack (>= 5.2)
activesupport (>= 5.2)
railties (>= 5.2)
rspec-core (~> 3.10)
rspec-expectations (~> 3.10)
rspec-mocks (~> 3.10)
Expand All @@ -231,7 +229,7 @@ GEM
rubocop (>= 0.68.1)
ruby-progressbar (1.11.0)
ruby2_keywords (0.0.4)
signet (0.14.1)
signet (0.15.0)
addressable (~> 2.3)
faraday (>= 0.17.3, < 2.0)
jwt (>= 1.5, < 3.0)
Expand Down Expand Up @@ -268,7 +266,7 @@ DEPENDENCIES
listen
pg
pry
rails (~> 6.1.2.1)
rails (~> 6.1.3.1)
redis
rspec-rails
rubocop (~> 0.75.1)
Expand Down
20 changes: 11 additions & 9 deletions lib/backup.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,15 +59,17 @@ def export(owner_id = nil)

def process_repo(repository) # rubocop:disable Metrics/AbcSize, Metrics/MethodLength
repository.builds.where('created_at < ?', @config.delay.months.ago.to_datetime)
.in_groups_of(@config.limit).map do |builds|
builds_export = export_builds(builds)
file_name = "repository_#{repository.id}_builds_#{builds.compact.first.id}-#{builds.compact.last.id}.json"
pretty_json = JSON.pretty_generate(builds_export)
if upload(file_name, pretty_json)
BuildBackup.new(repository_id: repository.id, file_name: file_name).save!
builds.compact.each(&:destroy)
.in_groups_of(@config.limit, false).map do |builds|
if builds.count == @config.limit
builds_export = export_builds(builds)
file_name = "repository_#{repository.id}_builds_#{builds.first.id}-#{builds.last.id}.json"
pretty_json = JSON.pretty_generate(builds_export)
if upload(file_name, pretty_json)
BuildBackup.new(repository_id: repository.id, file_name: file_name).save!
builds.each(&:destroy)
end
builds_export
end
builds_export
end
end

Expand Down Expand Up @@ -107,7 +109,7 @@ def generate_log_token(job_id)
end

def export_builds(builds)
builds.compact.map do |build|
builds.map do |build|
build_export = build.attributes
build_export[:build_config] = build.build_config&.attributes
build_export[:jobs] = export_jobs(build.jobs)
Expand Down