Skip to content

[WIP] Add an elastic search forward (inc. example usage) #68

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 24 commits into
base: main
Choose a base branch
from
Open

[WIP] Add an elastic search forward (inc. example usage) #68

wants to merge 24 commits into from

Conversation

s0undt3ch
Copy link
Contributor

No description provided.

@s0undt3ch s0undt3ch force-pushed the features/elastic branch 2 times, most recently from 11735e5 to fce5caa Compare July 5, 2023 13:07
@codecov-commenter
Copy link

codecov-commenter commented Jul 5, 2023 •
edited
Loading

Codecov Report

Patch coverage: 47.63% and project coverage change: -5.25 ⚠️

Comparison is base (adbe094) 84.43% compared to head (e4c773a) 79.18%.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main      #68      +/-   ##
==========================================
- Coverage   84.43%   79.18%   -5.25%     
==========================================
  Files          51       54       +3     
  Lines        1869     2056     +187     
  Branches      220      218       -2     
==========================================
+ Hits         1578     1628      +50     
- Misses        228      368     +140     
+ Partials       63       60       -3
Flag Coverage Δ
Linux 79.18% <47.63%> (-3.84%) ⬇️
Windows ?
core 78.68% <47.63%> (-5.47%) ⬇️
examples 56.37% <44.20%> (-10.98%) ⬇️
macOS ?
src 71.54% <47.63%> (-7.41%) ⬇️
tests 91.05% <ø> (-0.38%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/saf/utils/eventbus.py 22.07% <12.50%> (-0.90%) ⬇️
src/saf/collect/beacons.py 66.03% <14.28%> (-7.88%) ⬇️
src/saf/process/job_aggregate.py 33.33% <33.33%> (ø)
src/saf/models.py 83.49% <40.00%> (-3.51%) ⬇️
src/saf/forward/elasticsearch.py 51.47% <51.47%> (ø)
src/saf/collect/grains.py 70.58% <70.58%> (ø)
src/saf/collect/event_bus.py 78.26% <78.26%> (ø)

... and 7 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@s0undt3ch s0undt3ch force-pushed the features/elastic branch 3 times, most recently from 5c0ca15 to e759fe4 Compare July 6, 2023 05:52
s0undt3ch and others added 22 commits July 18, 2023 17:12
@s0undt3ch
When loading plugin configs, catch KeyError on unavailable plugins …
c68b404 
…and list all available.

Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Fix regression which prevented pipelines without processors
afffb16 
Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Pass the event tag and stamp explicitly when calling `saf.utils.event…
24076ca 
…bus._construct_event()`

Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Add the id attribute to BeaconCollectedEvent, which is minion ID.
88fa320 
Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Add an elastic search forwarder (inc. example usage)
50b03e1 
Fixes saltstack#69

Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Initial check in for job aggregations
230a994
@s0undt3ch
Add grains collector and state aggregate processor
925a440
@s0undt3ch
Renamed src/saf/process/state_aggregate.py -> `src/saf/process/job_…
248fa1c 
…aggregate.py`

Additionally, we listen to all job events, not just state events.

Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Make some additional improvements to the JobAggregate Processor
1a5f5ab
@s0undt3ch
Split master and minion pipelines
faccc76 
Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Add a scheduled test.ping job to the minions
6e113e7 
Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Add default debug logging for the salt containers
e204602 
Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Add debug logging to the job_aggregate processor
c9ed1a5 
Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Add a jobs_to_es processor
2a7deb5 
Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Add custom script to loop to jobs we want to trigger from master
9f792bc 
Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Some elastic example salt config file changes
88def5b 
Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Allow providing the elastic search index in the configuration
31a5f21 
Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
The jobs should be optionally set.
d522bea 
Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Fix paths to supervisord.conf
20b30af 
Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
fun_args also needs to be JSON dumped so that Elastic Search doesn'…
368d93a 
…t complain

Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Fix elastic search docker example after the include grains update
356c24d 
Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Add a state run index to the elastic search docker example
db0356b 
Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Setup the cache sooner
93a8bae 
Signed-off-by: Pedro Algarvio <[email protected]>
@s0undt3ch
Catch JSONDecodeError errors
e4c773a 
Signed-off-by: Pedro Algarvio <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
cla-not-required
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@s0undt3ch @codecov-commenter @vmwclabot