Skip to content

[STORY-2042] log-drains: Add OpenSearch support #1110

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
briceamen wants to merge 3 commits into from
Closed

[STORY-2042] log-drains: Add OpenSearch support #1110

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ad94b9a
refactor(logdrains): update log drain command usage and improve error…
briceamen May 21, 2025
4242ea8
Merge branch 'master' into story-2042/opensearch-logdrain
briceamen May 26, 2025
e028585
doc(logdrains): update opensearch log drain command URL example
briceamen May 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions cmd/log_drains.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,6 @@ Use the parameter "--with-addons" to list log drains of all addons connected to
autocomplete.CmdFlagsAutoComplete(c, "log-drains")
},
}

logDrainsAddCommand = cli.Command{
Name: "log-drains-add",
Category: "Log drains",
Expand All @@ -68,7 +67,7 @@ Use the parameter "--with-addons" to list log drains of all addons connected to
&cli.BoolFlag{Name: "with-addons", Usage: "also add the log drains to all addons"},
&cli.BoolFlag{Name: "with-databases", Usage: "also add the log drains to all databases"},
&cli.StringFlag{Name: "type", Usage: "Communication protocol", Required: true},
&cli.StringFlag{Name: "url", Usage: "URL of self hosted ELK"},
&cli.StringFlag{Name: "url", Usage: "URL of self hosted ELK or OpenSearch"},
&cli.StringFlag{Name: "host", Usage: "Host of logs management service"},
&cli.StringFlag{Name: "port", Usage: "Port of logs management service"},
&cli.StringFlag{Name: "token", Usage: "Used by certain vendor for authentication"},
Expand Down Expand Up @@ -96,6 +95,7 @@ Warning: At the moment, only databases addons are able to forward logs to a drai
"scalingo --app my-app log-drains-add --type syslog --host custom.logstash.com --port 12345",
"scalingo --app my-app log-drains-add --type syslog --token 123456789abcdef --host custom.logstash.com --port 12345",
"scalingo --app my-app log-drains-add --type elk --url https://my-user:123456789abcdef@logstash-app-name.osc-fr1.scalingo.io",
"scalingo --app my-app log-drains-add --type opensearch --url https://my-user:123456789abcdef@my-app-name.opensearch.osc-fr1.scalingo-dbs.com:30123/myindex/_doc?pipeline=my_pipeline",
Copy link

@semgrep-code-scalingo semgrep-code-scalingo bot May 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hardcoded sensitive data leaked

Semgrep has detected a leak of sensitive data in this code. This secret data could be used by internal or external malicious actors. We highly recommend you change, reset, or rotate the sensitive data.

A secret is hard-coded in the application. Secrets stored in source code, such as credentials, identifiers, and other types of sensitive data, can be leaked and used by internal or external malicious actors. It is recommended to rotate the secret and retrieve them from a secure secret vault or Hardware Security Module (HSM), alternatively environment variables can be used if allowed by your company policy.

💬 To ignore this, reply with:
• /fp <comment> for false positive
• /ar <comment> for acceptable risk
• /other <comment> for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by generic_uri_string.

Copy link
Contributor Author

@briceamen briceamen May 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/fp url example

"scalingo --app my-app --addon ad-3c2f8c81-99bd-4667-9791-466799bd4667 log-drains-add --type datadog --token 123456789abcdef --drain-region eu-west-2",
"scalingo --app my-app --with-addons log-drains-add --type datadog --token 123456789abcdef --drain-region eu-west-2",
},
Expand Down
23 changes: 14 additions & 9 deletions logdrains/remove.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@ package logdrains

import (
"context"

"gopkg.in/errgo.v1"
"strings"

"github.com/Scalingo/cli/config"
"github.com/Scalingo/cli/io"
"github.com/Scalingo/go-utils/errors/v2"
)

type RemoveAddonOpts struct {
Expand All @@ -18,36 +18,41 @@ type RemoveAddonOpts struct {
func Remove(ctx context.Context, app string, opts RemoveAddonOpts) error {
c, err := config.ScalingoClient(ctx)
if err != nil {
return errgo.Notef(err, "fail to get Scalingo client to remove a log drain from the application")
return errors.Wrap(ctx, err, "get Scalingo client")
}

if opts.AddonID != "" {
// addon only
err := c.LogDrainAddonRemove(ctx, app, opts.AddonID, opts.URL)
if err != nil {
return errgo.Notef(err, "fail to remove the log drain from the addon %s", opts.AddonID)
return errors.Wrap(ctx, err, "remove log drain from addon "+opts.AddonID)
}
io.Status("The log drain", opts.URL, "has been deleted from the addon", opts.AddonID)
return nil
}

err = c.LogDrainRemove(ctx, app, opts.URL)
if err != nil {
io.Status("fail to remove the log drain from the application:", app, "\n\t", err)
} else {
io.Status("Log drain", opts.URL, "has been deleted from the application", app)
return errors.Wrap(ctx, err, "remove log drain from application "+app)
}
io.Status("Log drain", opts.URL, "has been deleted from the application", app)

if !opts.OnlyApp {
addons, err := c.AddonsList(ctx, app)
if err != nil {
return errgo.Notef(err, "fail to list addons to remove log drain")
return errors.Wrap(ctx, err, "list addons to remove log drain")
}

for _, addon := range addons {
err := c.LogDrainAddonRemove(ctx, app, addon.ID, opts.URL)
if err != nil {
io.Status("fail to remove the log drain from the addon:", addon.AddonProvider.Name, "\n\t", err)
// Check if this is a "not found" error, which can happen if the log drain
// was already removed by the main API call
if strings.Contains(err.Error(), "not found") {
io.Status("Log drain", opts.URL, "was already removed from the addon", addon.AddonProvider.Name)
Copy link
Member

@curzolapierre curzolapierre May 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Questions: (Sorry for the late notice) is this message will be logged when an addon never got the log drains?
If yes, the message is not totally accurate, WDYT?

} else {
io.Status("Unable to remove the log drain from the addon:", addon.AddonProvider.Name, "\n\t", err)
}
} else {
io.Status("Log drain", opts.URL, "has been deleted from the addon", addon.AddonProvider.Name)
}
Expand Down
Loading