Skip to content

update to new monitoring index for kibana and logstash #8510

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

update to new monitoring index for kibana and logstash #8510

wants to merge 2 commits into from

Conversation

@kvalliyurnatt
Copy link
Contributor

@kvalliyurnatt kvalliyurnatt commented Mar 6, 2025

We have the e2e tests failing because the monitoring index was updated for 9.x release, updating this in our tests.

@kvalliyurnatt kvalliyurnatt added >test Related to unit/integration/e2e tests exclude-from-release-notes Exclude this PR from appearing in the release notes v3.0.0 labels Mar 6, 2025
@barkbay
Copy link
Contributor

barkbay commented Mar 6, 2025

buildkite test this -f p=gke,s=9.0.0-SNAPSHOT -m t=TestKBStackMonitoring,E2E_TAGS=kb,E2E_TAGS=e2e

@barkbay
Copy link
Contributor

barkbay commented Mar 6, 2025

I think TestKBStackMonitoring is still failing:

{"Time":"2025-03-06T08:08:39.939134261Z","Action":"output","Package":"github.com/elastic/cloud-on-k8s/v2/test/e2e/kb","Test":"TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb","Output":"=== RUN   TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb\n"}
--
  | {"Time":"2025-03-06T08:23:39.939453292Z","Action":"output","Package":"github.com/elastic/cloud-on-k8s/v2/test/e2e/kb","Test":"TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb","Output":"Retries (15m0s timeout): .......................................................................................................................................................................................................................................................................................................\n"}
  | {"Time":"2025-03-06T08:23:39.939586187Z","Action":"output","Package":"github.com/elastic/cloud-on-k8s/v2/test/e2e/kb","Test":"TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb","Output":"    step.go:51: \n"}
  | {"Time":"2025-03-06T08:23:39.939595129Z","Action":"output","Package":"github.com/elastic/cloud-on-k8s/v2/test/e2e/kb","Test":"TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb","Output":"        \tError Trace:\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/utils.go:94\n"}
  | {"Time":"2025-03-06T08:23:39.939617109Z","Action":"output","Package":"github.com/elastic/cloud-on-k8s/v2/test/e2e/kb","Test":"TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb","Output":"        \tError:      \tReceived unexpected error:\n"}
  | {
    "Time": "2025-03-06T08:23:39.939624825Z",
    "Action": "output",
    "Package": "github.com/elastic/cloud-on-k8s/v2/test/e2e/kb",
    "Test": "TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb",
    "Output": "        \t            \telasticsearch client failed for https://test-kb-mon-metrics-hppd-es-masterdata-1.test-kb-mon-metrics-hppd-es-masterdata.e2e-ps8i3-mercury:9200/_cat/indices/.monitoring-kibana-9-mb?format=json: 404 Not Found: {Status:404 Error:{CausedBy:{Reason: Type:} Reason:no such index [.monitoring-kibana-9-mb] Type:index_not_found_exception StackTrace: RootCause:[{Reason:no such index [.monitoring-kibana-9-mb] Type:index_not_found_exception}]}}\n"
}
  | {"Time":"2025-03-06T08:23:39.939636953Z","Action":"output","Package":"github.com/elastic/cloud-on-k8s/v2/test/e2e/kb","Test":"TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb","Output":"        \tTest:       \tTestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb\n"}
  | {
    "Time": "2025-03-06T08:23:39.943456874Z",
    "Action": "output",
    "Package": "github.com/elastic/cloud-on-k8s/v2/test/e2e/kb",
    "Test": "TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb",
    "Output": "{\"log.level\":\"error\",\"@timestamp\":\"2025-03-06T08:23:39.939Z\",\"message\":\"continuing with additional tests\",\"service.version\":\"0.0.0-SNAPSHOT+00000000\",\"service.type\":\"eck\",\"ecs.version\":\"1.4.0\",\"error\":\"test Check that documents are indexed in index .monitoring-kibana-9-mb failed\",\"error.stack_trace\":\"github.com/elastic/cloud-on-k8s/v2/test/e2e/test.StepList.RunSequential\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/step.go:52\ngithub.com/elastic/cloud-on-k8s/v2/test/e2e/kb.TestKBStackMonitoring\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/kb/stack_monitoring_test.go:45\ntesting.tRunner\n\t/usr/local/go/src/testing/testing.go:1792\"}\n"
}
  | {
    "Time": "2025-03-06T08:23:39.943707299Z",
    "Action": "output",
    "Package": "github.com/elastic/cloud-on-k8s/v2/test/e2e/kb",
    "Test": "TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb",
    "Output": "{\"log.level\":\"info\",\"@timestamp\":\"2025-03-06T08:23:39.943Z\",\"log.logger\":\"e2e\",\"message\":\"Running eck-diagnostics\",\"service.version\":\"0.0.0-SNAPSHOT+00000000\",\"service.type\":\"eck\",\"ecs.version\":\"1.4.0\",\"cluster\":\"eck-e2e-gke-testkbstackm-aqfi-10423\",\"test\":\"TestKBStackMonitoring\",\"step\":\"Check that documents are indexed in index .monitoring-kibana-9-mb\"}\n"
}

@kvalliyurnatt
Copy link
Contributor Author

kvalliyurnatt commented Mar 6, 2025

I think TestKBStackMonitoring is still failing:

{"Time":"2025-03-06T08:08:39.939134261Z","Action":"output","Package":"github.com/elastic/cloud-on-k8s/v2/test/e2e/kb","Test":"TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb","Output":"=== RUN   TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb\n"}
--
  | {"Time":"2025-03-06T08:23:39.939453292Z","Action":"output","Package":"github.com/elastic/cloud-on-k8s/v2/test/e2e/kb","Test":"TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb","Output":"Retries (15m0s timeout): .......................................................................................................................................................................................................................................................................................................\n"}
  | {"Time":"2025-03-06T08:23:39.939586187Z","Action":"output","Package":"github.com/elastic/cloud-on-k8s/v2/test/e2e/kb","Test":"TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb","Output":"    step.go:51: \n"}
  | {"Time":"2025-03-06T08:23:39.939595129Z","Action":"output","Package":"github.com/elastic/cloud-on-k8s/v2/test/e2e/kb","Test":"TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb","Output":"        \tError Trace:\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/utils.go:94\n"}
  | {"Time":"2025-03-06T08:23:39.939617109Z","Action":"output","Package":"github.com/elastic/cloud-on-k8s/v2/test/e2e/kb","Test":"TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb","Output":"        \tError:      \tReceived unexpected error:\n"}
  | {
    "Time": "2025-03-06T08:23:39.939624825Z",
    "Action": "output",
    "Package": "github.com/elastic/cloud-on-k8s/v2/test/e2e/kb",
    "Test": "TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb",
    "Output": "        \t            \telasticsearch client failed for https://test-kb-mon-metrics-hppd-es-masterdata-1.test-kb-mon-metrics-hppd-es-masterdata.e2e-ps8i3-mercury:9200/_cat/indices/.monitoring-kibana-9-mb?format=json: 404 Not Found: {Status:404 Error:{CausedBy:{Reason: Type:} Reason:no such index [.monitoring-kibana-9-mb] Type:index_not_found_exception StackTrace: RootCause:[{Reason:no such index [.monitoring-kibana-9-mb] Type:index_not_found_exception}]}}\n"
}
  | {"Time":"2025-03-06T08:23:39.939636953Z","Action":"output","Package":"github.com/elastic/cloud-on-k8s/v2/test/e2e/kb","Test":"TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb","Output":"        \tTest:       \tTestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb\n"}
  | {
    "Time": "2025-03-06T08:23:39.943456874Z",
    "Action": "output",
    "Package": "github.com/elastic/cloud-on-k8s/v2/test/e2e/kb",
    "Test": "TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb",
    "Output": "{\"log.level\":\"error\",\"@timestamp\":\"2025-03-06T08:23:39.939Z\",\"message\":\"continuing with additional tests\",\"service.version\":\"0.0.0-SNAPSHOT+00000000\",\"service.type\":\"eck\",\"ecs.version\":\"1.4.0\",\"error\":\"test Check that documents are indexed in index .monitoring-kibana-9-mb failed\",\"error.stack_trace\":\"github.com/elastic/cloud-on-k8s/v2/test/e2e/test.StepList.RunSequential\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/step.go:52\ngithub.com/elastic/cloud-on-k8s/v2/test/e2e/kb.TestKBStackMonitoring\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/kb/stack_monitoring_test.go:45\ntesting.tRunner\n\t/usr/local/go/src/testing/testing.go:1792\"}\n"
}
  | {
    "Time": "2025-03-06T08:23:39.943707299Z",
    "Action": "output",
    "Package": "github.com/elastic/cloud-on-k8s/v2/test/e2e/kb",
    "Test": "TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb",
    "Output": "{\"log.level\":\"info\",\"@timestamp\":\"2025-03-06T08:23:39.943Z\",\"log.logger\":\"e2e\",\"message\":\"Running eck-diagnostics\",\"service.version\":\"0.0.0-SNAPSHOT+00000000\",\"service.type\":\"eck\",\"ecs.version\":\"1.4.0\",\"cluster\":\"eck-e2e-gke-testkbstackm-aqfi-10423\",\"test\":\"TestKBStackMonitoring\",\"step\":\"Check that documents are indexed in index .monitoring-kibana-9-mb\"}\n"
}

aaah thanks, it passed for me locally I thought, I will double check what is going on

@kvalliyurnatt
Copy link
Contributor Author

kvalliyurnatt commented Mar 6, 2025

I rechecked this test locally, and it is passing for me. I am not really sure why this is failing

=== RUN   TestKBStackMonitoring/Check_that_documents_are_indexed_in_index_.monitoring-kibana-9-mb
Retries (15m0s timeout): .
=== RUN   TestKBStackMonitoring/Check_that_documents_are_indexed_in_one_filebeat-*_index
Retries (15m0s timeout): .

@kvalliyurnatt
Copy link
Contributor Author

kvalliyurnatt commented Mar 6, 2025

buildkite test this -f p=gke,s=9.0.0-SNAPSHOT -m t=TestKBStackMonitoring,E2E_TAGS=kb,E2E_TAGS=e2e

@kvalliyurnatt
Copy link
Contributor Author

kvalliyurnatt commented Mar 6, 2025

Seems like the version bump was reverted
elastic/elasticsearch#124112
so we do not need this change, I will check today's test run and close this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pebrc pebrc Awaiting requested review from pebrc

@barkbay barkbay Awaiting requested review from barkbay

@naemono naemono Awaiting requested review from naemono

@rhr323 rhr323 Awaiting requested review from rhr323

Assignees

No one assigned

Labels

exclude-from-release-notes Exclude this PR from appearing in the release notes >test Related to unit/integration/e2e tests v3.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kvalliyurnatt @barkbay