-
Notifications
You must be signed in to change notification settings - Fork 387
Add steps for local data ingestion in docs #7790
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 2 commits
Commits
Show all changes
31 commits
Select commit
Hold shift + click to select a range
84aa68e
Add steps for local data ingestion in docs
sumairq 19c6e33
Address comments
sumairq 2157aa9
Expand inter-MITTEN-t logic to account for confirm-failures. (#7773)
jmaher 44fbbbd
Upgrade eslint
vrigal 07ecfb5
Upgrade @mozilla/glean
vrigal e9987be
Upgrade webpack-cli
vrigal af8e753
Upgrade webpack
vrigal 6f08963
Restore the switch to OpenAPI (require missing dependency)
EvaBardou fd989ae
Serve ReDoc inside react app
da081e2
Serve Redoc in a dedicated chunk
EvaBardou a1ef2c7
Update rabbitmq Docker tag to v3.11.22
renovate[bot] a561a1f
Update redis Docker tag to v7.0.12
renovate[bot] fea64ba
Update dependency mobx to v6.10.2
renovate[bot] f7af91a
Update redis Docker tag to v7.0.13
renovate[bot] 97aadd9
Add tests for the Debug Tools column in alerts view (#7731)
alexandru-io cbe7b1d
Update dependency mobx to v6.10.2 (#7800)
renovate[bot] 99c669a
Pin dependency styled-components to 5.3.11 (#7799)
renovate[bot] 0321ab5
Add application field
beatrice-acasandrei ce06690
Shorten field name
beatrice-acasandrei e1ecec1
Upgrade CircleCI remote docker from 19.03.14 to 20.10.23
Archaeopteryx 9032d78
Update dependency @mozilla/glean to v2.0.2
renovate[bot] 67a84ff
Update dependency url to v0.11.3
renovate[bot] 9c0f975
Update rabbitmq Docker tag to v3.11.23
renovate[bot] 91cc0e6
Potential fix for Bug 1827920 (#7809)
CanadaHonk 26480cc
Renovate node20 (#7813)
Archaeopteryx 9574896
Add TaskclusterMetadata to the alert summary API endpoint (#7740)
alexandru-io 6eda5e3
Optimize summarygroup endpoint (#7812)
vrigal adbedbc
Fix mitten to work with -cf in the job names (#7810)
jmaher 7ddd91f
Update dependency docs/mkdocs to v1.5.3
renovate[bot] 095555e
Fix select all test checkboxes not scrolling properly
CanadaHonk 0101185
Add steps to installation section
sumairq File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,7 @@ | ||
| # Pulse Ingestion Configuration | ||
|
|
||
| By default, running the Docker container with `docker-compose up` will ingest data | ||
| from the `autoland` and `try` repositories using a shared [Pulse Guardian] user. You can configure this the following ways: | ||
| from the `autoland` and `try` repositories using a shared [Pulse Guardian] user. You can configure this the following ways: | ||
|
|
||
| 1. Specify a custom set of repositories for which to ingest data | ||
| 2. Create a custom **Pulse User** on [Pulse Guardian] | ||
|
|
@@ -19,7 +19,7 @@ export PROJECTS_TO_INGEST=autoland,try | |
| Visit [Pulse Guardian], sign in, and create a **Pulse User**. It will ask you to set a | ||
| username and password. Remember these as you'll use them in the next step. | ||
| This is recommended, because using the default value **MAY** cause you to miss some data, | ||
| if it was already ingested by another user Unfortunately, **Pulse** doesn't support creating | ||
| if it was already ingested by another user. Unfortunately **Pulse** doesn't support creating | ||
| queues with a guest account. | ||
|
|
||
| If your **Pulse User** was username: `foo` and password: `bar`, your Pulse URL | ||
|
|
@@ -62,3 +62,65 @@ ex: <https://community-tc.services.mozilla.com/pulse-messages/> | |
|
|
||
| [pulse guardian]: https://pulseguardian.mozilla.org/whats_pulse | ||
| [yml schema]: https://github.com/mozilla/treeherder/blob/master/schemas/pulse-job.yml | ||
|
|
||
| ## Setup for Perfherder Development | ||
|
|
||
| These steps should help you set up everything you need to ingest data locally while working on perfherder. | ||
|
|
||
| **Create a pulse guardian account and run:** | ||
|
|
||
| ```bash | ||
| export PULSE_URL=amqp://USER:PASSWORD@pulse.mozilla.org:5671/?ssl=1 | ||
| yarn install | ||
| ``` | ||
|
|
||
| **Run each of these commands in a seperate window:** | ||
|
|
||
| ```bash | ||
| docker-compose up --build | ||
| docker-compose run -e PROJECTS_TO_INGEST=autoland backend celery -A treeherder worker --concurrency 1 | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| **Run the db viewer** | ||
|
|
||
| You can use any database viewer of your choice `(e.g. dbeaver-ce, mysql workbench, etc.)` | ||
|
|
||
| **Connect to the following while `docker-compose up --build` from the previous step is running:** | ||
|
|
||
| `Serverhost: localhost` | ||
| `Port: 3306` | ||
| `Database: treeherder` | ||
| `Username: root` | ||
| `No password` | ||
|
|
||
| --- | ||
|
|
||
| ##### Run the following in separate window while running above to do ingestion | ||
|
sumairq marked this conversation as resolved.
Outdated
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hi @sumairq, sorry for the delayed review! In the image below, I noticed that this line is showing up as a new heading in the navigation list/bar on the right-hand side. Could you reformat this line to be in bold instead so it doesn't show up there? |
||
|
|
||
| <!-- prettier-ignore --> | ||
| !!! note | ||
| These commands perform fetches for the data, they are run sequentially in the same window. The first command makes the second one run faster. | ||
|
|
||
| **Ingest push:** | ||
|
|
||
| ```bash | ||
| docker-compose exec backend ./manage.py ingest push -p autoland -r 1ee42a54a431acdd6cbe43b49de0237fe67eddd9 | ||
| ``` | ||
|
|
||
| **Ingest all the tasks, run celery to trigger the log parsing, and performance data ingestion:** | ||
|
|
||
| > **Warning:** This command can take a long time to ingest and parse everything. | ||
|
|
||
| ```bash | ||
|
sumairq marked this conversation as resolved.
Outdated
|
||
| docker-compose exec backend ./manage.py ingest push -p autoland -r 1ee42a54a431acdd6cbe43b49de0237fe67eddd9 -a --enable-eager-celery | ||
| ``` | ||
|
|
||
| `--enable-eager-celery` triggers the log parsing which is required to capture the `PERFHERDER_DATA` output. | ||
|
|
||
| **For ingesting multiple pushes:** | ||
|
|
||
| ```bash | ||
| docker-compose exec backend ./manage.py ingest push -p autoland --last-n-pushes 100 | ||
| ``` | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.

Uh oh!
There was an error while loading. Please reload this page.