Skip to content
Closed
Changes from 2 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
84aa68e
Add steps for local data ingestion in docs
sumairq Sep 1, 2023
19c6e33
Address comments
sumairq Sep 1, 2023
2157aa9
Expand inter-MITTEN-t logic to account for confirm-failures. (#7773)
jmaher Sep 1, 2023
44fbbbd
Upgrade eslint
vrigal Sep 4, 2023
07ecfb5
Upgrade @mozilla/glean
vrigal Sep 4, 2023
e9987be
Upgrade webpack-cli
vrigal Sep 4, 2023
af8e753
Upgrade webpack
vrigal Sep 4, 2023
6f08963
Restore the switch to OpenAPI (require missing dependency)
EvaBardou Jul 11, 2023
fd989ae
Serve ReDoc inside react app
Jul 19, 2023
da081e2
Serve Redoc in a dedicated chunk
EvaBardou Sep 5, 2023
a1ef2c7
Update rabbitmq Docker tag to v3.11.22
renovate[bot] Sep 5, 2023
a561a1f
Update redis Docker tag to v7.0.12
renovate[bot] Sep 5, 2023
fea64ba
Update dependency mobx to v6.10.2
renovate[bot] Sep 11, 2023
f7af91a
Update redis Docker tag to v7.0.13
renovate[bot] Sep 11, 2023
97aadd9
Add tests for the Debug Tools column in alerts view (#7731)
alexandru-io Sep 13, 2023
cbe7b1d
Update dependency mobx to v6.10.2 (#7800)
renovate[bot] Sep 13, 2023
99c669a
Pin dependency styled-components to 5.3.11 (#7799)
renovate[bot] Sep 13, 2023
0321ab5
Add application field
beatrice-acasandrei Aug 28, 2023
ce06690
Shorten field name
beatrice-acasandrei Aug 29, 2023
e1ecec1
Upgrade CircleCI remote docker from 19.03.14 to 20.10.23
Archaeopteryx Sep 1, 2023
9032d78
Update dependency @mozilla/glean to v2.0.2
renovate[bot] Sep 18, 2023
67a84ff
Update dependency url to v0.11.3
renovate[bot] Sep 18, 2023
9c0f975
Update rabbitmq Docker tag to v3.11.23
renovate[bot] Sep 18, 2023
91cc0e6
Potential fix for Bug 1827920 (#7809)
CanadaHonk Sep 19, 2023
26480cc
Renovate node20 (#7813)
Archaeopteryx Sep 19, 2023
9574896
Add TaskclusterMetadata to the alert summary API endpoint (#7740)
alexandru-io Sep 20, 2023
6eda5e3
Optimize summarygroup endpoint (#7812)
vrigal Sep 20, 2023
adbedbc
Fix mitten to work with -cf in the job names (#7810)
jmaher Sep 21, 2023
7ddd91f
Update dependency docs/mkdocs to v1.5.3
renovate[bot] Sep 25, 2023
095555e
Fix select all test checkboxes not scrolling properly
CanadaHonk Sep 18, 2023
0101185
Add steps to installation section
sumairq Sep 29, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 64 additions & 2 deletions docs/pulseload.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Pulse Ingestion Configuration

By default, running the Docker container with `docker-compose up` will ingest data
from the `autoland` and `try` repositories using a shared [Pulse Guardian] user. You can configure this the following ways:
from the `autoland` and `try` repositories using a shared [Pulse Guardian] user. You can configure this the following ways:

1. Specify a custom set of repositories for which to ingest data
2. Create a custom **Pulse User** on [Pulse Guardian]
Expand All @@ -19,7 +19,7 @@ export PROJECTS_TO_INGEST=autoland,try
Visit [Pulse Guardian], sign in, and create a **Pulse User**. It will ask you to set a
username and password. Remember these as you'll use them in the next step.
This is recommended, because using the default value **MAY** cause you to miss some data,
if it was already ingested by another user Unfortunately, **Pulse** doesn't support creating
if it was already ingested by another user. Unfortunately **Pulse** doesn't support creating
queues with a guest account.

If your **Pulse User** was username: `foo` and password: `bar`, your Pulse URL
Expand Down Expand Up @@ -62,3 +62,65 @@ ex: <https://community-tc.services.mozilla.com/pulse-messages/>

[pulse guardian]: https://pulseguardian.mozilla.org/whats_pulse
[yml schema]: https://github.com/mozilla/treeherder/blob/master/schemas/pulse-job.yml

## Setup for Perfherder Development

These steps should help you set up everything you need to ingest data locally while working on perfherder.

**Create a pulse guardian account and run:**

```bash
export PULSE_URL=amqp://USER:PASSWORD@pulse.mozilla.org:5671/?ssl=1
yarn install
```

**Run each of these commands in a seperate window:**

```bash
docker-compose up --build
Comment thread
sumairq marked this conversation as resolved.
Outdated
docker-compose run -e PROJECTS_TO_INGEST=autoland backend celery -A treeherder worker --concurrency 1
```

---

**Run the db viewer**

You can use any database viewer of your choice `(e.g. dbeaver-ce, mysql workbench, etc.)`

**Connect to the following while `docker-compose up --build` from the previous step is running:**

`Serverhost: localhost`
`Port: 3306`
`Database: treeherder`
`Username: root`
`No password`

---

##### Run the following in separate window while running above to do ingestion
Comment thread
sumairq marked this conversation as resolved.
Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @sumairq, sorry for the delayed review! In the image below, I noticed that this line is showing up as a new heading in the navigation list/bar on the right-hand side. Could you reformat this line to be in bold instead so it doesn't show up there?

image


<!-- prettier-ignore -->
!!! note
These commands perform fetches for the data, they are run sequentially in the same window. The first command makes the second one run faster.

**Ingest push:**

```bash
docker-compose exec backend ./manage.py ingest push -p autoland -r 1ee42a54a431acdd6cbe43b49de0237fe67eddd9
```

**Ingest all the tasks, run celery to trigger the log parsing, and performance data ingestion:**

> **Warning:** This command can take a long time to ingest and parse everything.

```bash
Comment thread
sumairq marked this conversation as resolved.
Outdated
docker-compose exec backend ./manage.py ingest push -p autoland -r 1ee42a54a431acdd6cbe43b49de0237fe67eddd9 -a --enable-eager-celery
```

`--enable-eager-celery` triggers the log parsing which is required to capture the `PERFHERDER_DATA` output.

**For ingesting multiple pushes:**

```bash
docker-compose exec backend ./manage.py ingest push -p autoland --last-n-pushes 100
```