Merge pull request #2170 from brefphp/improve-monitoring-docs

Monitoring docs: mention the new "Performance" page and various improvements

Author: mnapoli

docs/cloud.mdx

@@ -8,13 +8,42 @@ import Image from 'next/image';
 
 # Bref Cloud documentation
 
-Bref Cloud makes it easy to deploy and monitor serverless PHP applications on AWS Lambda. Learn more about it [on the Bref Cloud homepage](https://bref.sh/cloud).
+[Bref Cloud](/cloud) is the service that complements the open-source Bref project. It deploys, monitors and operates serverless PHP applications on AWS Lambda, in your own AWS account, without the AWS complexity.
 
-To get started, [create a free Bref Cloud account](https://bref.cloud/register).
-
-Bref Cloud is a service that complements the open-source Bref project. It connects to all your AWS accounts to simplify permissions and access management, and provides a dashboard to manage your serverless applications.
+To get started, [create a Bref Cloud account](https://bref.cloud/register). There is a free plan for personal projects, and paid plans come with a free trial. See the [pricing](/cloud#pricing).
 
 <Cards num={2}>
     <Cards.Card icon={<Image src={brefIcon} className="size-6" />} title="Learn more about Bref Cloud" href="/cloud" />
     <Cards.Card icon={<Image src={brefIcon} className="size-6" />} arrow={true} title="Get started with Bref Cloud" href="https://bref.cloud/register" />
 </Cards>
+
+## What Bref Cloud does
+
+### Deploy
+
+- **One command**: `bref deploy` from your machine or [from GitHub Actions](./cloud-deploy.mdx). No AWS credentials to create, distribute or rotate: Bref Cloud generates [short-lived credentials](./cloud-security.mdx) for every deployment.
+- **Databases and networks**: create MySQL or PostgreSQL databases (fixed instances, or serverless databases that scale and pause on their own) and private networks in a few clicks.
+- **Environments**: manage dev, staging and production across AWS accounts and regions from one dashboard, with the history of every deployment.
+
+### Monitor
+
+- **Overview**: a diagram of each environment's architecture with live metrics on every component.
+- **Logs**: search logs or tail them in real time.
+- **Metrics**: Lambda invocations, concurrency, duration and errors, API Gateway, queues and dead-letter queues.
+- **Traces**: a trace explorer optimized for PHP applications, with a free [Bref X-Ray](/xray) license included. See database queries, HTTP calls and AWS calls inside each invocation, and filter traces by route, job, command, cold start, or your own annotations, over up to 30 days.
+- **Performance**: the slowest database queries, the latency of every route, and the slowest jobs of your application, computed from the last 30 days of traces.
+- **Health checks**: verify that the database, the cache and the Lambda configuration of a deployed application are healthy (Laravel only for now).
+
+[Learn more about monitoring with Bref Cloud](/docs/monitoring#bref-cloud).
+
+### Operate
+
+- **Queues**: watch queues and dead-letter queues, and retry, delete or flush failed Laravel jobs.
+- **Commands**: run console commands (Artisan, Symfony console…) on a deployed environment from the dashboard.
+- **Secrets**: create and manage secrets for all your applications.
+- **Files**: browse, upload, rename and delete files in S3 buckets.
+
+### Team and security
+
+- **Team members**: invite teammates with read-only, write or admin access. They never need AWS access.
+- **AWS access**: Bref Cloud connects to your AWS accounts through IAM roles and temporary credentials, never long-lived access keys. [Learn more](./cloud-security.mdx).

docs/monitoring/cloud-metrics.png

@@ -4,10 +4,11 @@ asIndexPage: true
 
 import Image from 'next/image';
 import cloudXrayTrace from './cloud-xray-trace.png';
-import cloudXrayFiltersGeneral from './cloud-xray-filters-general.png';
-import cloudXrayFiltersAnnotations from './cloud-xray-filters-annotations.png';
 import cloudOverview from './cloud-overview.png';
 import cloudLogs from './cloud-logs.png';
+import cloudMetrics from './cloud-metrics.png';
+import cloudTraces from './cloud-traces.png';
+import cloudPerformance from './cloud-performance.png';
 
 # Monitoring
 
@@ -17,7 +18,7 @@ Here is a summary of recommended tools for monitoring Bref applications:
 
 - **Logs and metrics**: CloudWatch (built-in), [Bref Cloud](/cloud)
 - **Error tracking**: [Sentry](https://sentry.io) and similar services
-- **Performance tracing**: [Bref Cloud](/cloud) with [X-Ray](/xray)
+- **Tracing and performance insights**: [Bref Cloud](/cloud) with [X-Ray](/xray)
 
 Let's dive into the details of each of them.
 
@@ -29,27 +30,59 @@ For more advanced use cases, such as tracking Lambda errors outside of PHP-FPM (
 
 ## Bref Cloud
 
-[Bref Cloud](/cloud) provides an all-in-one monitoring experience for serverless PHP applications:
+[Bref Cloud](/cloud) monitors serverless PHP applications. It reads logs, metrics and traces from your AWS account. There is no agent to install.
+
+### Overview
+
+The overview page shows a diagram of the environment: Lambda functions, API Gateway, CloudFront, SQS queues, S3 buckets and databases. Each component shows live metrics: requests, invocations, errors, queue size, storage.
 
 <Image className="mt-3 rounded-lg shadow-md border" src={cloudOverview} alt="Bref Cloud application overview" />
 
-- **Logs**: view and search CloudWatch logs in a simplified UI.
+### Logs
+
+View, search and tail CloudWatch logs. Laravel and Symfony logs are [structured](/docs/environment/logs): you can filter them by log level or exception class.
 
 <Image className="mt-3 rounded-lg shadow-md border" src={cloudLogs} alt="Log viewer in Bref Cloud" />
 
-- **Metrics**: monitor Lambda invocations, duration, errors, and more.
-- **Traces**: visualize X-Ray traces to understand the performance of your application, including database queries, HTTP calls, and AWS SDK calls.
+### Metrics
+
+The metrics page shows Lambda invocations, duration and errors, as well as API Gateway requests and latency. Queue metrics can also be found in the "Queues" page. You can add graphs to compare metrics over up to 30 days.
+
+<Image className="mt-3 rounded-lg shadow-md border" src={cloudMetrics} alt="Metrics in Bref Cloud" />
+
+### Traces
+
+A trace shows what happens inside an invocation: PHP code, database queries, HTTP calls, and AWS SDK calls. For each AWS call, the trace shows the queue, table, or topic it used.
 
 <Image className="mt-3 rounded-lg shadow-md border" src={cloudXrayTrace} alt="X-Ray trace in Bref Cloud" />
 
-With the [Bref X-Ray](/xray) package, traces are enriched with annotations that you can use to filter and search. For example, you can filter traces by specific controller, route, CLI command, or job class:
+The [Bref X-Ray package](/xray) adds annotations to traces: route, controller, job class, CLI command, cold start. You can add [your own annotations](/xray/docs#custom-annotations), for example the tenant or the plan of the user. The trace explorer uses annotations as filters. For example, you can list all the traces of one route, of one job, of one tenant, or all the invocations with a cold start.
+
+<Image className="mt-3 rounded-lg shadow-md border" src={cloudTraces} alt="Trace explorer in Bref Cloud" />
+
+Enable [AWS Transaction Search](/xray/docs#enabling-aws-transaction-search) on your AWS account, and the trace explorer searches up to 30 days of traces instead of 6 hours. It is a one-time switch in the AWS console.
+
+### Performance
+
+The Performance page aggregates the traces of the last 30 days. It shows the slowest database queries, the latency of each route, and the slowest jobs.
+
+<Image className="mt-3 rounded-lg shadow-md border" src={cloudPerformance} alt="Performance page in Bref Cloud" />
+
+- **Slowest database queries**: SQL queries grouped by statement, with the number of calls, the average duration and the p95 duration.
+- **Routes**: server-side processing time per route, sorted by number of requests or by latency. Click a route to open its traces.
+- **Slowest jobs**: queue worker invocations grouped by job class. Click a job to open its traces.
+
+The page requires the [Bref X-Ray package](/xray) in the application and [Transaction Search](/xray/docs#enabling-aws-transaction-search) on the AWS account. After that, there is nothing else to configure. Bref Cloud computes the statistics when you open the page, caches them for 24 hours, and lets you recompute them at any time. The statistics only include traced invocations (see the [X-Ray sampling rate](/xray/docs#costs)).
+
+### Failed jobs and health checks
+
+Laravel applications have a "Failed jobs" tab next to their queues. It shows the exception of each failed job. You can retry, delete or flush jobs from there.
+
+Health checks check that a deployed application works: the database is reachable, the cache works, the Lambda functions use the recommended settings. Laravel only for now.
 
-<div className="flex gap-4 mt-3 items-start">
-    <Image className="rounded-lg shadow-md border h-auto" style={{maxHeight: 350, width: 'auto'}} src={cloudXrayFiltersGeneral} alt="X-Ray trace filters in Bref Cloud" />
-    <Image className="rounded-lg shadow-md border h-auto" style={{maxHeight: 350, width: 'auto'}} src={cloudXrayFiltersAnnotations} alt="X-Ray trace annotation filters in Bref Cloud" />
-</div>
+### X-Ray license
 
-Bref Cloud users get a **free [Bref X-Ray](/xray) license** included in their plan to enable performance tracing. To activate it, contact support via [bref.cloud/support](https://bref.cloud/support) or [Slack](https://bref.sh/slack).
+Bref Cloud paid plans include a free [Bref X-Ray](/xray) license. To activate it, contact support via [bref.cloud/support](https://bref.cloud/support) or [Slack](https://bref.sh/slack).
 
 [Learn more about Bref Cloud](/cloud).
 

docs/monitoring/cloud-performance.png

@@ -104,13 +104,15 @@ Send me a Slack message or an email (matthieu@bref.sh) if you have any trouble,
 
 ## Enabling AWS Transaction Search
 
-[AWS Transaction Search](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search.html) is an AWS service that indexes X-Ray traces and replaces "X-Ray classic" search.
+[AWS Transaction Search](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search.html) is an AWS service that indexes X-Ray traces and replaces "X-Ray classic" search. It is opt-in, and **we recommend enabling it**, whether you explore traces in Bref Cloud or in the AWS console:
 
-**Enable it** and Bref Cloud tracing gets much better:
+- search up to 30 days of traces instead of only 6 hours
+- filter traces by route, job, command, cold start, or [any custom annotation](#custom-annotations)
+- spans are stored in CloudWatch Logs (the `aws/spans` log group), so you can also query them with CloudWatch Logs Insights, for example to aggregate durations by route or by SQL query
 
-- up to 30 days of traces instead of only 6 hours
-- filter traces by route, job, command, cold start, etc.
-- search all spans rather than only the percentage indexed by "classic X-Ray"
+In Bref Cloud, Transaction Search also unlocks the [Performance page](/docs/monitoring#performance): slowest database queries, latency per route, slowest jobs.
+
+If you don't use Bref Cloud, explore traces from the **Transaction Search** page of the CloudWatch console. The classic X-Ray console keeps working but only sees the indexed percentage (see below).
 
 Downsides:
 
@@ -127,9 +129,9 @@ The change is reversible. To enable Transaction Search ([official guide](https:/
 
     <Image className="mt-3 rounded-lg shadow-md border" src={transactionSearchSettings} alt="Transaction Search settings for X-Ray users" />
 
-The indexing percentage does _not_ impact Bref Cloud (it is **not** the Lambda sampling rate mentioned in [Costs](#costs) below), leave it at 1% by default. Increase it only if you still use classic X-Ray.
+**Leave the indexing percentage at 1%** unless you know what you are doing. It only affects the classic X-Ray console and APIs. It is not the Lambda sampling rate (see [Costs](#costs)), and it has no effect on Bref Cloud or on the Transaction Search page.
 
-It can take up to 10 minutes to activate Transaction Search and see it in Bref Cloud.
+It can take up to 10 minutes to activate Transaction Search and see spans show up, in Bref Cloud or in the CloudWatch console.
 
 ## Costs
 
@@ -150,11 +152,11 @@ This is quite useful: when an application has low traffic, most requests/invocat
 
 ### Laravel
 
-If you are using Laravel, the package automatically integrates with the Laravel service provider. You can see the traces in the AWS X-Ray console.
+If you are using Laravel, the package automatically integrates with the Laravel service provider. You can see the traces in [Bref Cloud](/docs/monitoring#bref-cloud) or in the AWS console.
 
 #### Request tracing
 
-The package automatically traces all requests to your application. You can see the traces in the AWS X-Ray console.
+The package automatically traces all requests to your application.
 
 #### Database queries
 
@@ -164,7 +166,7 @@ Eloquent queries are automatically traced and the SQL queries are added to metad
 
 #### Jobs
 
-The package automatically traces all jobs dispatched via the Laravel queue. You can see the traces in the AWS X-Ray console, and filter them by job name.
+The package automatically traces all jobs dispatched via the Laravel queue. Traces can be filtered by job name.
 
 ![](./laravel-jobs.png)
 
@@ -178,7 +180,7 @@ The package automatically traces all HTTP requests made via the Laravel HTTP cli
 
 ### Symfony
 
-If you are using Symfony, the package automatically integrates with the Symfony service provider. You can see the traces in the AWS X-Ray console.
+If you are using Symfony, the package automatically integrates with the Symfony service provider. You can see the traces in [Bref Cloud](/docs/monitoring#bref-cloud) or in the AWS console.
 
 #### Symfony Messenger
 
@@ -282,6 +284,51 @@ XRay::subSegment('doSomething', function () {
 });
 ```
 
+### Custom annotations
+
+Annotations are indexed key-value pairs attached to a trace. Bref adds `Route`, `Controller`, `Job`, `Command`, and cold start annotations automatically. You can add your own, for example the tenant or the plan of the current user:
+
+```php
+use Bref\Apm\XRay\XRay;
+
+XRay::trace()->addAnnotation('tenant', $tenant->slug);
+```
+
+Call it during the request or the job, for example from a middleware or an event listener. Do not call it at boot time (for example in the `boot()` method of a service provider): the trace is reset after every invocation, so an annotation added at boot is only attached to the first invocation of the Lambda instance.
+
+For example, with a Laravel middleware registered in the `web` middleware group:
+
+```php
+namespace App\Http\Middleware;
+
+use Bref\Apm\XRay\XRay;
+use Closure;
+use Illuminate\Http\Request;
+
+class AnnotateTrace
+{
+    public function handle(Request $request, Closure $next)
+    {
+        $tenant = $request->user()?->tenant;
+        if ($tenant) {
+            XRay::trace()->addAnnotation('tenant', $tenant->slug);
+        }
+
+        return $next($request);
+    }
+}
+```
+
+X-Ray has [constraints on annotations](https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-annotations):
+
+- keys can only contain letters, numbers and underscores (`feature_flag` is valid, `feature-flag` is not)
+- values are strings
+- a trace can have at most 50 annotations
+
+The package does not validate these constraints, and AWS does not report invalid annotations back to your application.
+
+In Bref Cloud, every annotation becomes a filter in the trace explorer (with [Transaction Search](#enabling-aws-transaction-search) enabled). For example, you can list all the traces of a given tenant over the last 30 days. Keep the number of possible values limited (an identifier or a category, not free text).
+
 ### Tracing AWS SDK calls
 
 Trace AWS SDK calls with the `AwsSdkTracer::trace($client)` method.
@@ -297,7 +344,7 @@ AwsSdkTracer::trace($client);
 
 ![](./sdk-tracing.png)
 
-All calls will be traced and appear in the X-Ray console. However, only the first 10 requests will be sampled (i.e. will result in propagating tracing to the target AWS service). This is to avoid oversampling and hitting the X-Ray limit of 100 linked traces. For example, if a Lambda function sends 100 messages to SQS, only the first 10 Lambda invocations triggered by the SQS messages will create sub-traces. The other 90 invocations will appear as segments in the original trace, but will not create sub-traces.
+All calls will be traced and appear in the trace. However, only the first 10 requests will be sampled (i.e. will result in propagating tracing to the target AWS service). This is to avoid oversampling and hitting the X-Ray limit of 100 linked traces. For example, if a Lambda function sends 100 messages to SQS, only the first 10 Lambda invocations triggered by the SQS messages will create sub-traces. The other 90 invocations will appear as segments in the original trace, but will not create sub-traces.
 
 ### Tracing Async-AWS SDK calls
 
@@ -326,7 +373,7 @@ $httpClient = new TracedAsyncAwsHttpClient(HttpClient::create([
 $sqs = new SqsClient([ /* config */ ], null, $httpClient);
 ```
 
-All calls will be traced and appear in the X-Ray console. However, only the first 10 requests will be sampled (i.e. will result in propagating tracing to the target AWS service). This is to avoid oversampling and hitting the X-Ray limit of 100 linked traces. For example, if a Lambda function sends 100 messages to SQS, only the first 10 Lambda invocations triggered by the SQS messages will create sub-traces. The other 90 invocations will appear as segments in the original trace, but will not create sub-traces.
+All calls will be traced and appear in the trace. However, only the first 10 requests will be sampled (i.e. will result in propagating tracing to the target AWS service). This is to avoid oversampling and hitting the X-Ray limit of 100 linked traces. For example, if a Lambda function sends 100 messages to SQS, only the first 10 Lambda invocations triggered by the SQS messages will create sub-traces. The other 90 invocations will appear as segments in the original trace, but will not create sub-traces.
 
 ### Tracing Guzzle calls