feat: serve markdown automatically based on `Accept` header #1997

B4nan · 2025-10-10T11:18:51Z

Alters the nginx config so it checks the Accept header, if it contains text/plain or text/markdown, we serve the markdown version directly. Also added tests for this behavior to the existing PR check.

The previews are likely using the config from master, so there it doesn't work yet (it should after we merge this).

Since the test workflow now effectively verifies the nginx config is valid, I guess we can drop the nginx.conf-test.yaml workflow.

apify-service-account · 2025-10-10T11:22:14Z

Preview for this PR was built for commit daf5be0 and is ready at https://pr-1997.preview.docs.apify.com!

apify-service-account · 2025-10-10T11:25:39Z

Preview for this PR was built for commit 6c7c301 and is ready at https://pr-1997.preview.docs.apify.com!

apify-service-account · 2025-10-10T11:28:43Z

Preview for this PR was built for commit 6690f27e and is ready at https://pr-1997.preview.docs.apify.com!

apify-service-account · 2025-10-10T11:32:00Z

Preview for this PR was built for commit bc6a4e82 and is ready at https://pr-1997.preview.docs.apify.com!

apify-service-account · 2025-10-10T11:35:09Z

Preview for this PR was built for commit 6da05f72 and is ready at https://pr-1997.preview.docs.apify.com!

apify-service-account · 2025-10-10T11:38:14Z

Preview for this PR was built for commit bae10977 and is ready at https://pr-1997.preview.docs.apify.com!

apify-service-account · 2025-10-10T11:46:56Z

Preview for this PR was built for commit e9e37bfd and is ready at https://pr-1997.preview.docs.apify.com!

apify-service-account · 2025-10-10T11:53:42Z

Preview for this PR was built for commit 6bac35b6 and is ready at https://pr-1997.preview.docs.apify.com!

Copilot

Pull Request Overview

This pull request adds automatic markdown content serving based on HTTP Accept headers. When a client requests text/plain or text/markdown content types, nginx serves the markdown version of pages directly instead of the HTML version.

Implements nginx map directive to detect markdown-accepting clients
Adds location-specific handling for different URL patterns (homepage, llms.txt files, and generic docs)
Includes comprehensive CI tests to validate header-based content negotiation
Fixes a typo in localhost IP address documentation

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
nginx.conf	Core nginx configuration implementing Accept header detection and conditional markdown serving
.github/workflows/test.yaml	Adds integration tests to verify proper Content-Type headers for various URL patterns
CONTRIBUTING.md	Fixes typo in localhost IP address (127.0.01 → 127.0.0.1)

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

apify-service-account · 2025-10-10T12:25:28Z

Preview for this PR was built for commit e5a4bfa9 and is ready at https://pr-1997.preview.docs.apify.com!

jancurn · 2025-10-10T13:37:34Z

Great job guys! Pls let me know once it's deployed, will test it

B4nan · 2025-10-10T13:41:59Z

One thing I just realized, this is only working for the apify-docs stuff, not for CLI, SDKs, and clients. I'd first get this out before I try to make it work there.

janbuchar · 2025-10-10T14:06:21Z

.github/workflows/test.yaml

+
+            -   name: Start Docusaurus server
+                run: |
+                    nohup npx docusaurus serve --port 3000 --no-open &


Instead of this nohup and sleep goulash, I suggest using background-action like I do here https://github.com/janbuchar/website/blob/cac7908f1834a79788264f1ca170fc70717d29b0/.github/workflows/deploy.yaml#L47-L52

I guess we need apt install wait? Or how come it works for you in the other repo? :D 0e1d0e2

Error: [Error: ENOENT: no such file or directory, lstat '/home/runner/work/apify-docs/apify-docs/wait'] {

edit: reverting for now, can't find anything about why this fails

janbuchar · 2025-10-10T14:10:09Z

nginx.conf

Good call with the tests, I never would've believed this without them

apify-service-account · 2025-10-10T14:25:18Z

Preview for this PR was built for commit 0e1d0e29 and is ready at https://pr-1997.preview.docs.apify.com!

apify-service-account · 2025-10-10T15:45:30Z

Preview for this PR was built for commit 24376b7a and is ready at https://pr-1997.preview.docs.apify.com!

apify-service-account · 2025-10-10T15:52:56Z

Preview for this PR was built for commit 7b4ff27d and is ready at https://pr-1997.preview.docs.apify.com!

apify-service-account · 2025-10-10T16:00:37Z

Preview for this PR was built for commit e5a4bfa9 and is ready at https://pr-1997.preview.docs.apify.com!

lukasmrtvy

I would suggest using containers (https://docs.github.com/en/actions/how-tos/write-workflows/choose-where-workflows-run/run-jobs-in-a-container) directly inside the workflow, otherwise apt might be slow and unstable.

Related: #1997

github-actions bot assigned B4nan Oct 10, 2025

github-actions bot added this to the 125th sprint - Tooling team milestone Oct 10, 2025

github-actions bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Oct 10, 2025

B4nan force-pushed the respect-accept-header branch from daf5be0 to 6c7c301 Compare October 10, 2025 11:20

B4nan added the adhoc Ad-hoc unplanned task added during the sprint. label Oct 10, 2025

B4nan force-pushed the respect-accept-header branch from 6c7c301 to 6690f27 Compare October 10, 2025 11:24

B4nan force-pushed the respect-accept-header branch from 6690f27 to bc6a4e8 Compare October 10, 2025 11:28

B4nan force-pushed the respect-accept-header branch from bc6a4e8 to 6da05f7 Compare October 10, 2025 11:29

B4nan force-pushed the respect-accept-header branch from 6da05f7 to bae1097 Compare October 10, 2025 11:33

B4nan force-pushed the respect-accept-header branch from bae1097 to e9e37bf Compare October 10, 2025 11:43

feat: serve markdown automatically based on Accept header

6bac35b

B4nan force-pushed the respect-accept-header branch from e9e37bf to 6bac35b Compare October 10, 2025 11:49

B4nan marked this pull request as ready for review October 10, 2025 11:50

B4nan requested review from Copilot, janbuchar and lukasmrtvy October 10, 2025 11:59

Copilot AI reviewed Oct 10, 2025

View reviewed changes

drop the old nginx config workflow

e5a4bfa

B4nan mentioned this pull request Oct 10, 2025

Improve markdown structure of docs llms.txt & Actor detail #1783

Open

janbuchar approved these changes Oct 10, 2025

View reviewed changes

B4nan force-pushed the respect-accept-header branch from 24376b7 to 7b4ff27 Compare October 10, 2025 15:49

B4nan force-pushed the respect-accept-header branch from 7b4ff27 to e5a4bfa Compare October 10, 2025 15:57

lukasmrtvy approved these changes Oct 13, 2025

View reviewed changes

B4nan merged commit a712db3 into master Oct 14, 2025
18 checks passed

B4nan deleted the respect-accept-header branch October 14, 2025 10:44

B4nan added a commit that referenced this pull request Oct 14, 2025

fix: use production URL for proxy pass

5245af9

Related: #1997

B4nan added a commit that referenced this pull request Oct 14, 2025

fix: use production URL for proxy pass

6825b4a

Related: #1997

B4nan added a commit that referenced this pull request Oct 14, 2025

fix: use production URL for proxy pass

e235d84

Related: #1997

B4nan added a commit that referenced this pull request Oct 14, 2025

fix: use production URL for proxy pass

e67a290

Related: #1997

B4nan added a commit that referenced this pull request Oct 14, 2025

fix: use production URL for proxy pass

226965e

Related: #1997

B4nan added a commit that referenced this pull request Oct 14, 2025

fix: use production URL for proxy pass

1b725bf

Related: #1997

B4nan added a commit that referenced this pull request Oct 14, 2025

fix: rewrite nginx config to support production GH pages URLs

b7eebba

Related: #1997

B4nan added a commit that referenced this pull request Oct 14, 2025

fix: rewrite nginx config to support production GH pages URLs

ab61103

Related: #1997

B4nan added a commit that referenced this pull request Oct 14, 2025

fix: revert nginx config back

4ffca08

Related: #1997

B4nan mentioned this pull request Oct 14, 2025

fix: revert nginx config back #2011

Merged

B4nan added a commit that referenced this pull request Oct 14, 2025

fix: revert nginx config back (#2011)

28780dd

Related: #1997

B4nan added a commit that referenced this pull request Oct 14, 2025

fix: revert nginx test

016cfd6

Related: #1997

B4nan mentioned this pull request Oct 14, 2025

fix: revert nginx test #2012

Merged

B4nan added a commit that referenced this pull request Oct 14, 2025

fix: revert nginx test

4242b17

Related: #1997

B4nan added a commit that referenced this pull request Oct 14, 2025

fix: revert nginx test (#2012)

0d43d7c

Related: #1997

B4nan added a commit that referenced this pull request Oct 14, 2025

fix: revert nginx 3

4196062

Related: #1997

B4nan mentioned this pull request Oct 14, 2025

fix: revert nginx 3 #2013

Merged

B4nan added a commit that referenced this pull request Oct 14, 2025

fix: revert nginx 3 (#2013)

f7000af

Related: #1997

feat: serve markdown automatically based on Accept header #1997

feat: serve markdown automatically based on Accept header #1997

Uh oh!

Conversation

B4nan commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

apify-service-account commented Oct 10, 2025

Uh oh!

apify-service-account commented Oct 10, 2025

Uh oh!

apify-service-account commented Oct 10, 2025

Uh oh!

apify-service-account commented Oct 10, 2025

Uh oh!

apify-service-account commented Oct 10, 2025

Uh oh!

apify-service-account commented Oct 10, 2025

Uh oh!

apify-service-account commented Oct 10, 2025

Uh oh!

apify-service-account commented Oct 10, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

apify-service-account commented Oct 10, 2025

Uh oh!

jancurn commented Oct 10, 2025

Uh oh!

B4nan commented Oct 10, 2025

Uh oh!

janbuchar Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

B4nan Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janbuchar Oct 10, 2025

Choose a reason for hiding this comment

Uh oh!

apify-service-account commented Oct 10, 2025

Uh oh!

apify-service-account commented Oct 10, 2025

Uh oh!

apify-service-account commented Oct 10, 2025

Uh oh!

apify-service-account commented Oct 10, 2025

Uh oh!

lukasmrtvy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

feat: serve markdown automatically based on `Accept` header #1997

feat: serve markdown automatically based on `Accept` header #1997

B4nan commented Oct 10, 2025 •

edited

Loading

B4nan Oct 10, 2025 •

edited

Loading