bug: Workday API returning 422 for ~30 enterprise companies — 0 jobs from Goldman, JPMorgan, Salesforce, etc.

[ambicuity]

Problem

Approximately 30 Workday-hosted companies return 422 Unprocessable Entity on every scrape run, yielding 0 collected jobs from major employers.

Affected companies (partial list)

Microsoft, Cisco, Oracle, Salesforce, PayPal, JPMorgan, Goldman Sachs, Morgan Stanley, Bank of America, Capital One, American Express, Deloitte, PwC, EY, Accenture, McKinsey, BCG, AMD, Qualcomm, SAP, VMware, Lenovo, ServiceNow, Intuit, Splunk, Starbucks, Lockheed Martin, Raytheon, General Dynamics, L3Harris, Merck, AbbVie, Bristol Myers Squibb, UnitedHealth Group, Johnson & Johnson, and more.

Companies still working

Adobe (200 jobs), PayPal (200), NVIDIA (200), Intel (200), Dell (200), HP (200), Citi (200), Workday (200), Autodesk (200), Northrop Grumman (200), Boeing (200), Pfizer (200).

Root Cause Hypothesis

HTTP 422 from Workday's /wday/cxs/{tenant}/{site}/jobs endpoint signals the request payload no longer matches their expected schema. Workday likely updated their CXS API contract. The working companies may be on an older API version or a different tenant configuration.

Current Payload (in fetch_workday_jobs)

{
  "appliedFacets": {},
  "limit": 20,
  "offset": 0,
  "searchText": ""
}

Acceptance Criteria

• Capture a working Workday API request (e.g. via Adobe's tenant) and compare the payload/headers against a failing one (e.g. Goldman Sachs)
• Update fetch_workday_jobs payload schema to match the current Workday CXS API contract
• Add the X-Workday-Client-Application-Version or similar required headers if that is the differentiator
• Verify that previously-failing companies return job listings after the fix

Difficulty: Intermediate — requires inspecting working vs. failing Workday tenant requests

[coderabbitai]

Coding Plan

Summary

• Add Referer and Origin headers to Workday POST requests to satisfy browser-origin validation that many tenants enforce
• Preserve full URL paths (including locale prefixes like en-US/) in API URL construction to fix edge cases like Microsoft
• Update both primary and 404-fallback request paths to use consistent, complete headers
• Add test coverage to prevent regression of header and URL handling behavior

Design Choices

Design Choice 1: Are `Referer` and `Origin` headers sufficient, or are additional headers like CSRF tokens required?

Options Considered:

1. Add only Referer and Origin headers (standard browser CORS pattern)
2. Add Referer, Origin, plus attempt to extract CSRF tokens from initial page load
3. Add comprehensive browser-like headers including Accept-Language, Sec-Fetch-*, etc.

Chosen Option: 1

Rationale: Multiple sources indicate CSRF tokens are not required for the public listings endpoint, and Referer/Origin are the commonly cited missing pieces for 422 fixes.

Design Choice 2: Should the full `site_path` be used in API URL construction, or continue extracting just the last segment?

Options Considered:

1. Continue using site_id = site_path.split('/')[-1] (last segment only)
2. Use full site_path for all companies in URL construction

Chosen Option: 2

Rationale: Using the full site_path preserves locale prefixes like en-US/ while remaining backward-compatible with companies that have single-segment paths.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Add Required Browser-Origin Headers

This phase addresses the primary root cause—Workday's CXS API validates that requests appear to originate from the career site domain. Without proper Referer and Origin headers, many tenants reject requests with HTTP 422.

Task 1: Add Referer and Origin Headers to Primary Request

Update the headers dictionary in fetch_workday_jobs to include browser-origin headers that make the request look like it comes from the career site.

• In scripts/update_jobs.py within fetch_workday_jobs, construct Origin as https://{host} (the base domain)
• Construct Referer as the original workday_url from the company config (the human-facing career site URL)
• Add both headers to the existing headers dictionary alongside Content-Type, Accept, and User-Agent
• Consider adding Accept-Language: en-US,en;q=0.9 for consistency with browser requests

Task 2: Update 404 Fallback Headers

The 404 retry path currently strips headers down to just Content-Type, which defeats the fix. Ensure fallback requests also include the required headers.

• In the 404 fallback branch (around line 721), update the minimal header set to include Referer and Origin
• Either reuse the headers variable or construct a complete headers dict for the retry request
• Maintain consistency between primary and fallback request headers

🤖 Prompt for AI agents

Update `fetch_workday_jobs` in `scripts/update_jobs.py` to add the missing
`Referer` and `Origin` headers to all Workday POST requests so that tenants no
longer reject them with HTTP 422.

- In the primary request within `fetch_workday_jobs`, construct:
- `Origin` as `https://{host}` (the base domain derived from the company's
`workday_url`)
- `Referer` as the original `workday_url` from the company config (the
human-facing career site URL)
- Add both `Origin` and `Referer` to the existing headers dictionary alongside
`Content-Type`, `Accept`, and `User-Agent`
- Consider also adding `Accept-Language: en-US,en;q=0.9` for additional
browser-consistency
- In the 404 fallback branch (around line 721), update the minimal headers set
to also include `Referer` and `Origin` (either by reusing the
already-constructed headers variable or building a complete headers dict)
- Ensure the header set is consistent between the primary request and the 404
fallback retry

Phase 2: Fix URL Path Handling for Edge Cases

This phase corrects the API URL construction to handle locale-prefixed paths like Microsoft's en-US/Microsoft, which currently gets incorrectly truncated.

Task 1: Use Full Site Path in API URL Construction

Modify the URL construction logic to preserve the complete path instead of extracting only the last segment.

• In scripts/update_jobs.py, change the API URL construction from using site_id = site_path.split('/')[-1] to using site_path directly
• The URL template becomes: f"https://{host}/wday/cxs/{tenant}/{site_path}/jobs"
• This preserves locale prefixes (e.g., en-US/Microsoft) while remaining backward-compatible with single-segment paths (e.g., NVIDIAExternalCareerSite)
• Remove the now-unused site_id variable assignment

Task 2: Update 404 Fallback URL Logic

Verify the fallback URL construction remains correct for the alternate URL format where tenant is in the path.

• Review the 404 fallback branch where path_parts[0] becomes tenant and path_parts[-1] becomes site
• For URLs like https://wd5.myworkdayjobs.com/tenant/site, this logic should remain unchanged
• Ensure the fallback doesn't inadvertently break for multi-segment paths in this URL format

🤖 Prompt for AI agents

Fix the API URL construction logic in `fetch_workday_jobs` in
`scripts/update_jobs.py` to use the full `site_path` instead of only the last
path segment, resolving truncation issues for locale-prefixed URLs like
Microsoft's `en-US/Microsoft`.

- Replace the `site_id = site_path.split('/')[-1]` extraction with direct use of
`site_path` in the URL template
- Update the URL template to:
`f"https://{host}/wday/cxs/{tenant}/{site_path}/jobs"`
- Remove the now-unused `site_id` variable assignment
- In the 404 fallback branch, review the logic where `path_parts[0]` becomes
tenant and `path_parts[-1]` becomes site (for URLs like
`https://wd5.myworkdayjobs.com/tenant/site`)
- This fallback logic should remain unchanged since it handles a different URL
format
- Verify it does not inadvertently break for multi-segment paths in that URL
format

Phase 3: Update Test Coverage

Ensure the existing Workday test validates the new header behavior and add coverage for edge case URL patterns.

Task 1: Update Existing Workday Integration Test

Modify the test to verify that requests include the required Referer and Origin headers.

• In tests/test_domain_limiter.py, update test_limiter_integrates_with_workday_post_path to capture and assert on headers passed to limited_post
• Verify Referer matches the company's workday_url
• Verify Origin matches the expected https://{host} format

Task 2: Add Test for Locale-Prefixed URL Handling

Add a test case that verifies Microsoft-style URLs with locale prefixes construct the correct API endpoint.

• Add a test case in tests/test_domain_limiter.py using a company config like {"name": "TestCo", "workday_url": "https://testco.wd10.myworkdayjobs.com/en-US/TestCo_Careers"}
• Assert the constructed API URL includes the full path: /wday/cxs/testco/en-US/TestCo_Careers/jobs
• Verify the en-US/ prefix is preserved, not stripped

🤖 Prompt for AI agents

Update and extend the Workday tests in `tests/test_domain_limiter.py` to
validate the new header behavior and locale-prefixed URL handling introduced in
Phases 1 and 2.

- In the existing `test_limiter_integrates_with_workday_post_path` test:
  - Capture the headers passed to `limited_post`
  - Assert that `Referer` matches the company's `workday_url`
  - Assert that `Origin` matches the expected `https://{host}` format
- Add a new test case for locale-prefixed URL handling:
- Use a company config like `{"name": "TestCo", "workday_url":
"https://testco.wd10.myworkdayjobs.com/en-US/TestCo_Careers"}`
- Assert the constructed API URL includes the full path:
`/wday/cxs/testco/en-US/TestCo_Careers/jobs`
  - Verify the `en-US/` prefix is preserved and not stripped

Research

The Workday scraping implementation lives in scripts/update_jobs.py (lines 662-773) as fetch_workday_jobs. It iterates over companies defined in config.yml, constructs API URLs from the workday_url config values, and makes POST requests to /wday/cxs/{tenant}/{site}/jobs. The function uses limited_post() which wraps the shared HTTP_SESSION with domain-aware rate limiting. Currently, headers include only Content-Type, Accept, and User-Agent—missing the Referer and Origin headers that Workday commonly requires. Tests exist in tests/test_domain_limiter.py that mock the Workday fetch path.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Update `fetch_workday_jobs` in `scripts/update_jobs.py` to add the missing
`Referer` and `Origin` headers to all Workday POST requests so that tenants no
longer reject them with HTTP 422.

- In the primary request within `fetch_workday_jobs`, construct:
- `Origin` as `https://{host}` (the base domain derived from the company's
`workday_url`)
- `Referer` as the original `workday_url` from the company config (the
human-facing career site URL)
- Add both `Origin` and `Referer` to the existing headers dictionary alongside
`Content-Type`, `Accept`, and `User-Agent`
- Consider also adding `Accept-Language: en-US,en;q=0.9` for additional
browser-consistency
- In the 404 fallback branch (around line 721), update the minimal headers set
to also include `Referer` and `Origin` (either by reusing the
already-constructed headers variable or building a complete headers dict)
- Ensure the header set is consistent between the primary request and the 404
fallback retry
===============================================================================

Task: 2

Fix the API URL construction logic in `fetch_workday_jobs` in
`scripts/update_jobs.py` to use the full `site_path` instead of only the last
path segment, resolving truncation issues for locale-prefixed URLs like
Microsoft's `en-US/Microsoft`.

- Replace the `site_id = site_path.split('/')[-1]` extraction with direct use of
`site_path` in the URL template
- Update the URL template to:
`f"https://{host}/wday/cxs/{tenant}/{site_path}/jobs"`
- Remove the now-unused `site_id` variable assignment
- In the 404 fallback branch, review the logic where `path_parts[0]` becomes
tenant and `path_parts[-1]` becomes site (for URLs like
`https://wd5.myworkdayjobs.com/tenant/site`)
- This fallback logic should remain unchanged since it handles a different URL
format
- Verify it does not inadvertently break for multi-segment paths in that URL
format
===============================================================================

Task: 3

Update and extend the Workday tests in `tests/test_domain_limiter.py` to
validate the new header behavior and locale-prefixed URL handling introduced in
Phases 1 and 2.

- In the existing `test_limiter_integrates_with_workday_post_path` test:
  - Capture the headers passed to `limited_post`
  - Assert that `Referer` matches the company's `workday_url`
  - Assert that `Origin` matches the expected `https://{host}` format
- Add a new test case for locale-prefixed URL handling:
- Use a company config like `{"name": "TestCo", "workday_url":
"https://testco.wd10.myworkdayjobs.com/en-US/TestCo_Careers"}`
- Assert the constructed API URL includes the full path:
`/wday/cxs/testco/en-US/TestCo_Careers/jobs`
  - Verify the `en-US/` prefix is preserved and not stripped

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!