feat: revive datacareer.ch and integrate basel-jobs.ch

datacareer.ch had been returning 403 on every page for months. The cause
was our own User-Agent: its nginx inverts the usual bot rule and blocks
browser-like UAs while serving honest ones (no UA, curl, Googlebot).
Give it PLAIN_HEADERS instead of the shared spoofed-Chrome HEADERS.
Basel has 8 jobs on a single page with no pagination, so page 2 returning
0 items is correct rather than truncation.

basel-jobs.ch was documented as un-integrable because DataDome blocks
plain curl. Only half of that was true. Measured: plain fetch 403s on any
UA, headless Chrome over CDP 403s, Playwright's own launch 403s even
headful — but a real Chrome launched normally with --remote-debugging-port
and attached over CDP afterwards gets 200. Clearance then lives entirely
in a datadome cookie that replays fine over plain fetch with a matching
UA, so a browser is touched only to mint, never to scrape. The earlier
"cookie can't be extracted" note was true of the browser extension, not
of CDP.

Minting needs a visible window, so launchd runs can't do it; a stale
cookie makes the source log and yield nothing instead of failing the run.
Refresh by hand with `bun run mint-datadome`, or set LINKME_NO_BROWSER=1.

Three passes, all needed — they cover different sets. The informatik
category is a curated taxonomy, not the whole board, and the two search
modes query different indexes: websearch=1 finds role words and postings
filed outside IT (+147 over the category), while the bare search is the
only mode matching stack terms — react, frontend and backend return zero
under websearch=1. Language checks run against div.plain-content only;
against the whole page the German site chrome alone clears the DE/FR
threshold and drops every job, English ones included.

Also fix a pre-existing double-wrapped Promise in the extraSources type.
This commit is contained in:
2026-09-01 12:23:59 +02:00
parent b61a75eca0
commit d5e6605176
5 changed files with 446 additions and 10 deletions
+32 -4
View File
@@ -12,6 +12,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
| `bun daily --dry-run` | same, but print the message instead of sending |
| `bun daily --resend` | skip fetching, re-send today's jobs from the CSV (`--date=YYYY-MM-DD` to pick a day) |
| `bun run telegram.ts --setup` | print the chat ID for `.env` after messaging the bot |
| `bun run mint-datadome` | open a real Chrome window to refresh the basel-jobs.ch DataDome cookie |
Temp scripts: `bun run scripts/<name>-temp.ts` — never edit `config.ts` for one-off searches.
@@ -20,7 +21,7 @@ Temp scripts: `bun run scripts/<name>-temp.ts` — never edit `config.ts` for on
```
config.ts # search keywords and areas
fetch.ts # scraper, HTML parser, all filters
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch)
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch)
index.ts # orchestrator: runs searches, writes CSV
recheck.ts # raw-mode fetch, lists filtered-out jobs
daily.ts # digest: fetch → match against profile.ts → send to Telegram
@@ -42,11 +43,38 @@ plans/ # LinkedIn post drafts
- Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform) and `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform and Field add a Switzerland-wide remote pass.
- `RESEARCH_KEYWORDS` deliberately contains no bioinformatics/computational-biology terms — see `BIO_DOMAIN_RE`.
- **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`.
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch). Imported by `index.ts`.
- **datacareer.ch** currently returns **403** on every page — the source is dead and yields nothing. Needs new headers or a different entry point.
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch). Imported by `index.ts`.
- **datacareer.ch** inverts the usual bot rule: its nginx **403s browser-like User-Agents** and serves honest ones (no UA, `curl/8.0`, Googlebot) fine. The spoofed Chrome UA in the shared `HEADERS` was the entire cause of the long-standing 403 — it uses `PLAIN_HEADERS` instead. Small board: Basel returns **8 jobs on one page with no pagination**, so `?page=2` legitimately yields 0 items.
- **jobbasel.ch** looks like a JS-only React/Vite SPA (no jobs in the homepage's raw HTML), but its search route is server-rendered: `GET /job/suche-{slug}` returns full HTML with a `window.__PRELOADED_STATE__` blob containing the matched vacancies as JSON — no headless browser needed. Pagination is a URL suffix (`-seite-{n}`); a bare `?page=` query param is silently ignored. Swept with the same fixed keyword list as job-room.ch (no keyword-union endpoint exists) and deduped by vacancy id. Each item carries its full German description, so the DE/FR check runs in-process.
- **englishjobsearch.ch** also looks JS-only at first glance, but any page accepts `?format=markdown` and returns clean server-rendered markdown with full job cards — title, company, city, date, description. Hit the canton pages directly (`/in/canton-basel-stadt`, `/in/canton-basel-landschaft`) rather than the `/in/basel/` landing page, which only lists canton counts. Every job link is a `/clickout/<hash>` tracking redirect rather than the real posting URL — there's no way to recover the underlying employer link without following the redirect, so the clickout link is stored as-is.
- **basel-jobs.ch** was evaluated but is **not integrated**: it's protected by DataDome (not basic Cloudflare), which blocks plain `curl` outright. A real Chrome session gets through, confirming the content is genuine, but that only proves the block isn't automatable within this project's curl-only architecture — `bun daily`/`bun fetch` run headless via launchd with no browser available, and the session cookie that lets a real browser through can't be extracted for replay (browser tooling blocks cookie access). Would need persistent Playwright + stealth infra to chase further, which isn't worth it for what's likely a small regional board already covered by jobbasel.ch and stellen-basel.ch.
- **basel-jobs.ch** is behind DataDome and **is now integrated** (`fetchBaselJobs`), contrary to the earlier note that it couldn't be. What the block actually permits, tested:
- plain `curl`/`fetch` → 403 on **every** UA, honest or spoofed;
- Chrome `--headless=new` over CDP → 403 (headless is detected);
- Playwright's own `chromium.launch()`, even headful → 403 (launch-path artefacts);
- a **real Chrome launched normally** with `--remote-debugging-port`, attached over CDP afterwards → **200**.
Clearance lives entirely in a `datadome` cookie (Max-Age 1 year) that **replays fine over plain `fetch`** with a matching User-Agent — the old claim that it "can't be extracted" was true of the browser-extension tooling, not of CDP. So `mintDataDomeSession()` opens a real window once, caches `{cookie, ua}` to `results/.datadome.json` (gitignored), and every run after that is plain HTTP. A rotated `Set-Cookie` is folded back into the cache; a 403 triggers one re-mint.
**Minting needs a visible window, so launchd runs can't do it.** With a stale cookie an unattended run logs `unavailable`, returns `[]`, and the rest of the fetch continues — it never fails the run. Refresh by hand with `bun run mint-datadome`, or set `LINKME_NO_BROWSER=1` to suppress minting entirely.
**Two passes, both needed.** Pass 1 walks the `informatik-jobs` category (~300 postings over 3 pages). Pass 2 sweeps `BASELJOBS_TERMS` through `/search?terms={kw}&page=1&websearch=1`. The category is a *curated taxonomy, not the whole board* — it holds only what the board filed under IT, so tech roles parked under Industrie/Pharma/Bau and roles whose relevance lives in the description never show up in pass 1. Measured 2026-09-01: pass 2 added **147 postings the category missed** (incl. "Senior AI Infrastructure & Platform Engineer (Basel)", "Senior DevOps / Cloud Engineer (Kubernetes, CI/CD, Linux)"), while the category held **257 the search missed**. 296 → 442 unique, 48 → 68 viable.
**Both search modes are swept — they query different indexes and each is blind to the other.** Do not drop one as redundant:
| term | `websearch=1` | bare `?terms=` |
|---|---|---|
| engineer | 90 | 260 found |
| frontend | **0** | 2 |
| backend | **0** | 4 |
| react | **0** | 2 |
`websearch=1` matches role/domain words and reaches postings filed outside the IT category. The bare search is the **only** mode that matches stack terms — `react`, `frontend`, `backend`, `javascript` return literally zero under `websearch=1`. Neither paginates: both ignore `page` and return the whole result set in one response (bare caps around 100 cards). Measured 2026-09-01 against the category: websearch added 147, bare a further 139 raw (23 past the title filters). 48 → 68 → **88 viable**.
German stems (`entwickler`, `informatik`) are in the term list deliberately: search matches *description* text, so they surface English-titled postings the category never filed under IT; the language filters drop the German-language ones afterwards. The bare search on `architect`/`security` drags in a lot of German noise (*Architekt* = building architect, *Sicherheitsassistent*, *Gerüstbauer*) — that's expected and `NON_ENGLISH_RE`/`SOFTWARE_TITLE_RE` handle it, 139 raw → 23.
Pagination/dedup: page 4+ of the category still serves 2 sponsored slots, so **"no new ids" is the end condition**, not an empty list. Sponsored cards repeat within a page, across pages, and across both passes — dedupe on `data-gtm-jid`. Cards carry no date, so `postedAt` is the discovery date. Canton comes from `data-gtm-jp` (sometimes `"BL,BS"`).
**Language-check the `div.plain-content` block only, never the whole page.** The site chrome is German and on its own clears the ≥ 6 DE/FR word threshold — checking the full page dropped *every* job, English ones included. That block is the mobile rendering of the description; the desktop view iframes the same text, so it's the only copy in the server-rendered HTML.
- **Basel international orgs** (`fetchBaselIntlOrgs`) sweeps three boards under one label, each only a handful of postings:
- **BIS** — the career site is JS-rendered with no usable API, so we read the UNjobs mirror at `unjobs.org/offices/bis_182320855689338880`. Titles arrive as `Role, City, Country` and get split back apart. BIS also cycles Senior Software / Data / Data & Analytics Engineer roles through this board.
- **Swiss TPH** — Umantis at `jobs.swisstph.ch/Jobs/All`. Site chrome is German even on English postings, so `isEnglishUrl` would reject everything; `contentChecked: true` and the title filters decide instead.