feat: add data-jobs.ch and arbeitnow.ch sources

CLAUDE.md

@@ -21,7 +21,7 @@ Temp scripts: `bun run scripts/<name>-temp.ts` — never edit `config.ts` for on
 ```
 config.ts          # search keywords and areas
 fetch.ts           # scraper, HTML parser, all filters
-fetch-other.ts     # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch, startup.jobs)
+fetch-other.ts     # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch + data-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch, startup.jobs, arbeitnow.ch)
 index.ts           # orchestrator: runs searches, writes CSV
 recheck.ts         # raw-mode fetch, lists filtered-out jobs
 daily.ts           # digest: fetch → match against profile.ts → send to Telegram
@@ -40,10 +40,10 @@ plans/             # LinkedIn post drafts
 ## Architecture
 
 - **`config.ts`** — `KEYWORDS` string (joined with ` OR `) and `searches: SearchConfig[]`. Edit here to add/remove search areas or keywords.
-  - Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform) and `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform and Field add a Switzerland-wide remote pass.
+  - Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform), `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product) and `FRONTEND_KEYWORDS` (frontend/React/web, data visualisation, geospatial/GIS, WebGL — added 2026-10-20 after a one-off search found 14 of 18 relevant frontend/viz roles missing from the CSV). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform, Field and Frontend add a Switzerland-wide remote pass.
   - `RESEARCH_KEYWORDS` deliberately contains no bioinformatics/computational-biology terms — see `BIO_DOMAIN_RE`.
 - **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`.
-- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch, startup.jobs). Imported by `index.ts`.
+- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch + data-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch, startup.jobs, arbeitnow.ch). Imported by `index.ts`.
   - **novartis.com** (`fetchNovartisBasel`, added 2026-09-21) — Rafael has an internal contact at Novartis Basel who can refer him, so this source exists purely to catch a genuine software/engineering opening there, not to widen the CSV generally. Its career-search page (`novartis.com/careers/career-search?country[]=LOC_CH`) is a server-rendered Drupal view — no headless browser needed — but only at the **locale-less English path** (`/careers/career-search`); the `ch-de` German path's job-detail links 302 straight back to the search page for both locales when hit without a session, so detail pages only resolve through the English path. Pagination is 0-indexed (`page=0` implicit, `page=1` is the second page of 10) and the page's own "Showing N results" line gives the exact total, so the page-loop bound is computed rather than probed — the listing has also been observed to **repeat rows across pages within one sweep** (measured 2026-09-21: several titles appeared twice across 5 pages), so results are deduped by `req-<id>` before filtering, not trusted from the pager.
 
     Novartis Basel's titling is almost entirely non-technical "Director, `<business function>`" roles (Portfolio Analytics, Therapeutic Area Strategy, Policy Communications, ...) plus device/hardware "Engineering" roles (autoinjector/drug-delivery hardware, not software) — the generic `SOFTWARE_TITLE_RE` used by every other source in this file matched **13 of 47** CH listings on title alone in an initial run, and every single one was a false positive (no engineering-team role among them). `NOVARTIS_TITLE_RE` replaces it here: it requires an explicit software/IT engineering job-family term (`software engineer`, `platform engineer`, `data engineer`, `devops`, `sre`, `solutions architect`, ...), not just a bare seniority or business-function word. `BIO_DOMAIN_RE` and job-room.ch's `PHARMA_ROLE_RE` still run on top to catch domain-science titles the stricter regex wouldn't otherwise need to worry about. Narrowed to Basel/Muttenz specifically (`NOVARTIS_BASEL_RE`) — the CH listing also includes Stein (Aargau), a different commute the contact couldn't refer into. As of 2026-09-21 this source keeps **zero** listings — expected; it's watching for the rare opening, not sweeping volume. Each detail page carries the full description inline (`job_details_content_center`…`job_details_content_bottom`), so the DE/FR + `requiresGerman` check runs in-process per candidate rather than costing `index.ts` a page fetch.
@@ -78,6 +78,8 @@ plans/             # LinkedIn post drafts
 
     Pagination/dedup: page 4+ of the category still serves 2 sponsored slots, so **"no new ids" is the end condition**, not an empty list. Sponsored cards repeat within a page, across pages, and across both passes — dedupe on `data-gtm-jid`. Cards carry no date, so `postedAt` is the discovery date. Canton comes from `data-gtm-jp` (sometimes `"BL,BS"`).
 
+    **Pass 3: data-jobs.ch** (added 2026-10-22) — the same jobchannel network's data/analytics board, swept for BS/BL/ZH/BE/LU (`DATAJOBS_PROVINCES`) inside `fetchBaselJobs`: same card markup and parser, and the same DataDome cookie clears both hosts (a cookie rotated by one host works on the other). `/search?provinces=…&lang=en-ch&page=n` walks the board's own listings (100 per page; past the end it still serves a few sponsored cards, so "no new ids" ends the walk), then one `&page=1&websearch=1` request returns all the "further jobs from the web search" (it ignores `page`). Measured 2026-10-22: 166 own + 98 web search = 264 unique for the five cantons. Vacancy ids are network-wide, so it runs last and shares `seen` — a posting on both boards keeps its basel-jobs.ch link. Its inventory is **not** a subset of basel-jobs.ch (data-only postings 404 there): measured 2026-10-22, 26 ids the basel-jobs passes missed. A posting on both boards keeps the same `/job/<slug>/<id>` path, so `knownJobchannelTails()` skips any card whose path is already in the CSV under either host — before its detail fetch, not after. Re-checking every known posting on every run is what got the cookie blocked on the first five-canton test (2026-10-22). For the same reason a detail page that fails to load is now **skipped** (retried next run) rather than kept unchecked: a blocked cookie fails every remaining page at once. Rows get `source`/`search` = `data-jobs.ch`.
+
     **Language-check the `div.plain-content` block only, never the whole page.** The site chrome is German and on its own clears the ≥ 6 DE/FR word threshold — checking the full page dropped *every* job, English ones included. That block is the mobile rendering of the description; the desktop view iframes the same text, so it's the only copy in the server-rendered HTML.
   - **Basel international orgs** (`fetchBaselIntlOrgs`) sweeps three boards under one label, each only a handful of postings:
     - **BIS** — the career site is JS-rendered with no usable API, so we read the UNjobs mirror at `unjobs.org/offices/bis_182320855689338880`. Titles arrive as `Role, City, Country` and get split back apart. BIS also cycles Senior Software / Data / Data & Analytics Engineer roles through this board.
@@ -93,10 +95,11 @@ plans/             # LinkedIn post drafts
   - **startup.jobs** (`fetchStartupJobs`, added 2026-10-06) — global startup board, swept Switzerland-wide. **Every path is behind a Cloudflare managed challenge** (403 on plain HTTP for any UA, feeds and `/api` included), and unlike basel-jobs.ch's DataDome cookie, `cf_clearance` **does not replay** over Bun's fetch — measured 403 with the exact cookie + UA; it's bound to Chrome's TLS fingerprint. What works: Chrome `--headless=new` with a throwaway profile clears the challenge **only if the UA has no "HeadlessChrome" token** (the version is read from `Chrome --version` so it doesn't drift). So this source launches a headless browser on every run — unlike basel-jobs.ch's minting, that works fine under launchd. `LINKME_NO_BROWSER=1` disables it; any failure logs `unavailable` and returns `[]`.
 
     Listings come from **Algolia** (app `4CQMTMMK73`, index `Post_production`), not the HTML: the page embeds a *secured* search key in `meta[name="current-algolia-api-key-search"]`, and algolia.net is not behind Cloudflare, so the queries run over plain fetch. The key's `validUntil` is **~3h after page load** — read fresh every run, never cached. The key rejects every country filter (`country_code`/`country_id` → 0 hits), so the sweep is a geo bounding box around Switzerland with `country_code === "CH"` checked locally (the box also catches DE/FR/AT/LI/IT), limited to the last 60 days. Algolia caps a query at **500 retrievable hits** (100 × 5 pages) — a box over 500 is split into quadrants recursively (all ~1400 CH-box postings vs 556 in the 60-day window, measured 2026-10-06). `tagFilters` on `_tags` (Developer/Engineer) was tried and dropped: it covered only 365 of them. Records carry no description: each title-filtered candidate's detail page is fetched *inside* the browser (`page.evaluate(fetch)`, same TLS) and its JSON-LD `JobPosting.description` goes through `isViableDescription` (`contentChecked: true`). Links already in `results/jobs.csv` are skipped before that fetch. It aggregates company ATS feeds (Wise, Stripe, Avaloq, Nebius…) that LinkedIn also carries, so its `extraSources` entry sets `dedupeByTitle` like more-jobs.ch. The user's original URL was `?q=Developer&w=hybrid` around Basel; the source deliberately keeps on-site/remote too and uses `SOFTWARE_TITLE_RE` instead of the `Developer` query, matching every other source.
+  - **arbeitnow.ch** (`fetchArbeitnow`, added 2026-11-03) — free public JSON API (`/api/job-board-api?page=n`, 100/page, no auth), fed straight from company ATS feeds. Rolling ~7-day window (~195 jobs), so 2 requests cover it. Full description inline → in-process DE/FR + `requiresGerman` check. `dedupeByTitle` (overlaps ~50% with LinkedIn). **pilotjob.ch was evaluated and rejected**: its listing only offers `src=arbeitnow|jobroom`, i.e. it re-serves this feed plus job-room.ch. englishjobsearch.ch/jobs/remote shares only 1 of arbeitnow's 19 remote postings.
 - **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV. Writes both `search` (the query/source that found the job) and `location` (the posting's own location, as parsed off its card by `fetch.ts`/`fetch-other.ts` — no extra page fetch) to the CSV. Migrates old CSVs on startup: adds `source` if missing, then `location` (backfilled `""` for rows written before the column existed).
   - Every LinkedIn search and every `fetch-other.ts` source hits a different host, so they run **concurrently** (bounded: 3 at a time within each group, both groups started together) instead of one after another — cut a full `bun fetch` from ~16-20 min to well under 2 min. `existingLinks.add()` happens synchronously right after the `.has()` check and before any `await`, which is what makes concurrent workers safe against double-claiming the same link. Concurrency surfaced a real gap: LinkedIn started returning occasional 429s under parallel load, which `fetchJobs` used to treat as end-of-results (silently truncating that search) — `fetchJobs` now retries a 429 up to 3× with backoff (1s/2s/4s) before giving up.
 - **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions.
-- **`daily.ts`** — the notification layer on top of `index.ts`. Snapshots the CSV's links, shells out to `bun fetch`, then diffs to find what's new. Survivors are filtered through `matchesProfile` and grouped by area (Basel / Remote / Zürich / Bern) so the message scans on a phone. `results/sent.txt` is written **only after a successful send**, so a Telegram failure retries next run instead of silently losing jobs. A non-zero exit from `bun fetch` sends a failure notice rather than failing quietly.
+- **`daily.ts`** — the notification layer on top of `index.ts`. Snapshots the CSV's links, shells out to `bun fetch`, then diffs to find what's new. Survivors are filtered through `matchesProfile` and grouped by area (Basel / Remote / Zürich / Bern / Luzern) so the message scans on a phone. `results/sent.txt` is written **only after a successful send**, so a Telegram failure retries next run instead of silently losing jobs. A non-zero exit from `bun fetch` sends a failure notice rather than failing quietly.
   - The area grouping matches against the CSV's `location` column (the posting's real location) first, falling back to `search` (the query/source label) only when `location` is empty — i.e. for rows written before the `location` column existed. This fixed two real mislabels: a LinkedIn "Bern" sweep can surface a job actually based in Lausanne (search label ≠ real location), and `fetch-other.ts` sources (job-room.ch, itjobs.ch, stellen-basel.ch, datacareer.ch) write the source name into `search` instead of a city, which used to dump their jobs into "⚪️ Other" regardless of where they actually were.
 - **`profile.ts`** — `matchesProfile(title, company, minScore = 4)`. Hard rejects run first (Microsoft stack, Java/C++/SAP, life-science, non-software functions, blocked companies Jobgether and ALLPS), then `SIGNALS` scores stack and seniority terms from `cv/polo.md`. This is a **second, stricter gate than the fetch filters**: `fetch.ts` decides what lands in the CSV, `profile.ts` decides what's worth a phone buzz. Widen `SOFTWARE_TITLE_RE` to catch more in the CSV; tune `SIGNALS`/`minScore` to change what gets sent.
 - **`telegram.ts`** — reads `.env` directly (no dotenv dep). Accepts `TELEGRAM_BOT` or `TELEGRAM_TOKEN` for the token. Splits at 3900 chars on line boundaries (Telegram hard-caps at 4096) and sleeps 400 ms between parts to stay under the rate limit. Messages are HTML — run titles/companies through `escapeHtml`.