diff --git a/CLAUDE.md b/CLAUDE.md index aa5159f..ec2d6ae 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -45,6 +45,7 @@ plans/ # LinkedIn post drafts - **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`. - **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch). Imported by `index.ts`. - **datacareer.ch** inverts the usual bot rule: its nginx **403s browser-like User-Agents** and serves honest ones (no UA, `curl/8.0`, Googlebot) fine. The spoofed Chrome UA in the shared `HEADERS` was the entire cause of the long-standing 403 — it uses `PLAIN_HEADERS` instead. Small board: Basel returns **8 jobs on one page with no pagination**, so `?page=2` legitimately yields 0 items. + - **job-room.ch** carries its language requirement as a **structured field** (`languageSkills[]`, e.g. `{languageIsoCode: "de", spokenLevel: "PROFICIENT"}`), entirely separate from the free-text `jobDescriptions[].description` — the UI renders it as "Deutsch (Fliessend)". `requiresGerman()`/`isViableDescription()` only ever scan the prose, so this field slipped through unchecked until 2026-09-04: measured against 1843 ads, 69 required PROFICIENT-level German and the prose check caught only 3 of them incidentally. `jobRoomRequiresGerman()` now checks the structured field directly (`spokenLevel`/`writtenLevel === "PROFICIENT"`) before the description-based check runs. Levels seen: `NONE`, `BASIC`, `INTERMEDIATE`, `PROFICIENT` — only `PROFICIENT` (fluent/C1+) is dropped, matching the B1 cutoff used everywhere else. - **jobbasel.ch** looks like a JS-only React/Vite SPA (no jobs in the homepage's raw HTML), but its search route is server-rendered: `GET /job/suche-{slug}` returns full HTML with a `window.__PRELOADED_STATE__` blob containing the matched vacancies as JSON — no headless browser needed. Pagination is a URL suffix (`-seite-{n}`); a bare `?page=` query param is silently ignored. Swept with the same fixed keyword list as job-room.ch (no keyword-union endpoint exists) and deduped by vacancy id. Each item carries its full German description, so the DE/FR check runs in-process. - **englishjobsearch.ch** also looks JS-only at first glance, but any page accepts `?format=markdown` and returns clean server-rendered markdown with full job cards — title, company, city, date, description. Hit the canton pages directly (`/in/canton-basel-stadt`, `/in/canton-basel-landschaft`) rather than the `/in/basel/` landing page, which only lists canton counts. Every job link is a `/clickout/` tracking redirect rather than the real posting URL — there's no way to recover the underlying employer link without following the redirect, so the clickout link is stored as-is. - **basel-jobs.ch** is behind DataDome and **is now integrated** (`fetchBaselJobs`), contrary to the earlier note that it couldn't be. What the block actually permits, tested: diff --git a/fetch-other.ts b/fetch-other.ts index 99b035a..d040807 100644 --- a/fetch-other.ts +++ b/fetch-other.ts @@ -236,9 +236,28 @@ interface JobRoomAd { jobDescriptions?: Array<{ title: string; description?: string }>; company?: { name?: string; city?: string }; location?: { city?: string; cantonCode?: string }; + languageSkills?: JobRoomLanguageSkill[]; }; } +interface JobRoomLanguageSkill { + languageIsoCode?: string; + spokenLevel?: string; + writtenLevel?: string; +} + +// job-room's language requirement is a structured field, not prose — it never +// touches jobDescriptions[].description, so isViableDescription()/requiresGerman() +// can't see it. The UI renders it as e.g. "Deutsch (Fliessend)"; PROFICIENT is +// that "Fliessend" (fluent/C1+) tier, well above Rafael's German B1. BASIC and +// INTERMEDIATE stay in scope. +function jobRoomRequiresGerman(skills?: JobRoomLanguageSkill[]): boolean { + if (!skills) return false; + return skills.some( + (s) => s.languageIsoCode === "de" && (s.spokenLevel === "PROFICIENT" || s.writtenLevel === "PROFICIENT") + ); +} + // jobs.ch links arrive with a tracking query string that would break dedup // across runs if it ever changes; the path alone identifies the posting. function stripTracking(url: string): string { @@ -299,6 +318,10 @@ export async function fetchJobRoom(): Promise { if (MS_STACK_RE.test(title)) continue; if (PHARMA_ROLE_RE.test(title)) continue; + // Structured language requirement — separate from the free-text description + // entirely, so it must be checked even when the description reads clean. + if (jobRoomRequiresGerman(content?.languageSkills)) { langDropped++; continue; } + // A handful of ads ship without a description. isViableDescription passes // those through by design, so leave contentChecked unset and let index.ts // fall back to fetching the page.