feat: add startup.jobs as a switzerland-wide job source

This commit is contained in:
2026-10-06 15:17:17 +02:00
parent ab659008e6
commit e70ae1e932
4 changed files with 245 additions and 8 deletions
+5 -2
View File
@@ -21,7 +21,7 @@ Temp scripts: `bun run scripts/<name>-temp.ts` — never edit `config.ts` for on
```
config.ts # search keywords and areas
fetch.ts # scraper, HTML parser, all filters
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch)
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch, startup.jobs)
index.ts # orchestrator: runs searches, writes CSV
recheck.ts # raw-mode fetch, lists filtered-out jobs
daily.ts # digest: fetch → match against profile.ts → send to Telegram
@@ -43,7 +43,7 @@ plans/ # LinkedIn post drafts
- Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform) and `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform and Field add a Switzerland-wide remote pass.
- `RESEARCH_KEYWORDS` deliberately contains no bioinformatics/computational-biology terms — see `BIO_DOMAIN_RE`.
- **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`.
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch). Imported by `index.ts`.
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch, startup.jobs). Imported by `index.ts`.
- **novartis.com** (`fetchNovartisBasel`, added 2026-09-21) — Rafael has an internal contact at Novartis Basel who can refer him, so this source exists purely to catch a genuine software/engineering opening there, not to widen the CSV generally. Its career-search page (`novartis.com/careers/career-search?country[]=LOC_CH`) is a server-rendered Drupal view — no headless browser needed — but only at the **locale-less English path** (`/careers/career-search`); the `ch-de` German path's job-detail links 302 straight back to the search page for both locales when hit without a session, so detail pages only resolve through the English path. Pagination is 0-indexed (`page=0` implicit, `page=1` is the second page of 10) and the page's own "Showing N results" line gives the exact total, so the page-loop bound is computed rather than probed — the listing has also been observed to **repeat rows across pages within one sweep** (measured 2026-09-21: several titles appeared twice across 5 pages), so results are deduped by `req-<id>` before filtering, not trusted from the pager.
Novartis Basel's titling is almost entirely non-technical "Director, `<business function>`" roles (Portfolio Analytics, Therapeutic Area Strategy, Policy Communications, ...) plus device/hardware "Engineering" roles (autoinjector/drug-delivery hardware, not software) — the generic `SOFTWARE_TITLE_RE` used by every other source in this file matched **13 of 47** CH listings on title alone in an initial run, and every single one was a false positive (no engineering-team role among them). `NOVARTIS_TITLE_RE` replaces it here: it requires an explicit software/IT engineering job-family term (`software engineer`, `platform engineer`, `data engineer`, `devops`, `sre`, `solutions architect`, ...), not just a bare seniority or business-function word. `BIO_DOMAIN_RE` and job-room.ch's `PHARMA_ROLE_RE` still run on top to catch domain-science titles the stricter regex wouldn't otherwise need to worry about. Narrowed to Basel/Muttenz specifically (`NOVARTIS_BASEL_RE`) — the CH listing also includes Stein (Aargau), a different commute the contact couldn't refer into. As of 2026-09-21 this source keeps **zero** listings — expected; it's watching for the rare opening, not sweeping volume. Each detail page carries the full description inline (`job_details_content_center`…`job_details_content_bottom`), so the DE/FR + `requiresGerman` check runs in-process per candidate rather than costing `index.ts` a page fetch.
@@ -90,6 +90,9 @@ plans/ # LinkedIn post drafts
Swisscom mostly hires in German — of 47 postings, only ~16 clear the English/software title filters, and nearly all of those turned out to be written in German too; only genuinely-English, German-optional roles survive. Adding this source surfaced a real bug in `requiresGerman()`: the stock line *"Fluency in English is required; proficiency in German is considered an advantage"* had the noun "proficiency" (a required-marker word) sitting right next to "German", closer than the real optional-marker "advantage" — nearest() picked the wrong one and wrongly flagged the posting as requiring German. Fixed by adding a local-predicate override (`GERMAN_LOCAL_PREDICATE_RE`): when a copula ("is"/"are", optionally "considered"/"deemed") directly follows the German mention, that immediate predicate decides the verdict outright instead of competing on raw character distance with markers elsewhere in the window — "German is considered an advantage" resolves as optional regardless of what else is nearby. It only fires when a copula immediately follows the mention, so *"Work proficiency in German, English, and French"* (no copula after "German", a genuine hard requirement — Senior Operation Architect (FR/GER/ENG)) still falls through to the general nearest-marker logic and is correctly caught as required. Re-check both cases (and the pre-existing Helvetia/albedis/Karakun/J. Safra Sarasin/BBF/spec-sheet cases in the paragraph above) before touching `requiresGerman` again.
- **more-jobs.ch** (`fetchMoreJobs`, added 2026-10-02) — server-rendered PHP aggregator. `/recherche?q=<term>&page=n` is fast (~0.5 s) with ~20 cards/page, but `/emplois` and out-of-range pages are slow (20 s+) and **clamp to the last page instead of going empty**, so a term's sweep ends on "no new ids", not an empty list. Swept with `MOREJOBS_TERMS` (≤5 pages each), then each title-filtered candidate gets one detail fetch (4 in parallel, 20 s timeout) for the real `Publication date` (dd.mm.yyyy) and the in-process DE/FR + `requiresGerman` check (`contentChecked: true`). The board **reposts one opening under several IDs** (measured: Accelleron, Swissquote, Bayer ×3), and it also mirrors jobs that LinkedIn/jobs.ch carry, so its `extraSources` entry sets `dedupeByTitle`: `index.ts` skips any job whose `langKey(title, company)` is already in the CSV (or claimed earlier in the same run). Other sources are deliberately not deduped this way. The board also lists **foreign postings** ("Eastern Europe, Poland", "95050 Santa Clara", "London") and its JSON-LD stamps `addressCountry: "CH"` on every one, so the card's location text is the only signal: `MOREJOBS_FOREIGN_RE` drops locations naming another country/foreign city or carrying a 5-digit postcode before the detail fetch (19 such rows had leaked into the CSV by 2026-10-03, several reaching the Telegram digest).
- **startup.jobs** (`fetchStartupJobs`, added 2026-10-06) — global startup board, swept Switzerland-wide. **Every path is behind a Cloudflare managed challenge** (403 on plain HTTP for any UA, feeds and `/api` included), and unlike basel-jobs.ch's DataDome cookie, `cf_clearance` **does not replay** over Bun's fetch — measured 403 with the exact cookie + UA; it's bound to Chrome's TLS fingerprint. What works: Chrome `--headless=new` with a throwaway profile clears the challenge **only if the UA has no "HeadlessChrome" token** (the version is read from `Chrome --version` so it doesn't drift). So this source launches a headless browser on every run — unlike basel-jobs.ch's minting, that works fine under launchd. `LINKME_NO_BROWSER=1` disables it; any failure logs `unavailable` and returns `[]`.
Listings come from **Algolia** (app `4CQMTMMK73`, index `Post_production`), not the HTML: the page embeds a *secured* search key in `meta[name="current-algolia-api-key-search"]`, and algolia.net is not behind Cloudflare, so the queries run over plain fetch. The key's `validUntil` is **~3h after page load** — read fresh every run, never cached. The key rejects every country filter (`country_code`/`country_id` → 0 hits), so the sweep is a geo bounding box around Switzerland with `country_code === "CH"` checked locally (the box also catches DE/FR/AT/LI/IT), limited to the last 60 days. Algolia caps a query at **500 retrievable hits** (100 × 5 pages) — a box over 500 is split into quadrants recursively (all ~1400 CH-box postings vs 556 in the 60-day window, measured 2026-10-06). `tagFilters` on `_tags` (Developer/Engineer) was tried and dropped: it covered only 365 of them. Records carry no description: each title-filtered candidate's detail page is fetched *inside* the browser (`page.evaluate(fetch)`, same TLS) and its JSON-LD `JobPosting.description` goes through `isViableDescription` (`contentChecked: true`). Links already in `results/jobs.csv` are skipped before that fetch. It aggregates company ATS feeds (Wise, Stripe, Avaloq, Nebius…) that LinkedIn also carries, so its `extraSources` entry sets `dedupeByTitle` like more-jobs.ch. The user's original URL was `?q=Developer&w=hybrid` around Basel; the source deliberately keeps on-site/remote too and uses `SOFTWARE_TITLE_RE` instead of the `Developer` query, matching every other source.
- **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV. Writes both `search` (the query/source that found the job) and `location` (the posting's own location, as parsed off its card by `fetch.ts`/`fetch-other.ts` — no extra page fetch) to the CSV. Migrates old CSVs on startup: adds `source` if missing, then `location` (backfilled `""` for rows written before the column existed).
- Every LinkedIn search and every `fetch-other.ts` source hits a different host, so they run **concurrently** (bounded: 3 at a time within each group, both groups started together) instead of one after another — cut a full `bun fetch` from ~16-20 min to well under 2 min. `existingLinks.add()` happens synchronously right after the `.has()` check and before any `await`, which is what makes concurrent workers safe against double-claiming the same link. Concurrency surfaced a real gap: LinkedIn started returning occasional 429s under parallel load, which `fetchJobs` used to treat as end-of-results (silently truncating that search) — `fetchJobs` now retries a 429 up to 3× with backoff (1s/2s/4s) before giving up.
- **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions.