feat: add role-family sweeps and domain/language filters

Broaden the search beyond engineer/developer titles and stop three
classes of bad result from ever reaching the CSV.

config.ts
- PLATFORM_KEYWORDS (architect, SRE, infra, security, MLOps),
  RESEARCH_KEYWORDS (research/scientific software, HPC, data platform)
  and FIELD_KEYWORDS (forward-deployed, customer/sales, founding).
  Separate OR-strings, not folded into KEYWORDS: LinkedIn ranks a
  focused query far better than one 50-term disjunction.
- sweep() generates one search per area per family. 7 searches -> 17.

fetch.ts
- BIO_DOMAIN_RE drops life-science roles (bioinformatics, biostats,
  GMP/MSAT, genomics, pathology, drug discovery). Filters on the role,
  never the employer: software jobs at Roche/Novartis/sciCORE stay.
- requiresGerman() drops English postings demanding German above B1.
  Two rules, both from real false results: the nearest marker wins
  ("Infrastructure skills are a plus" sat 95 chars before "Good
  knowledge of German"), and an intervening language name breaks the
  binding ("Fluent in English; German and/or French advantageous").
- GERMAN_TITLE_RE catches "(German-speaking)" without a page fetch.
- isViableContent/isViableUrl run the DE/FR and German-requirement
  checks on a single fetch; isEnglishContent/isEnglishUrl kept for
  recheck.ts and temp scripts.
- SOFTWARE_TITLE_RE widened so the new families survive it.

index.ts
- both content-check paths use the combined check and report which
  filter dropped a job.
This commit is contained in:
2026-08-27 09:02:50 +02:00
parent fb67c38edd
commit 3aa4365fb9
4 changed files with 327 additions and 24 deletions
+25 -6
View File
@@ -16,7 +16,7 @@ Temp scripts: `bun run scripts/<name>-temp.ts` — never edit `config.ts` for on
```
config.ts # search keywords and areas
fetch.ts # scraper, HTML parser, all filters
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch)
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs)
index.ts # orchestrator: runs searches, writes CSV
recheck.ts # raw-mode fetch, lists filtered-out jobs
scripts/ # one-off search scripts; import from ../config.ts and ../fetch.ts
@@ -30,8 +30,17 @@ plans/ # LinkedIn post drafts
## Architecture
- **`config.ts`** — `KEYWORDS` string (joined with ` OR `) and `searches: SearchConfig[]`. Edit here to add/remove search areas or keywords.
- Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform) and `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform and Field add a Switzerland-wide remote pass.
- `RESEARCH_KEYWORDS` deliberately contains no bioinformatics/computational-biology terms — see `BIO_DOMAIN_RE`.
- **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`.
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch). Imported by `index.ts`.
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs). Imported by `index.ts`.
- **datacareer.ch** currently returns **403** on every page — the source is dead and yields nothing. Needs new headers or a different entry point.
- **Basel international orgs** (`fetchBaselIntlOrgs`) sweeps three boards under one label, each only a handful of postings:
- **BIS** — the career site is JS-rendered with no usable API, so we read the UNjobs mirror at `unjobs.org/offices/bis_182320855689338880`. Titles arrive as `Role, City, Country` and get split back apart. BIS also cycles Senior Software / Data / Data & Analytics Engineer roles through this board.
- **Swiss TPH** — Umantis at `jobs.swisstph.ch/Jobs/All`. Site chrome is German even on English postings, so `isEnglishUrl` would reject everything; `contentChecked: true` and the title filters decide instead.
- **Basel Institute on Governance** — `baselgovernance.org/about/careers`. Each vacancy is a linked PDF, so there's no HTML description to language-check. The `rounded-3xl` card class is what separates vacancy tiles from the annual-report PDF link.
- These boards use UN-flavoured titles ("Information Management Officer", "ICT Specialist") that `SOFTWARE_TITLE_RE` alone drops, so `INTL_TECH_RE` widens the net **for these sources only**. Boards this small can afford the noise; missing a rare opening can't.
- **itjobs.ch** is swept Switzerland-wide (no `location` param) — the board is small, ~210 jobs over 7 pages, and locking it to Basel returned only ~9. It 404s one page past the end instead of serving an empty list. Each item embeds its full `description`, so the DE/FR check runs in-process via `isEnglishDescription` rather than costing a page fetch per job. Use `job_location.name` for location — it's the English form; the bare `location` field is German.
- **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV.
- **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions.
@@ -41,8 +50,14 @@ plans/ # LinkedIn post drafts
2. **`NON_ENGLISH_RE`** (title) — drops German gender markers, umlauts, `/in` suffix, French accents/markers. Runs inside `fetchJobs`.
3. **`JUNIOR_TITLE_RE`** (title) — drops junior/intern/trainee/werkstudent. Runs inside `fetchJobs`.
4. **`JAVA_TITLE_RE`** (title) — drops Java-titled roles. Runs inside `fetchJobs`.
5. **`SOFTWARE_TITLE_RE`** (title) — requires a relevant tech/role term. Skipped when `raw: true`. Runs inside `fetchJobs`.
6. **`isEnglishContent`** (description page) — fetches the job page, counts DE/FR indicator words, drops if ≥ 6 matches. Runs in `index.ts` only (not `recheck.ts`). Adds 350 ms delay per job.
5. **`BIO_DOMAIN_RE`** (title) — drops roles whose *domain* is biology/chemistry/the lab bench (bioinformatics, biostatistics, GMP/MSAT, genomics, pathology, drug discovery, bench Scientist titles). Filters on the **role, not the employer** — software/platform/data jobs at Roche, Novartis, Swiss TPH and sciCORE stay in scope. Runs inside `fetchJobs`.
6. **`SOFTWARE_TITLE_RE`** (title) — requires a relevant tech/role term. Skipped when `raw: true`. Runs inside `fetchJobs`.
6.5. **`GERMAN_TITLE_RE`** (title) — drops "(German-speaking)" / "deutschsprachig" titles without a description fetch. Runs inside `fetchJobs`.
7. **`isViableContent`** / **`isViableUrl`** (description page) — one fetch, two checks: (a) counts DE/FR indicator words, drops if ≥ 6; (b) `requiresGerman()` drops English postings that demand German above B1. Runs in `index.ts` only (not `recheck.ts`). Adds 350 ms delay per job. `isEnglishContent` / `isEnglishUrl` remain exported for `recheck.ts` and temp scripts.
**`requiresGerman(text)`** is deliberately more than a keyword match — Rafael is German B1 (TELC), so B1/"basic"/"a plus" is fine and B2+ is not. For each mention of German it takes the *nearest* qualifying marker, and a marker is ignored when another language name sits between it and the mention. Both rules come from real false results:
- *"Infrastructure skills are **a plus** … Good knowledge of **German**"* (Helvetia) — presence-based matching let an unrelated "a plus" 95 chars away cancel a real requirement.
- *"Fluent in **English**; **German** and/or French advantageous"* (albedis) — "Fluent" is nearer to "German" than "advantageous" is, but qualifies English.
Deduplication: job IDs are extracted from `results/jobs.csv` on startup; any ID already seen is skipped before content-checking.
@@ -110,12 +125,16 @@ Read `cv/polo.md` for the full profile. Key signals:
- **Core stack**: Python · TypeScript · Ruby on Rails · React · Node.js
- **Domain strengths**: platform engineering · data pipelines · AI/LLM integrations · DevOps (Docker, Kubernetes, CI/CD, AWS)
- **Background edges**: fintech/CBDC (Moedax), civic tech (Tactical Tech / GFOSS), independent consulting (ExtraPolo, 40+ projects)
- **Languages**: English (full) · German B1 TELC certified · Spanish (full) — flag roles requiring C1+ German
- **Languages**: English (full) · German B1 TELC certified · Spanish (full) — **drop** roles requiring German above B1; `requiresGerman()` does this automatically at fetch time
- **Hard no: biological domains** — drop bioinformatics, computational biology, biostatistics, lab/bench Scientist, GMP/MSAT, clinical/assay and drug-discovery-science roles; he has no life-science background. Software roles *at* pharma companies are still strong targets
- **Hard no: Microsoft stack** — drop .NET/C#, Azure-first, Windows/SCCM/Intune, SharePoint, Dynamics, Power Platform roles entirely; don't present them even as weak fits
- **Location**: based in Basel, open to Basel / Bern / Zürich; EU citizen, no permit needed
## Output format
Present jobs as a markdown table: `#`, `Title`, `Company`, `Why it fits`, `Link`.
**Always** present jobs as a markdown table — never as a bullet list or prose: `#`, `Title`, `Company`, `Canton`, `Why it fits`, `Link`.
`Canton` is mandatory on every table, including single-city searches and the list of dropped/rejected jobs (there `Why dropped` replaces `Why it fits`). Derive it from the actual location on the posting, not from the search label — recruiters and multi-site companies often post under a different area than the job sits in. Use the two-letter code: BS, BL, ZH, BE, ZG, LU, VD, GE, TI, AG, SG, TG. For remote roles write `Remote` (add scope if known, e.g. `Remote (EU)`).
Links use just the job ID as anchor text: `[4192954199](https://www.linkedin.com/jobs/view/4192954199/)`