datacareer.ch had been returning 403 on every page for months. The cause
was our own User-Agent: its nginx inverts the usual bot rule and blocks
browser-like UAs while serving honest ones (no UA, curl, Googlebot).
Give it PLAIN_HEADERS instead of the shared spoofed-Chrome HEADERS.
Basel has 8 jobs on a single page with no pagination, so page 2 returning
0 items is correct rather than truncation.
basel-jobs.ch was documented as un-integrable because DataDome blocks
plain curl. Only half of that was true. Measured: plain fetch 403s on any
UA, headless Chrome over CDP 403s, Playwright's own launch 403s even
headful — but a real Chrome launched normally with --remote-debugging-port
and attached over CDP afterwards gets 200. Clearance then lives entirely
in a datadome cookie that replays fine over plain fetch with a matching
UA, so a browser is touched only to mint, never to scrape. The earlier
"cookie can't be extracted" note was true of the browser extension, not
of CDP.
Minting needs a visible window, so launchd runs can't do it; a stale
cookie makes the source log and yield nothing instead of failing the run.
Refresh by hand with `bun run mint-datadome`, or set LINKME_NO_BROWSER=1.
Three passes, all needed — they cover different sets. The informatik
category is a curated taxonomy, not the whole board, and the two search
modes query different indexes: websearch=1 finds role words and postings
filed outside IT (+147 over the category), while the bare search is the
only mode matching stack terms — react, frontend and backend return zero
under websearch=1. Language checks run against div.plain-content only;
against the whole page the German site chrome alone clears the DE/FR
threshold and drops every job, English ones included.
Also fix a pre-existing double-wrapped Promise in the extraSources type.
Every LinkedIn search and every fetch-other.ts source hits a different
host, so there was no real reason for the old for-loop to wait on each
one in turn - that's what made a full run take 30-40+ minutes. Runs
them concurrently now (bounded: 3 at a time within each group, both
groups started together), which brought a real run down to under 90
seconds.
existingLinks.add() now happens synchronously right after the .has()
check and before any await, which is what makes concurrent workers
safe against double-claiming the same link.
Concurrency surfaced a real regression: LinkedIn started returning
occasional 429s under parallel load, and fetchJobs treated that the
same as end-of-results, silently truncating the rest of that search.
Added retry-with-backoff (3 attempts, 1s/2s/4s) so a rate limit no
longer costs real listings.
jobbasel.ch's search route (/job/suche-{slug}) is server-rendered with
a window.__PRELOADED_STATE__ JSON blob, no headless browser needed.
englishjobsearch.ch works the same way via its ?format=markdown param.
Both swept with the same fixed keyword list job-room.ch already uses.
basel-jobs.ch was evaluated too but left out: it's DataDome-protected,
which blocks plain curl outright, and this project's fetchers are
intentionally browser-free.
While wiring these in, found that isEnglishDescription (now
isViableDescription) only ever ran the DE/FR word-count check against
inline descriptions, never requiresGerman() - so an English-written
posting demanding fluent German (e.g. PRODYNA's "you can communicate
fluently in both German and English") sailed straight through. That
gap existed in itjobs.ch and job-room.ch too, not just the two new
sources, so the fix applies to all four.
fetch.ts already parses each job card's own location, but index.ts
discarded it and stored only the search-area label in the CSV, so
daily.ts's telegram grouping used the query that found a job instead
of where it actually is (e.g. a Bern search surfacing a Lausanne job).
Now the location is written to a new CSV column and used for grouping,
with a one-time migration for existing rows.
Rewrite the README around the current pipeline: the twice-daily Telegram
digest, the sciCORE watcher, Telegram setup, and the two-stage filter
pipeline.
CLAUDE.md gains the bun daily commands, the daily/profile/telegram
architecture notes, and a scheduled-automations section covering both
launchd agents. Records that fetch.ts gates what reaches the CSV while
profile.ts gates what reaches the phone.
Broaden the search beyond engineer/developer titles and stop three
classes of bad result from ever reaching the CSV.
config.ts
- PLATFORM_KEYWORDS (architect, SRE, infra, security, MLOps),
RESEARCH_KEYWORDS (research/scientific software, HPC, data platform)
and FIELD_KEYWORDS (forward-deployed, customer/sales, founding).
Separate OR-strings, not folded into KEYWORDS: LinkedIn ranks a
focused query far better than one 50-term disjunction.
- sweep() generates one search per area per family. 7 searches -> 17.
fetch.ts
- BIO_DOMAIN_RE drops life-science roles (bioinformatics, biostats,
GMP/MSAT, genomics, pathology, drug discovery). Filters on the role,
never the employer: software jobs at Roche/Novartis/sciCORE stay.
- requiresGerman() drops English postings demanding German above B1.
Two rules, both from real false results: the nearest marker wins
("Infrastructure skills are a plus" sat 95 chars before "Good
knowledge of German"), and an intervening language name breaks the
binding ("Fluent in English; German and/or French advantageous").
- GERMAN_TITLE_RE catches "(German-speaking)" without a page fetch.
- isViableContent/isViableUrl run the DE/FR and German-requirement
checks on a single fetch; isEnglishContent/isEnglishUrl kept for
recheck.ts and temp scripts.
- SOFTWARE_TITLE_RE widened so the new families survive it.
index.ts
- both content-check paths use the combined check and report which
filter dropped a job.
- move temp scripts to scripts/, cv files to cv/, jobs.csv to results/
- update index.ts and recheck.ts CSV path to results/jobs.csv
- fix imports in scripts/ to use ../config.ts and ../fetch.ts
- update CLAUDE.md and README.md to reflect new structure
- add plans/ with LinkedIn post drafts
- update .gitignore: ignore results/, jobs/, cv binaries, .DS_Store
- isEnglishContent() fetches each job page and drops descriptions with ≥6 German or French indicator words; called for every new job in index.ts before writing to CSV
- New `bun recheck` command (recheck.ts) runs searches in raw mode, lists filtered-out jobs, and prints word-frequency suggestions for SOFTWARE_TITLE_RE
- Expanded KEYWORDS and SOFTWARE_TITLE_RE to cover ML/AI, system/applications engineer, director, informatics, and more title variants
- Added `raw` flag to fetchJobs() to bypass the software-role filter
- Add CLAUDE.md with project workflow docs; add CV files and city temp scripts
rodado