The link column was matched by an http(s):// pattern, but job-room.ch and
jobs.ch links are stored without a protocol (e.g. "www.jobs.ch/de/..."),
so the pattern silently skipped every one of them and re-added the same
posting on every run. Read the link by CSV column position instead.
Jobs rejected for German/French content never reach jobs.csv, so daily.ts
had no way to see they existed — the digest silently under-reported how
much the run actually looked at.
index.ts now counts them across both the LinkedIn and extra-source paths
and writes results/.lang-dropped.json; daily.ts reads it back, keyed on
the digest date, and appends a count-only category. Count-only because
there is no Row to list — these were dropped before ever being parsed.
datacareer.ch had been returning 403 on every page for months. The cause
was our own User-Agent: its nginx inverts the usual bot rule and blocks
browser-like UAs while serving honest ones (no UA, curl, Googlebot).
Give it PLAIN_HEADERS instead of the shared spoofed-Chrome HEADERS.
Basel has 8 jobs on a single page with no pagination, so page 2 returning
0 items is correct rather than truncation.
basel-jobs.ch was documented as un-integrable because DataDome blocks
plain curl. Only half of that was true. Measured: plain fetch 403s on any
UA, headless Chrome over CDP 403s, Playwright's own launch 403s even
headful — but a real Chrome launched normally with --remote-debugging-port
and attached over CDP afterwards gets 200. Clearance then lives entirely
in a datadome cookie that replays fine over plain fetch with a matching
UA, so a browser is touched only to mint, never to scrape. The earlier
"cookie can't be extracted" note was true of the browser extension, not
of CDP.
Minting needs a visible window, so launchd runs can't do it; a stale
cookie makes the source log and yield nothing instead of failing the run.
Refresh by hand with `bun run mint-datadome`, or set LINKME_NO_BROWSER=1.
Three passes, all needed — they cover different sets. The informatik
category is a curated taxonomy, not the whole board, and the two search
modes query different indexes: websearch=1 finds role words and postings
filed outside IT (+147 over the category), while the bare search is the
only mode matching stack terms — react, frontend and backend return zero
under websearch=1. Language checks run against div.plain-content only;
against the whole page the German site chrome alone clears the DE/FR
threshold and drops every job, English ones included.
Also fix a pre-existing double-wrapped Promise in the extraSources type.
Every LinkedIn search and every fetch-other.ts source hits a different
host, so there was no real reason for the old for-loop to wait on each
one in turn - that's what made a full run take 30-40+ minutes. Runs
them concurrently now (bounded: 3 at a time within each group, both
groups started together), which brought a real run down to under 90
seconds.
existingLinks.add() now happens synchronously right after the .has()
check and before any await, which is what makes concurrent workers
safe against double-claiming the same link.
Concurrency surfaced a real regression: LinkedIn started returning
occasional 429s under parallel load, and fetchJobs treated that the
same as end-of-results, silently truncating the rest of that search.
Added retry-with-backoff (3 attempts, 1s/2s/4s) so a rate limit no
longer costs real listings.
fetch.ts already parses each job card's own location, but index.ts
discarded it and stored only the search-area label in the CSV, so
daily.ts's telegram grouping used the query that found a job instead
of where it actually is (e.g. a Bern search surfacing a Lausanne job).
Now the location is written to a new CSV column and used for grouping,
with a one-time migration for existing rows.
Broaden the search beyond engineer/developer titles and stop three
classes of bad result from ever reaching the CSV.
config.ts
- PLATFORM_KEYWORDS (architect, SRE, infra, security, MLOps),
RESEARCH_KEYWORDS (research/scientific software, HPC, data platform)
and FIELD_KEYWORDS (forward-deployed, customer/sales, founding).
Separate OR-strings, not folded into KEYWORDS: LinkedIn ranks a
focused query far better than one 50-term disjunction.
- sweep() generates one search per area per family. 7 searches -> 17.
fetch.ts
- BIO_DOMAIN_RE drops life-science roles (bioinformatics, biostats,
GMP/MSAT, genomics, pathology, drug discovery). Filters on the role,
never the employer: software jobs at Roche/Novartis/sciCORE stay.
- requiresGerman() drops English postings demanding German above B1.
Two rules, both from real false results: the nearest marker wins
("Infrastructure skills are a plus" sat 95 chars before "Good
knowledge of German"), and an intervening language name breaks the
binding ("Fluent in English; German and/or French advantageous").
- GERMAN_TITLE_RE catches "(German-speaking)" without a page fetch.
- isViableContent/isViableUrl run the DE/FR and German-requirement
checks on a single fetch; isEnglishContent/isEnglishUrl kept for
recheck.ts and temp scripts.
- SOFTWARE_TITLE_RE widened so the new families survive it.
index.ts
- both content-check paths use the combined check and report which
filter dropped a job.
- move temp scripts to scripts/, cv files to cv/, jobs.csv to results/
- update index.ts and recheck.ts CSV path to results/jobs.csv
- fix imports in scripts/ to use ../config.ts and ../fetch.ts
- update CLAUDE.md and README.md to reflect new structure
- add plans/ with LinkedIn post drafts
- update .gitignore: ignore results/, jobs/, cv binaries, .DS_Store
- isEnglishContent() fetches each job page and drops descriptions with ≥6 German or French indicator words; called for every new job in index.ts before writing to CSV
- New `bun recheck` command (recheck.ts) runs searches in raw mode, lists filtered-out jobs, and prints word-frequency suggestions for SOFTWARE_TITLE_RE
- Expanded KEYWORDS and SOFTWARE_TITLE_RE to cover ML/AI, system/applications engineer, director, informatics, and more title variants
- Added `raw` flag to fetchJobs() to bypass the software-role filter
- Add CLAUDE.md with project workflow docs; add CV files and city temp scripts
- Paginated HTML scraper (up to 6 pages, 400ms delay between requests)
- Card-boundary-safe parsing to prevent field bleed across cards
- Language filter: drops German (m/w/d, umlauts) and French (H/F, accents) titles
- Software-role filter: requires title to contain a relevant tech term
- Global deduplication by job ID across runs
- 30-day recency filter via LinkedIn f_TPR parameter
- CSV output: date_found, posted_at, search, title, company, link
rodado