An audit of 89 basel-jobs.ch postings against their live descriptions
found 24 that mention German and 0 of them flagged. Three separate bugs,
each verified against real postings:
Coordination broke the intervening-language rule. In "Excellent
communication skills in both English and German" (Karakun) and "Fluency
in English and German" (J. Safra Sarasin), "English" sits between the
marker and the mention, so the marker was handed to English and a hard
requirement waved through. A marker now still binds when the other
language is coordinated with German (and/und/sowie/&/as well as), but
not through "and/or" or a bare "or" — "Very good English and/or German"
is satisfiable by English alone and must stay optional.
German-language markers never matched at all: `sehr\s+gute?` cannot match
"Sehr gutes Deutsch" because the trailing \b fails on the "s". Widened to
`sehr\s+gut\w*` plus fliessend\w*, Deutschkenntnisse and erforderlich.
This is why German-written postings passed the filter untouched.
Spec-sheet language lines ("Languages: German & English") have no
adjective to bind to, so the Languages:/Sprachen: label now counts as a
required marker. It still goes through the nearest-marker comparison, so
"Language: English and German (bonus)" correctly stays optional.
28 regression cases pass, covering the 15 "a plus" phrasings that must
not be dropped. On basel-jobs.ch this moves 88 viable to 85, removing
exactly the three hard requirements and keeping every optional one.
The filter is shared by every source, so rows already in jobs.csv were
written under the broken logic and are not retroactively corrected.
datacareer.ch had been returning 403 on every page for months. The cause
was our own User-Agent: its nginx inverts the usual bot rule and blocks
browser-like UAs while serving honest ones (no UA, curl, Googlebot).
Give it PLAIN_HEADERS instead of the shared spoofed-Chrome HEADERS.
Basel has 8 jobs on a single page with no pagination, so page 2 returning
0 items is correct rather than truncation.
basel-jobs.ch was documented as un-integrable because DataDome blocks
plain curl. Only half of that was true. Measured: plain fetch 403s on any
UA, headless Chrome over CDP 403s, Playwright's own launch 403s even
headful — but a real Chrome launched normally with --remote-debugging-port
and attached over CDP afterwards gets 200. Clearance then lives entirely
in a datadome cookie that replays fine over plain fetch with a matching
UA, so a browser is touched only to mint, never to scrape. The earlier
"cookie can't be extracted" note was true of the browser extension, not
of CDP.
Minting needs a visible window, so launchd runs can't do it; a stale
cookie makes the source log and yield nothing instead of failing the run.
Refresh by hand with `bun run mint-datadome`, or set LINKME_NO_BROWSER=1.
Three passes, all needed — they cover different sets. The informatik
category is a curated taxonomy, not the whole board, and the two search
modes query different indexes: websearch=1 finds role words and postings
filed outside IT (+147 over the category), while the bare search is the
only mode matching stack terms — react, frontend and backend return zero
under websearch=1. Language checks run against div.plain-content only;
against the whole page the German site chrome alone clears the DE/FR
threshold and drops every job, English ones included.
Also fix a pre-existing double-wrapped Promise in the extraSources type.
Every LinkedIn search and every fetch-other.ts source hits a different
host, so there was no real reason for the old for-loop to wait on each
one in turn - that's what made a full run take 30-40+ minutes. Runs
them concurrently now (bounded: 3 at a time within each group, both
groups started together), which brought a real run down to under 90
seconds.
existingLinks.add() now happens synchronously right after the .has()
check and before any await, which is what makes concurrent workers
safe against double-claiming the same link.
Concurrency surfaced a real regression: LinkedIn started returning
occasional 429s under parallel load, and fetchJobs treated that the
same as end-of-results, silently truncating the rest of that search.
Added retry-with-backoff (3 attempts, 1s/2s/4s) so a rate limit no
longer costs real listings.
jobbasel.ch's search route (/job/suche-{slug}) is server-rendered with
a window.__PRELOADED_STATE__ JSON blob, no headless browser needed.
englishjobsearch.ch works the same way via its ?format=markdown param.
Both swept with the same fixed keyword list job-room.ch already uses.
basel-jobs.ch was evaluated too but left out: it's DataDome-protected,
which blocks plain curl outright, and this project's fetchers are
intentionally browser-free.
While wiring these in, found that isEnglishDescription (now
isViableDescription) only ever ran the DE/FR word-count check against
inline descriptions, never requiresGerman() - so an English-written
posting demanding fluent German (e.g. PRODYNA's "you can communicate
fluently in both German and English") sailed straight through. That
gap existed in itjobs.ch and job-room.ch too, not just the two new
sources, so the fix applies to all four.
fetch.ts already parses each job card's own location, but index.ts
discarded it and stored only the search-area label in the CSV, so
daily.ts's telegram grouping used the query that found a job instead
of where it actually is (e.g. a Bern search surfacing a Lausanne job).
Now the location is written to a new CSV column and used for grouping,
with a one-time migration for existing rows.
Rewrite the README around the current pipeline: the twice-daily Telegram
digest, the sciCORE watcher, Telegram setup, and the two-stage filter
pipeline.
CLAUDE.md gains the bun daily commands, the daily/profile/telegram
architecture notes, and a scheduled-automations section covering both
launchd agents. Records that fetch.ts gates what reaches the CSV while
profile.ts gates what reaches the phone.
Keep each vacancy's href when flattening the listing HTML, resolved to an
absolute URL, so the Telegram alert links straight to the application page
instead of the listing index.
The extractor runs as python3 -c inside shell single quotes, so the anchor
pattern's quote class is built with chr(39) rather than written inline.
Adds the one-off scripts accumulated across recent sessions, removes
temp scripts whose searches now live in config.ts as role-family
sweeps, and updates fetch-other.ts plus the LinkedIn post drafts.
telegram.ts
- Bot API wrapper: HTML escaping, 4096-char chunking, rate-limit
spacing, and --setup to discover the chat ID from the bot's inbox.
Reads TELEGRAM_BOT (or TELEGRAM_TOKEN) and TELEGRAM_CHAT_ID.
profile.ts
- matchesProfile() scores a title against cv/polo.md. Hard rejects for
Microsoft stack, Java/C++/SAP, aggregators (Jobgether, ALLPS),
non-software functions and life-science domains; then weighted
signals for the core stack, platform/DevOps, data, AI/ML, cloud and
seniority. Threshold is 4 so short bullseye titles ("Data Engineer",
"DevOps Engineer") pass, and forward-deployed is weighted to clear it
on its own.
daily.ts
- Runs the fetch, diffs jobs.csv, keeps only profile matches, groups by
area and pushes to Telegram. results/sent.txt records every link
actually sent, written only after a successful send, so a failure
retries instead of losing jobs and an ad is never sent twice.
- --dry-run previews, --resend --date= re-sends a past day.
- A failed fetch reports itself rather than going quiet.
Scheduled at 08:00 and 18:00 by a launchd agent
(com.rafapolo.linkme-daily). launchd runs missed jobs on wake, and the
digest is diff-based, so downtime delays delivery without losing jobs.
Broaden the search beyond engineer/developer titles and stop three
classes of bad result from ever reaching the CSV.
config.ts
- PLATFORM_KEYWORDS (architect, SRE, infra, security, MLOps),
RESEARCH_KEYWORDS (research/scientific software, HPC, data platform)
and FIELD_KEYWORDS (forward-deployed, customer/sales, founding).
Separate OR-strings, not folded into KEYWORDS: LinkedIn ranks a
focused query far better than one 50-term disjunction.
- sweep() generates one search per area per family. 7 searches -> 17.
fetch.ts
- BIO_DOMAIN_RE drops life-science roles (bioinformatics, biostats,
GMP/MSAT, genomics, pathology, drug discovery). Filters on the role,
never the employer: software jobs at Roche/Novartis/sciCORE stay.
- requiresGerman() drops English postings demanding German above B1.
Two rules, both from real false results: the nearest marker wins
("Infrastructure skills are a plus" sat 95 chars before "Good
knowledge of German"), and an intervening language name breaks the
binding ("Fluent in English; German and/or French advantageous").
- GERMAN_TITLE_RE catches "(German-speaking)" without a page fetch.
- isViableContent/isViableUrl run the DE/FR and German-requirement
checks on a single fetch; isEnglishContent/isEnglishUrl kept for
recheck.ts and temp scripts.
- SOFTWARE_TITLE_RE widened so the new families survive it.
index.ts
- both content-check paths use the combined check and report which
filter dropped a job.
- move temp scripts to scripts/, cv files to cv/, jobs.csv to results/
- update index.ts and recheck.ts CSV path to results/jobs.csv
- fix imports in scripts/ to use ../config.ts and ../fetch.ts
- update CLAUDE.md and README.md to reflect new structure
- add plans/ with LinkedIn post drafts
- update .gitignore: ignore results/, jobs/, cv binaries, .DS_Store
- isEnglishContent() fetches each job page and drops descriptions with ≥6 German or French indicator words; called for every new job in index.ts before writing to CSV
- New `bun recheck` command (recheck.ts) runs searches in raw mode, lists filtered-out jobs, and prints word-frequency suggestions for SOFTWARE_TITLE_RE
- Expanded KEYWORDS and SOFTWARE_TITLE_RE to cover ML/AI, system/applications engineer, director, informatics, and more title variants
- Added `raw` flag to fetchJobs() to bypass the software-role filter
- Add CLAUDE.md with project workflow docs; add CV files and city temp scripts
- Add lead, technical, consultant, solutions, product manager, IT manager
to the software-role allowlist so relevant non-"software" titles pass
- Add JUNIOR_TITLE_RE to drop junior, intern, trainee, apprentice titles
- Paginated HTML scraper (up to 6 pages, 400ms delay between requests)
- Card-boundary-safe parsing to prevent field bleed across cards
- Language filter: drops German (m/w/d, umlauts) and French (H/F, accents) titles
- Software-role filter: requires title to contain a relevant tech term
- Global deduplication by job ID across runs
- 30-day recency filter via LinkedIn f_TPR parameter
- CSV output: date_found, posted_at, search, title, company, link
rodado