Every LinkedIn search and every fetch-other.ts source hits a different
host, so there was no real reason for the old for-loop to wait on each
one in turn - that's what made a full run take 30-40+ minutes. Runs
them concurrently now (bounded: 3 at a time within each group, both
groups started together), which brought a real run down to under 90
seconds.
existingLinks.add() now happens synchronously right after the .has()
check and before any await, which is what makes concurrent workers
safe against double-claiming the same link.
Concurrency surfaced a real regression: LinkedIn started returning
occasional 429s under parallel load, and fetchJobs treated that the
same as end-of-results, silently truncating the rest of that search.
Added retry-with-backoff (3 attempts, 1s/2s/4s) so a rate limit no
longer costs real listings.
jobbasel.ch's search route (/job/suche-{slug}) is server-rendered with
a window.__PRELOADED_STATE__ JSON blob, no headless browser needed.
englishjobsearch.ch works the same way via its ?format=markdown param.
Both swept with the same fixed keyword list job-room.ch already uses.
basel-jobs.ch was evaluated too but left out: it's DataDome-protected,
which blocks plain curl outright, and this project's fetchers are
intentionally browser-free.
While wiring these in, found that isEnglishDescription (now
isViableDescription) only ever ran the DE/FR word-count check against
inline descriptions, never requiresGerman() - so an English-written
posting demanding fluent German (e.g. PRODYNA's "you can communicate
fluently in both German and English") sailed straight through. That
gap existed in itjobs.ch and job-room.ch too, not just the two new
sources, so the fix applies to all four.
fetch.ts already parses each job card's own location, but index.ts
discarded it and stored only the search-area label in the CSV, so
daily.ts's telegram grouping used the query that found a job instead
of where it actually is (e.g. a Bern search surfacing a Lausanne job).
Now the location is written to a new CSV column and used for grouping,
with a one-time migration for existing rows.
Rewrite the README around the current pipeline: the twice-daily Telegram
digest, the sciCORE watcher, Telegram setup, and the two-stage filter
pipeline.
CLAUDE.md gains the bun daily commands, the daily/profile/telegram
architecture notes, and a scheduled-automations section covering both
launchd agents. Records that fetch.ts gates what reaches the CSV while
profile.ts gates what reaches the phone.
Keep each vacancy's href when flattening the listing HTML, resolved to an
absolute URL, so the Telegram alert links straight to the application page
instead of the listing index.
The extractor runs as python3 -c inside shell single quotes, so the anchor
pattern's quote class is built with chr(39) rather than written inline.
Adds the one-off scripts accumulated across recent sessions, removes
temp scripts whose searches now live in config.ts as role-family
sweeps, and updates fetch-other.ts plus the LinkedIn post drafts.
telegram.ts
- Bot API wrapper: HTML escaping, 4096-char chunking, rate-limit
spacing, and --setup to discover the chat ID from the bot's inbox.
Reads TELEGRAM_BOT (or TELEGRAM_TOKEN) and TELEGRAM_CHAT_ID.
profile.ts
- matchesProfile() scores a title against cv/polo.md. Hard rejects for
Microsoft stack, Java/C++/SAP, aggregators (Jobgether, ALLPS),
non-software functions and life-science domains; then weighted
signals for the core stack, platform/DevOps, data, AI/ML, cloud and
seniority. Threshold is 4 so short bullseye titles ("Data Engineer",
"DevOps Engineer") pass, and forward-deployed is weighted to clear it
on its own.
daily.ts
- Runs the fetch, diffs jobs.csv, keeps only profile matches, groups by
area and pushes to Telegram. results/sent.txt records every link
actually sent, written only after a successful send, so a failure
retries instead of losing jobs and an ad is never sent twice.
- --dry-run previews, --resend --date= re-sends a past day.
- A failed fetch reports itself rather than going quiet.
Scheduled at 08:00 and 18:00 by a launchd agent
(com.rafapolo.linkme-daily). launchd runs missed jobs on wake, and the
digest is diff-based, so downtime delays delivery without losing jobs.
Broaden the search beyond engineer/developer titles and stop three
classes of bad result from ever reaching the CSV.
config.ts
- PLATFORM_KEYWORDS (architect, SRE, infra, security, MLOps),
RESEARCH_KEYWORDS (research/scientific software, HPC, data platform)
and FIELD_KEYWORDS (forward-deployed, customer/sales, founding).
Separate OR-strings, not folded into KEYWORDS: LinkedIn ranks a
focused query far better than one 50-term disjunction.
- sweep() generates one search per area per family. 7 searches -> 17.
fetch.ts
- BIO_DOMAIN_RE drops life-science roles (bioinformatics, biostats,
GMP/MSAT, genomics, pathology, drug discovery). Filters on the role,
never the employer: software jobs at Roche/Novartis/sciCORE stay.
- requiresGerman() drops English postings demanding German above B1.
Two rules, both from real false results: the nearest marker wins
("Infrastructure skills are a plus" sat 95 chars before "Good
knowledge of German"), and an intervening language name breaks the
binding ("Fluent in English; German and/or French advantageous").
- GERMAN_TITLE_RE catches "(German-speaking)" without a page fetch.
- isViableContent/isViableUrl run the DE/FR and German-requirement
checks on a single fetch; isEnglishContent/isEnglishUrl kept for
recheck.ts and temp scripts.
- SOFTWARE_TITLE_RE widened so the new families survive it.
index.ts
- both content-check paths use the combined check and report which
filter dropped a job.
- move temp scripts to scripts/, cv files to cv/, jobs.csv to results/
- update index.ts and recheck.ts CSV path to results/jobs.csv
- fix imports in scripts/ to use ../config.ts and ../fetch.ts
- update CLAUDE.md and README.md to reflect new structure
- add plans/ with LinkedIn post drafts
- update .gitignore: ignore results/, jobs/, cv binaries, .DS_Store
- isEnglishContent() fetches each job page and drops descriptions with ≥6 German or French indicator words; called for every new job in index.ts before writing to CSV
- New `bun recheck` command (recheck.ts) runs searches in raw mode, lists filtered-out jobs, and prints word-frequency suggestions for SOFTWARE_TITLE_RE
- Expanded KEYWORDS and SOFTWARE_TITLE_RE to cover ML/AI, system/applications engineer, director, informatics, and more title variants
- Added `raw` flag to fetchJobs() to bypass the software-role filter
- Add CLAUDE.md with project workflow docs; add CV files and city temp scripts
- Add lead, technical, consultant, solutions, product manager, IT manager
to the software-role allowlist so relevant non-"software" titles pass
- Add JUNIOR_TITLE_RE to drop junior, intern, trainee, apprentice titles
- Paginated HTML scraper (up to 6 pages, 400ms delay between requests)
- Card-boundary-safe parsing to prevent field bleed across cards
- Language filter: drops German (m/w/d, umlauts) and French (H/F, accents) titles
- Software-role filter: requires title to contain a relevant tech term
- Global deduplication by job ID across runs
- 30-day recency filter via LinkedIn f_TPR parameter
- CSV output: date_found, posted_at, search, title, company, link
rodado