262 lines
38 KiB
Markdown
262 lines
38 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## Commands
|
||
|
||
| command | what it does |
|
||
|---|---|
|
||
| `bun fetch` | run all searches in `config.ts`, append new jobs to `results/jobs.csv` |
|
||
| `bun recheck` | fetch in raw mode, list filtered-out jobs + word-frequency keyword suggestions |
|
||
| `bun daily` | fetch, then push new profile-matching jobs to Telegram |
|
||
| `bun daily --dry-run` | same, but print the message instead of sending |
|
||
| `bun daily --resend` | skip fetching, re-send today's jobs from the CSV (`--date=YYYY-MM-DD` to pick a day) |
|
||
| `bun run telegram.ts --setup` | print the chat ID for `.env` after messaging the bot |
|
||
| `bun run mint-datadome` | open a real Chrome window to refresh the basel-jobs.ch DataDome cookie |
|
||
|
||
Temp scripts: `bun run scripts/<name>-temp.ts` — never edit `config.ts` for one-off searches.
|
||
|
||
## Structure
|
||
|
||
```
|
||
config.ts # search keywords and areas
|
||
fetch.ts # scraper, HTML parser, all filters
|
||
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch + data-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch, startup.jobs, arbeitnow.ch)
|
||
index.ts # orchestrator: runs searches, writes CSV
|
||
recheck.ts # raw-mode fetch, lists filtered-out jobs
|
||
daily.ts # digest: fetch → match against profile.ts → send to Telegram
|
||
profile.ts # title-level scoring; decides what's worth a notification
|
||
telegram.ts # Telegram delivery, chunking, --setup chat-ID discovery
|
||
scripts/ # one-off search scripts; import from ../config.ts and ../fetch.ts
|
||
scripts/scicore-watch.sh # daily launchd watch on sciCORE's open-positions page
|
||
results/sent.txt # links already pushed to Telegram — never sent twice
|
||
cv/ → ../rafapolo/job/cv/ # symlink; CVs (cv-source.md is canonical, polo.md is the ranking profile)
|
||
cv/polo.md # Rafael's profile — use for ranking jobs
|
||
results/jobs.csv # output — one row per job
|
||
jobs/ → ../rafapolo/job/applications/ # symlink; one folder per company: cover/motivation letters, dossiers, notes
|
||
plans/ # LinkedIn post drafts
|
||
```
|
||
|
||
## Architecture
|
||
|
||
- **`config.ts`** — `KEYWORDS` string (joined with ` OR `) and `searches: SearchConfig[]`. Edit here to add/remove search areas or keywords.
|
||
- Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform), `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product) and `FRONTEND_KEYWORDS` (frontend/React/web, data visualisation, geospatial/GIS, WebGL — added 2026-10-20 after a one-off search found 14 of 18 relevant frontend/viz roles missing from the CSV). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform, Field and Frontend add a Switzerland-wide remote pass.
|
||
- `RESEARCH_KEYWORDS` deliberately contains no bioinformatics/computational-biology terms — see `BIO_DOMAIN_RE`.
|
||
- **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`.
|
||
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch + data-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch, startup.jobs, arbeitnow.ch). Imported by `index.ts`.
|
||
- **novartis.com** (`fetchNovartisBasel`, added 2026-09-21) — Rafael has an internal contact at Novartis Basel who can refer him, so this source exists purely to catch a genuine software/engineering opening there, not to widen the CSV generally. Its career-search page (`novartis.com/careers/career-search?country[]=LOC_CH`) is a server-rendered Drupal view — no headless browser needed — but only at the **locale-less English path** (`/careers/career-search`); the `ch-de` German path's job-detail links 302 straight back to the search page for both locales when hit without a session, so detail pages only resolve through the English path. Pagination is 0-indexed (`page=0` implicit, `page=1` is the second page of 10) and the page's own "Showing N results" line gives the exact total, so the page-loop bound is computed rather than probed — the listing has also been observed to **repeat rows across pages within one sweep** (measured 2026-09-21: several titles appeared twice across 5 pages), so results are deduped by `req-<id>` before filtering, not trusted from the pager.
|
||
|
||
Novartis Basel's titling is almost entirely non-technical "Director, `<business function>`" roles (Portfolio Analytics, Therapeutic Area Strategy, Policy Communications, ...) plus device/hardware "Engineering" roles (autoinjector/drug-delivery hardware, not software) — the generic `SOFTWARE_TITLE_RE` used by every other source in this file matched **13 of 47** CH listings on title alone in an initial run, and every single one was a false positive (no engineering-team role among them). `NOVARTIS_TITLE_RE` replaces it here: it requires an explicit software/IT engineering job-family term (`software engineer`, `platform engineer`, `data engineer`, `devops`, `sre`, `solutions architect`, ...), not just a bare seniority or business-function word. `BIO_DOMAIN_RE` and job-room.ch's `PHARMA_ROLE_RE` still run on top to catch domain-science titles the stricter regex wouldn't otherwise need to worry about. Narrowed to Basel/Muttenz specifically (`NOVARTIS_BASEL_RE`) — the CH listing also includes Stein (Aargau), a different commute the contact couldn't refer into. As of 2026-09-21 this source keeps **zero** listings — expected; it's watching for the rare opening, not sweeping volume. Each detail page carries the full description inline (`job_details_content_center`…`job_details_content_bottom`), so the DE/FR + `requiresGerman` check runs in-process per candidate rather than costing `index.ts` a page fetch.
|
||
- **datacareer.ch** inverts the usual bot rule: its nginx **403s browser-like User-Agents** and serves honest ones (no UA, `curl/8.0`, Googlebot) fine. The spoofed Chrome UA in the shared `HEADERS` was the entire cause of the long-standing 403 — it uses `PLAIN_HEADERS` instead. Small board: Basel returns **8 jobs on one page with no pagination**, so `?page=2` legitimately yields 0 items.
|
||
- **job-room.ch** carries its language requirement as a **structured field** (`languageSkills[]`, e.g. `{languageIsoCode: "de", spokenLevel: "PROFICIENT"}`), entirely separate from the free-text `jobDescriptions[].description` — the UI renders it as "Deutsch (Fliessend)". `requiresGerman()`/`isViableDescription()` only ever scan the prose, so this field slipped through unchecked until 2026-09-04: measured against 1843 ads, 69 required PROFICIENT-level German and the prose check caught only 3 of them incidentally. `jobRoomRequiresGerman()` now checks the structured field directly (`spokenLevel`/`writtenLevel === "PROFICIENT"`) before the description-based check runs. Levels seen: `NONE`, `BASIC`, `INTERMEDIATE`, `PROFICIENT` — only `PROFICIENT` (fluent/C1+) is dropped, matching the B1 cutoff used everywhere else.
|
||
- **jobbasel.ch** looks like a JS-only React/Vite SPA (no jobs in the homepage's raw HTML), but its search route is server-rendered: `GET /job/suche-{slug}` returns full HTML with a `window.__PRELOADED_STATE__` blob containing the matched vacancies as JSON — no headless browser needed. Pagination is a URL suffix (`-seite-{n}`); a bare `?page=` query param is silently ignored. Swept with the same fixed keyword list as job-room.ch (no keyword-union endpoint exists) and deduped by vacancy id. Each item carries its full German description, so the DE/FR check runs in-process.
|
||
- **englishjobsearch.ch** also looks JS-only at first glance, but any page accepts `?format=markdown` and returns clean server-rendered markdown with full job cards — title, company, city, date, description. Hit the canton pages directly (`/in/canton-basel-stadt`, `/in/canton-basel-landschaft`) rather than the `/in/basel/` landing page, which only lists canton counts. Every job link is a `/clickout/<hash>` tracking redirect rather than the real posting URL — there's no way to recover the underlying employer link without following the redirect, so the clickout link is stored as-is.
|
||
- **basel-jobs.ch** is behind DataDome and **is now integrated** (`fetchBaselJobs`), contrary to the earlier note that it couldn't be. What the block actually permits, tested:
|
||
- plain `curl`/`fetch` → 403 on **every** UA, honest or spoofed;
|
||
- Chrome `--headless=new` over CDP → 403 (headless is detected);
|
||
- Playwright's own `chromium.launch()`, even headful → 403 (launch-path artefacts);
|
||
- a **real Chrome launched normally** with `--remote-debugging-port`, attached over CDP afterwards → **200**.
|
||
|
||
Clearance lives entirely in a `datadome` cookie (Max-Age 1 year) that **replays fine over plain `fetch`** with a matching User-Agent — the old claim that it "can't be extracted" was true of the browser-extension tooling, not of CDP. So `mintDataDomeSession()` opens a real window once, caches `{cookie, ua}` to `results/.datadome.json` (gitignored), and every run after that is plain HTTP. A rotated `Set-Cookie` is folded back into the cache; a 403 triggers one re-mint.
|
||
|
||
**Minting needs a visible window, so launchd runs can't do it.** With a stale cookie an unattended run logs `unavailable`, returns `[]`, and the rest of the fetch continues — it never fails the run. Refresh by hand with `bun run mint-datadome`, or set `LINKME_NO_BROWSER=1` to suppress minting entirely.
|
||
|
||
**Two passes, both needed.** Pass 1 walks the `informatik-jobs` category (~300 postings over 3 pages). Pass 2 sweeps `BASELJOBS_TERMS` through `/search?terms={kw}&page=1&websearch=1`. The category is a *curated taxonomy, not the whole board* — it holds only what the board filed under IT, so tech roles parked under Industrie/Pharma/Bau and roles whose relevance lives in the description never show up in pass 1. Measured 2026-09-01: pass 2 added **147 postings the category missed** (incl. "Senior AI Infrastructure & Platform Engineer (Basel)", "Senior DevOps / Cloud Engineer (Kubernetes, CI/CD, Linux)"), while the category held **257 the search missed**. 296 → 442 unique, 48 → 68 viable.
|
||
|
||
**Both search modes are swept — they query different indexes and each is blind to the other.** Do not drop one as redundant:
|
||
|
||
| term | `websearch=1` | bare `?terms=` |
|
||
|---|---|---|
|
||
| engineer | 90 | 260 found |
|
||
| frontend | **0** | 2 |
|
||
| backend | **0** | 4 |
|
||
| react | **0** | 2 |
|
||
|
||
`websearch=1` matches role/domain words and reaches postings filed outside the IT category. The bare search is the **only** mode that matches stack terms — `react`, `frontend`, `backend`, `javascript` return literally zero under `websearch=1`. Neither paginates: both ignore `page` and return the whole result set in one response (bare caps around 100 cards). Measured 2026-09-01 against the category: websearch added 147, bare a further 139 raw (23 past the title filters). 48 → 68 → **88 viable**.
|
||
|
||
German stems (`entwickler`, `informatik`) are in the term list deliberately: search matches *description* text, so they surface English-titled postings the category never filed under IT; the language filters drop the German-language ones afterwards. The bare search on `architect`/`security` drags in a lot of German noise (*Architekt* = building architect, *Sicherheitsassistent*, *Gerüstbauer*) — that's expected and `NON_ENGLISH_RE`/`SOFTWARE_TITLE_RE` handle it, 139 raw → 23.
|
||
|
||
Pagination/dedup: page 4+ of the category still serves 2 sponsored slots, so **"no new ids" is the end condition**, not an empty list. Sponsored cards repeat within a page, across pages, and across both passes — dedupe on `data-gtm-jid`. Cards carry no date, so `postedAt` is the discovery date. Canton comes from `data-gtm-jp` (sometimes `"BL,BS"`).
|
||
|
||
**Pass 3: data-jobs.ch** (added 2026-10-22) — the same jobchannel network's data/analytics board, swept for BS/BL/ZH/BE/LU (`DATAJOBS_PROVINCES`) inside `fetchBaselJobs`: same card markup and parser, and the same DataDome cookie clears both hosts (a cookie rotated by one host works on the other). `/search?provinces=…&lang=en-ch&page=n` walks the board's own listings (100 per page; past the end it still serves a few sponsored cards, so "no new ids" ends the walk), then one `&page=1&websearch=1` request returns all the "further jobs from the web search" (it ignores `page`). Measured 2026-10-22: 166 own + 98 web search = 264 unique for the five cantons. Vacancy ids are network-wide, so it runs last and shares `seen` — a posting on both boards keeps its basel-jobs.ch link. Its inventory is **not** a subset of basel-jobs.ch (data-only postings 404 there): measured 2026-10-22, 26 ids the basel-jobs passes missed. A posting on both boards keeps the same `/job/<slug>/<id>` path, so `knownJobchannelTails()` skips any card whose path is already in the CSV under either host — before its detail fetch, not after. Re-checking every known posting on every run is what got the cookie blocked on the first five-canton test (2026-10-22). For the same reason a detail page that fails to load is now **skipped** (retried next run) rather than kept unchecked: a blocked cookie fails every remaining page at once. Rows get `source`/`search` = `data-jobs.ch`.
|
||
|
||
**Language-check the `div.plain-content` block only, never the whole page.** The site chrome is German and on its own clears the ≥ 6 DE/FR word threshold — checking the full page dropped *every* job, English ones included. That block is the mobile rendering of the description; the desktop view iframes the same text, so it's the only copy in the server-rendered HTML.
|
||
- **Basel international orgs** (`fetchBaselIntlOrgs`) sweeps three boards under one label, each only a handful of postings:
|
||
- **BIS** — the career site is JS-rendered with no usable API, so we read the UNjobs mirror at `unjobs.org/offices/bis_182320855689338880`. Titles arrive as `Role, City, Country` and get split back apart. BIS also cycles Senior Software / Data / Data & Analytics Engineer roles through this board.
|
||
- **Swiss TPH** — Umantis at `jobs.swisstph.ch/Jobs/All`. Site chrome is German even on English postings, so `isEnglishUrl` would reject everything; `contentChecked: true` and the title filters decide instead.
|
||
- **Basel Institute on Governance** — `baselgovernance.org/about/careers`. Each vacancy is a linked PDF, so there's no HTML description to language-check. The `rounded-3xl` card class is what separates vacancy tiles from the annual-report PDF link.
|
||
- These boards use UN-flavoured titles ("Information Management Officer", "ICT Specialist") that `SOFTWARE_TITLE_RE` alone drops, so `INTL_TECH_RE` widens the net **for these sources only**. Boards this small can afford the noise; missing a rare opening can't.
|
||
- **itjobs.ch** is swept Switzerland-wide (no `location` param) — the board is small, ~210 jobs over 7 pages, and locking it to Basel returned only ~9. It 404s one page past the end instead of serving an empty list. Each item embeds its full `description`, so the DE/FR check runs in-process via `isEnglishDescription` rather than costing a page fetch per job. Use `job_location.name` for location — it's the English form; the bare `location` field is German.
|
||
- **scicore.unibas.ch** (`fetchSciCore`) replaces the standalone `scripts/scicore-watch.sh` launchd watcher (removed 2026-09-16) — sciCORE now runs inside `bun fetch`/`bun daily` like every other source instead of its own 08:47 agent. The open-positions page is a plain-text announcement list, not a real job board: each `<li>` under `<h1>Join the team!</h1>` is either an active posting linked to `careers.unibas.ch` or a closed/in-review notice with no link — items matching `deadline has passed`/`interview process is (still) ongoing` are skipped since they're not open applications. `contentChecked: true` because the listing itself is the only English text available; there's no separate description page to fetch. Title still has to clear `SOFTWARE_TITLE_RE`, same as every other source — this is deliberately broader than the old watcher's "software"-only match (it also catches "security", matching the IT Security Engineer role Rafael prepped for — see `project_scicore_security_report` in memory) but still drops pure sysadmin titles like "Linux Systems Administrator".
|
||
- **swisscom.com** (`fetchSwisscom`, added 2026-09-18) — Swisscom's own Workday career site (`swisscom.wd103.myworkdayjobs.com/SwisscomExternalCareers`). Small board (~47 openings, Switzerland-wide), so no keyword sweep — the full listing is pulled and filtered locally, like itjobs.ch. The listing page itself is a bare JS shell with no jobs in the raw HTML; postings live behind the standard Workday CXS JSON API (`POST /wday/cxs/swisscom/SwisscomExternalCareers/jobs`). `total` on the first page (offset 0) is authoritative — later pages report `total: 0` (a tenant quirk), and an offset past the true total silently wraps around and re-serves page 1 instead of an empty list, so the loop's stop condition is the first page's total, never a later one's. Listing rows carry no description, so each title-filtered candidate costs one extra fetch against the job detail endpoint (`/job{externalPath}`), which returns the full HTML description — the DE/FR + `requiresGerman` check runs on that in-process (`contentChecked: true`).
|
||
|
||
Swisscom mostly hires in German — of 47 postings, only ~16 clear the English/software title filters, and nearly all of those turned out to be written in German too; only genuinely-English, German-optional roles survive. Adding this source surfaced a real bug in `requiresGerman()`: the stock line *"Fluency in English is required; proficiency in German is considered an advantage"* had the noun "proficiency" (a required-marker word) sitting right next to "German", closer than the real optional-marker "advantage" — nearest() picked the wrong one and wrongly flagged the posting as requiring German. Fixed by adding a local-predicate override (`GERMAN_LOCAL_PREDICATE_RE`): when a copula ("is"/"are", optionally "considered"/"deemed") directly follows the German mention, that immediate predicate decides the verdict outright instead of competing on raw character distance with markers elsewhere in the window — "German is considered an advantage" resolves as optional regardless of what else is nearby. It only fires when a copula immediately follows the mention, so *"Work proficiency in German, English, and French"* (no copula after "German", a genuine hard requirement — Senior Operation Architect (FR/GER/ENG)) still falls through to the general nearest-marker logic and is correctly caught as required. Re-check both cases (and the pre-existing Helvetia/albedis/Karakun/J. Safra Sarasin/BBF/spec-sheet cases in the paragraph above) before touching `requiresGerman` again.
|
||
- **more-jobs.ch** (`fetchMoreJobs`, added 2026-10-02) — server-rendered PHP aggregator. `/recherche?q=<term>&page=n` is fast (~0.5 s) with ~20 cards/page, but `/emplois` and out-of-range pages are slow (20 s+) and **clamp to the last page instead of going empty**, so a term's sweep ends on "no new ids", not an empty list. Swept with `MOREJOBS_TERMS` (≤5 pages each), then each title-filtered candidate gets one detail fetch (4 in parallel, 20 s timeout) for the real `Publication date` (dd.mm.yyyy) and the in-process DE/FR + `requiresGerman` check (`contentChecked: true`). The board **reposts one opening under several IDs** (measured: Accelleron, Swissquote, Bayer ×3), and it also mirrors jobs that LinkedIn/jobs.ch carry, so its `extraSources` entry sets `dedupeByTitle`: `index.ts` skips any job whose `langKey(title, company)` is already in the CSV (or claimed earlier in the same run). Other sources are deliberately not deduped this way. The board also lists **foreign postings** ("Eastern Europe, Poland", "95050 Santa Clara", "London") and its JSON-LD stamps `addressCountry: "CH"` on every one, so the card's location text is the only signal: `MOREJOBS_FOREIGN_RE` drops locations naming another country/foreign city or carrying a 5-digit postcode before the detail fetch (19 such rows had leaked into the CSV by 2026-10-03, several reaching the Telegram digest).
|
||
- **startup.jobs** (`fetchStartupJobs`, added 2026-10-06) — global startup board, swept Switzerland-wide. **Every path is behind a Cloudflare managed challenge** (403 on plain HTTP for any UA, feeds and `/api` included), and unlike basel-jobs.ch's DataDome cookie, `cf_clearance` **does not replay** over Bun's fetch — measured 403 with the exact cookie + UA; it's bound to Chrome's TLS fingerprint. What works: Chrome `--headless=new` with a throwaway profile clears the challenge **only if the UA has no "HeadlessChrome" token** (the version is read from `Chrome --version` so it doesn't drift). So this source launches a headless browser on every run — unlike basel-jobs.ch's minting, that works fine under launchd. `LINKME_NO_BROWSER=1` disables it; any failure logs `unavailable` and returns `[]`.
|
||
|
||
Listings come from **Algolia** (app `4CQMTMMK73`, index `Post_production`), not the HTML: the page embeds a *secured* search key in `meta[name="current-algolia-api-key-search"]`, and algolia.net is not behind Cloudflare, so the queries run over plain fetch. The key's `validUntil` is **~3h after page load** — read fresh every run, never cached. The key rejects every country filter (`country_code`/`country_id` → 0 hits), so the sweep is a geo bounding box around Switzerland with `country_code === "CH"` checked locally (the box also catches DE/FR/AT/LI/IT), limited to the last 60 days. Algolia caps a query at **500 retrievable hits** (100 × 5 pages) — a box over 500 is split into quadrants recursively (all ~1400 CH-box postings vs 556 in the 60-day window, measured 2026-10-06). `tagFilters` on `_tags` (Developer/Engineer) was tried and dropped: it covered only 365 of them. Records carry no description: each title-filtered candidate's detail page is fetched *inside* the browser (`page.evaluate(fetch)`, same TLS) and its JSON-LD `JobPosting.description` goes through `isViableDescription` (`contentChecked: true`). Links already in `results/jobs.csv` are skipped before that fetch. It aggregates company ATS feeds (Wise, Stripe, Avaloq, Nebius…) that LinkedIn also carries, so its `extraSources` entry sets `dedupeByTitle` like more-jobs.ch. The user's original URL was `?q=Developer&w=hybrid` around Basel; the source deliberately keeps on-site/remote too and uses `SOFTWARE_TITLE_RE` instead of the `Developer` query, matching every other source.
|
||
- **arbeitnow.ch** (`fetchArbeitnow`, added 2026-11-03) — free public JSON API (`/api/job-board-api?page=n`, 100/page, no auth), fed straight from company ATS feeds. Rolling ~7-day window (~195 jobs), so 2 requests cover it. Full description inline → in-process DE/FR + `requiresGerman` check. `dedupeByTitle` (overlaps ~50% with LinkedIn). **pilotjob.ch was evaluated and rejected**: its listing only offers `src=arbeitnow|jobroom`, i.e. it re-serves this feed plus job-room.ch. englishjobsearch.ch/jobs/remote shares only 1 of arbeitnow's 19 remote postings.
|
||
- **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV. Writes both `search` (the query/source that found the job) and `location` (the posting's own location, as parsed off its card by `fetch.ts`/`fetch-other.ts` — no extra page fetch) to the CSV. Migrates old CSVs on startup: adds `source` if missing, then `location` (backfilled `""` for rows written before the column existed).
|
||
- Every LinkedIn search and every `fetch-other.ts` source hits a different host, so they run **concurrently** (bounded: 3 at a time within each group, both groups started together) instead of one after another — cut a full `bun fetch` from ~16-20 min to well under 2 min. `existingLinks.add()` happens synchronously right after the `.has()` check and before any `await`, which is what makes concurrent workers safe against double-claiming the same link. Concurrency surfaced a real gap: LinkedIn started returning occasional 429s under parallel load, which `fetchJobs` used to treat as end-of-results (silently truncating that search) — `fetchJobs` now retries a 429 up to 3× with backoff (1s/2s/4s) before giving up.
|
||
- **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions.
|
||
- **`daily.ts`** — the notification layer on top of `index.ts`. Snapshots the CSV's links, shells out to `bun fetch`, then diffs to find what's new. Survivors are filtered through `matchesProfile` and grouped by area (Basel / Remote / Zürich / Bern / Luzern) so the message scans on a phone. `results/sent.txt` is written **only after a successful send**, so a Telegram failure retries next run instead of silently losing jobs. A non-zero exit from `bun fetch` sends a failure notice rather than failing quietly.
|
||
- The area grouping matches against the CSV's `location` column (the posting's real location) first, falling back to `search` (the query/source label) only when `location` is empty — i.e. for rows written before the `location` column existed. This fixed two real mislabels: a LinkedIn "Bern" sweep can surface a job actually based in Lausanne (search label ≠ real location), and `fetch-other.ts` sources (job-room.ch, itjobs.ch, stellen-basel.ch, datacareer.ch) write the source name into `search` instead of a city, which used to dump their jobs into "⚪️ Other" regardless of where they actually were.
|
||
- **`profile.ts`** — `matchesProfile(title, company, minScore = 4)`. Hard rejects run first (Microsoft stack, Java/C++/SAP, life-science, non-software functions, blocked companies Jobgether and ALLPS), then `SIGNALS` scores stack and seniority terms from `cv/polo.md`. This is a **second, stricter gate than the fetch filters**: `fetch.ts` decides what lands in the CSV, `profile.ts` decides what's worth a phone buzz. Widen `SOFTWARE_TITLE_RE` to catch more in the CSV; tune `SIGNALS`/`minScore` to change what gets sent.
|
||
- **`telegram.ts`** — reads `.env` directly (no dotenv dep). Accepts `TELEGRAM_BOT` or `TELEGRAM_TOKEN` for the token. Splits at 3900 chars on line boundaries (Telegram hard-caps at 4096) and sleeps 400 ms between parts to stay under the rate limit. Messages are HTML — run titles/companies through `escapeHtml`.
|
||
|
||
### Filter pipeline (applied in order)
|
||
|
||
1. **Switzerland location guard** (card HTML) — extracts `job-search-card__location` from each card; drops jobs where location doesn't contain "Switzerland". Prevents LinkedIn from returning global results when geoId is ignored on unauthenticated requests.
|
||
2. **`NON_ENGLISH_RE`** (title) — drops German gender markers, umlauts, `/in` suffix, French accents/markers. Runs inside `fetchJobs`.
|
||
3. **`JUNIOR_TITLE_RE`** (title) — drops junior/intern/trainee/werkstudent. Runs inside `fetchJobs`.
|
||
4. **`JAVA_TITLE_RE`** (title) — drops Java-titled roles. Runs inside `fetchJobs`.
|
||
5. **`BIO_DOMAIN_RE`** (title) — drops roles whose *domain* is biology/chemistry/the lab bench (bioinformatics, biostatistics, GMP/MSAT, genomics, pathology, drug discovery, bench Scientist titles). Filters on the **role, not the employer** — software/platform/data jobs at Roche, Novartis, Swiss TPH and sciCORE stay in scope. Runs inside `fetchJobs`.
|
||
6. **`SOFTWARE_TITLE_RE`** (title) — requires a relevant tech/role term. Skipped when `raw: true`. Runs inside `fetchJobs`.
|
||
6.5. **`GERMAN_TITLE_RE`** (title) — drops "(German-speaking)" / "deutschsprachig" titles without a description fetch. Runs inside `fetchJobs`.
|
||
7. **`isViableContent`** / **`isViableUrl`** (description page) — one fetch, two checks: (a) counts DE/FR indicator words, drops if ≥ 6; (b) `requiresGerman()` drops English postings that demand German above B1. Runs in `index.ts` only (not `recheck.ts`). Adds 350 ms delay per job. `isEnglishContent` / `isEnglishUrl` remain exported for `recheck.ts` and temp scripts.
|
||
|
||
**`requiresGerman(text)`** is deliberately more than a keyword match — Rafael is German B1 (TELC), so B1/"basic"/"a plus" is fine and B2+ is not. For each mention of German it takes the *nearest* qualifying marker, and a marker is ignored when another language name sits between it and the mention. Both rules come from real false results:
|
||
- *"Infrastructure skills are **a plus** … Good knowledge of **German**"* (Helvetia) — presence-based matching let an unrelated "a plus" 95 chars away cancel a real requirement.
|
||
- *"Fluent in **English**; **German** and/or French advantageous"* (albedis) — "Fluent" is nearer to "German" than "advantageous" is, but qualifies English.
|
||
|
||
Three further rules, added 2026-09-01 after an audit found the filter catching **0 of 24** basel-jobs.ch postings that mention German:
|
||
- **Coordination beats the intervening-language rule.** *"Excellent communication skills in both **English** and **German**"* (Karakun) and *"Fluency in **English** and **German**"* (J. Safra Sarasin) are hard requirements, but "English" sitting between the marker and the mention handed the marker to English and waved them through. A marker now still binds when the other language is *coordinated* with German (`and`/`und`/`sowie`/`&`/`as well as`). Deliberately **excludes `and/or` and a bare `or`** — *"Very good English and/or German"* (BBF) is satisfied by English alone and must stay optional.
|
||
- **German-language markers actually match now.** `sehr\s+gute?` could never match *"Sehr gutes Deutsch"* — the trailing `\b` fails against the "s". It's `sehr\s+gut\w*` now, plus `fliessend\w*`, `Deutschkenntnisse` and `erforderlich`. This is why every German-written Swisscom posting ("Sehr gutes Deutsch in Wort und Schrift") passed the filter.
|
||
- **Spec-sheet language lines.** *"Languages: German & English"* (contractor postings) has no adjective to bind to, so the `Languages:`/`Sprachen:` label itself counts as a required marker. It still goes through the nearest-marker comparison, so *"Language: English and German (bonus)"* correctly loses to the closer "bonus".
|
||
|
||
Regression cases for all of the above live in the git history of this change; re-check them before touching `requiresGerman` — the optional/required boundary is where every past bug has been.
|
||
|
||
Deduplication: job IDs are extracted from `results/jobs.csv` on startup; any ID already seen is skipped before content-checking.
|
||
|
||
### SearchConfig
|
||
|
||
`url` overrides all filter params. Otherwise build from: `keywords`, `geoId`/`location`, `workType`, `jobType`, `timePosted`, `seniority`.
|
||
|
||
Known Swiss geoIds:
|
||
- Basel city: `103383283`
|
||
- Basel Metropolitan Area: `90009885`
|
||
- Basel-Country: `107117530`
|
||
- Zürich city: `102436504`
|
||
- Zürich Metropolitan Area: `107814425`
|
||
- Zürich Area: `90009888`
|
||
|
||
Use `location: "City, Switzerland"` (no `geoId`) when the geoId is unknown — LinkedIn resolves by name.
|
||
|
||
**Auth warning:** Do NOT send the `li_at` cookie in job search requests (`/jobs/search/`). With the cookie, LinkedIn serves the React SPA instead of server-rendered HTML, breaking the card parser entirely. The cookie is only useful for individual profile pages.
|
||
|
||
## Searching any city / mode temporarily
|
||
|
||
Write a throwaway script in `scripts/` (e.g. `scripts/bern-temp.ts`) — **never touch `config.ts`** unless the user explicitly asks to save the change. Delete temp scripts after the session unless the user asks to keep them.
|
||
|
||
Common patterns:
|
||
|
||
```ts
|
||
// City search (onsite + hybrid)
|
||
import { KEYWORDS } from "../config.ts";
|
||
import { fetchJobs } from "../fetch.ts";
|
||
import type { SearchConfig } from "../config.ts";
|
||
|
||
const config: SearchConfig = {
|
||
label: "Bern", keywords: KEYWORDS, location: "Bern, Switzerland",
|
||
workType: ["onsite", "hybrid"], timePosted: "month",
|
||
};
|
||
const jobs = await fetchJobs(config);
|
||
for (const j of jobs) console.log(j.title, j.company, j.link);
|
||
```
|
||
|
||
```ts
|
||
// Remote-only across all Switzerland
|
||
const config: SearchConfig = {
|
||
label: "Switzerland Remote", keywords: KEYWORDS, location: "Switzerland",
|
||
workType: ["remote"], timePosted: "month",
|
||
};
|
||
```
|
||
|
||
```ts
|
||
// Description-level keyword check (e.g. Ruby/Rails)
|
||
const RUBY_RE = /\bruby\b|\brails\b/i;
|
||
// fetch each job page, test RUBY_RE against the description text
|
||
```
|
||
|
||
Extra filters for temp scripts (import from `../fetch.ts`):
|
||
- `JAVA_TITLE_RE` — drop Java roles
|
||
- `JUNIOR_TITLE_RE` — drop junior/intern
|
||
- `isEnglishContent(id)` — drop DE/FR description content
|
||
- Add your own inline RE for EMEA, .NET, etc.
|
||
|
||
## Application history (`jobs/`, `cv/`)
|
||
|
||
`cv/` and `jobs/` are symlinks into the **private** `../rafapolo/job/` (gitignored in both repos; `rafapolo` is a public GitHub Pages site, so never move these into a non-ignored path there). Everything else about past applications lives beside them: `../rafapolo/job/career-strategy-notes.md`, `credentials/` (certs, diplomas, references) and the dossier PDFs at the root.
|
||
|
||
Before presenting a job or writing a letter, check `jobs/<company-slug>/` (e.g. `ls jobs | grep -i roche`): if a folder exists, Rafael already applied there — say so, reuse the earlier letter as a starting point, and mention any track record in memory (e.g. Adobe and BIS cold applies are dead). New letters go in `jobs/<company-slug>/`.
|
||
|
||
## Ranking jobs against the CV
|
||
|
||
Read `cv/polo.md` for the full profile. Key signals:
|
||
|
||
- **Seniority**: Senior / Staff / Principal / Lead / Architect — weight titles accordingly
|
||
- **Core stack**: Python · TypeScript · Ruby on Rails · React · Node.js
|
||
- **Domain strengths**: platform engineering · data pipelines · AI/LLM integrations · DevOps (Docker, Kubernetes, CI/CD, AWS)
|
||
- **Background edges**: fintech/CBDC (Moedax), civic tech (Tactical Tech / GFOSS), independent consulting (ExtraPolo, 40+ projects)
|
||
- **Languages**: English (full) · German B1 TELC certified · Spanish (full) — **drop** roles requiring German above B1; `requiresGerman()` does this automatically at fetch time
|
||
- **Hard no: biological domains** — drop bioinformatics, computational biology, biostatistics, lab/bench Scientist, GMP/MSAT, clinical/assay and drug-discovery-science roles; he has no life-science background. Software roles *at* pharma companies are still strong targets
|
||
- **Hard no: Microsoft stack** — drop .NET/C#, Azure-first, Windows/SCCM/Intune, SharePoint, Dynamics, Power Platform roles entirely; don't present them even as weak fits
|
||
- **Location**: based in Basel, open to Basel / Bern / Zürich; EU citizen, no permit needed
|
||
|
||
## Output format
|
||
|
||
**Always** present jobs as a markdown table — never as a bullet list or prose: `#`, `Title`, `Company`, `Canton`, `Why it fits`, `Link`.
|
||
|
||
`Canton` is mandatory on every table, including single-city searches and the list of dropped/rejected jobs (there `Why dropped` replaces `Why it fits`). Derive it from the actual location on the posting, not from the search label — recruiters and multi-site companies often post under a different area than the job sits in. Use the two-letter code: BS, BL, ZH, BE, ZG, LU, VD, GE, TI, AG, SG, TG. For remote roles write `Remote` (add scope if known, e.g. `Remote (EU)`).
|
||
|
||
Links use just the job ID as anchor text: `[4192954199](https://www.linkedin.com/jobs/view/4192954199/)`
|
||
|
||
Some employers (Optiml, LocalBini seen repeatedly) repost the same role under a new job ID every few weeks rather than keeping one listing open. When several rows share the same title + company, show only the most recent job ID and drop the older duplicates from the table — they're the same opening, not new leads.
|
||
|
||
## Workflow: "find jobs in X city / remote / custom search"
|
||
|
||
1. Write a temp script in `scripts/` with the right `SearchConfig` (city, workType, custom keywords).
|
||
2. Apply extra inline filters as needed: EMEA, .NET, description-level keyword check.
|
||
3. Run `isEnglishContent` per job to drop DE/FR content.
|
||
4. Rank survivors against `cv/polo.md`, present as a table.
|
||
5. Delete temp scripts after the session unless asked to keep them.
|
||
|
||
## Scheduled automations
|
||
|
||
Runs as a **launchd agent** — this machine has no cron at all, so don't reach for `crontab`.
|
||
|
||
| plist (`~/Library/LaunchAgents/`) | when | runs |
|
||
|---|---|---|
|
||
| `com.rafapolo.linkme-daily` | 09:00 / 12:00 / 15:00 / 17:00 / 20:00 | `bun run daily.ts` → `results/daily.log` |
|
||
| `com.rafapolo.linkme-watch-delivery-lead` | 10:00 | `bun run scripts/watch-delivery-lead.ts` → `results/watch-delivery-lead.log`; watches for the MRP-Global "Delivery Lead – Agentic Solutions" opening (LinkedIn 4473851998) resurfacing elsewhere and for the original closing. Telegram only on a hit. Unload once the lead is resolved |
|
||
|
||
If the Mac is asleep at the scheduled time, launchd fires the job on wake rather than skipping the day. `launchctl list | grep linkme` to check it's loaded; `launchctl unload <plist>` to stop it.
|
||
|
||
**`scripts/scicore-watch.sh`** is retired (2026-09-16) — sciCORE is now swept by `fetchSciCore` in `fetch-other.ts` as part of `bun fetch`/`bun daily`, see the `fetch-other.ts` entry above. Its `ch.polo.scicore-watch` launchd agent (08:47) was unloaded and its plist removed; the script and `results/.scicore-watch/` (snapshot + log) are left in the repo for reference/history, not run anymore.
|
||
|
||
## LinkedIn session token
|
||
|
||
`.env` holds `LI_AT` (LinkedIn session cookie), plus `TELEGRAM_BOT` and `TELEGRAM_CHAT_ID` for the digest. It is gitignored.
|
||
|
||
`LI_AT` is the `li_at` cookie value. Use it to fetch authenticated LinkedIn pages via curl:
|
||
|
||
```bash
|
||
curl -s "https://www.linkedin.com/in/<slug>" \
|
||
-H "Cookie: li_at=$(grep LI_AT .env | cut -d= -f2-)" \
|
||
-H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" \
|
||
--compressed -o /tmp/profile.html
|
||
```
|
||
|
||
**What works:** Full HTML pages for profiles that don't 999-block (varies by profile). Extract data with Python regex on the HTML.
|
||
|
||
**What doesn't:** LinkedIn Voyager API returns 401/403 (CSRF issues). `ch.linkedin.com` slugs sometimes work when `www.linkedin.com` returns HTTP 999. Public profiles without login redirect to a 1530-byte JS stub — no useful data.
|
||
|
||
**Tip:** Profile data is embedded as escaped JSON in the HTML. Use `html.unescape()` + regex for `"headline"`, `"locationName"`, `defaultLocalizedName` fields.
|
||
|
||
## Workflow: "check if new keywords should be added"
|
||
|
||
1. Run `bun recheck`.
|
||
2. Identify false negatives — real software roles that got dropped.
|
||
3. Ignore generic frequency suggestions (`engineer`, `development`, `senior`).
|
||
4. Propose specific additions to `SOFTWARE_TITLE_RE` in `fetch.ts` or words to `NON_ENGLISH_RE`.
|