linkme/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Commands

| command | what it does |
|---|---|
| `bun fetch` | run all searches in `config.ts`, append new jobs to `results/jobs.csv` |
| `bun recheck` | fetch in raw mode, list filtered-out jobs + word-frequency keyword suggestions |
| `bun daily` | fetch, then push new profile-matching jobs to Telegram |
| `bun daily --dry-run` | same, but print the message instead of sending |
| `bun daily --resend` | skip fetching, re-send today's jobs from the CSV (`--date=YYYY-MM-DD` to pick a day) |
| `bun run telegram.ts --setup` | print the chat ID for `.env` after messaging the bot |

Temp scripts: `bun run scripts/<name>-temp.ts` — never edit `config.ts` for one-off searches.

## Structure

```
config.ts          # search keywords and areas
fetch.ts           # scraper, HTML parser, all filters
fetch-other.ts     # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch)
index.ts           # orchestrator: runs searches, writes CSV
recheck.ts         # raw-mode fetch, lists filtered-out jobs
daily.ts           # digest: fetch → match against profile.ts → send to Telegram
profile.ts         # title-level scoring; decides what's worth a notification
telegram.ts        # Telegram delivery, chunking, --setup chat-ID discovery
scripts/           # one-off search scripts; import from ../config.ts and ../fetch.ts
scripts/scicore-watch.sh  # daily launchd watch on sciCORE's open-positions page
results/sent.txt   # links already pushed to Telegram — never sent twice
cv/polo.md         # Rafael's profile — use for ranking jobs
cv/                # CV files (html, pdf, odt)
results/jobs.csv   # output — one row per job
jobs/              # notes on specific job leads
plans/             # LinkedIn post drafts
```

## Architecture

- **`config.ts`** — `KEYWORDS` string (joined with ` OR `) and `searches: SearchConfig[]`. Edit here to add/remove search areas or keywords.
  - Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform) and `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform and Field add a Switzerland-wide remote pass.
  - `RESEARCH_KEYWORDS` deliberately contains no bioinformatics/computational-biology terms — see `BIO_DOMAIN_RE`.
- **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`.
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch). Imported by `index.ts`.
  - **datacareer.ch** currently returns **403** on every page — the source is dead and yields nothing. Needs new headers or a different entry point.
  - **jobbasel.ch** looks like a JS-only React/Vite SPA (no jobs in the homepage's raw HTML), but its search route is server-rendered: `GET /job/suche-{slug}` returns full HTML with a `window.__PRELOADED_STATE__` blob containing the matched vacancies as JSON — no headless browser needed. Pagination is a URL suffix (`-seite-{n}`); a bare `?page=` query param is silently ignored. Swept with the same fixed keyword list as job-room.ch (no keyword-union endpoint exists) and deduped by vacancy id. Each item carries its full German description, so the DE/FR check runs in-process.
  - **englishjobsearch.ch** also looks JS-only at first glance, but any page accepts `?format=markdown` and returns clean server-rendered markdown with full job cards — title, company, city, date, description. Hit the canton pages directly (`/in/canton-basel-stadt`, `/in/canton-basel-landschaft`) rather than the `/in/basel/` landing page, which only lists canton counts. Every job link is a `/clickout/<hash>` tracking redirect rather than the real posting URL — there's no way to recover the underlying employer link without following the redirect, so the clickout link is stored as-is.
  - **basel-jobs.ch** was evaluated but is **not integrated**: it's protected by DataDome (not basic Cloudflare), which blocks plain `curl` outright. A real Chrome session gets through, confirming the content is genuine, but that only proves the block isn't automatable within this project's curl-only architecture — `bun daily`/`bun fetch` run headless via launchd with no browser available, and the session cookie that lets a real browser through can't be extracted for replay (browser tooling blocks cookie access). Would need persistent Playwright + stealth infra to chase further, which isn't worth it for what's likely a small regional board already covered by jobbasel.ch and stellen-basel.ch.
  - **Basel international orgs** (`fetchBaselIntlOrgs`) sweeps three boards under one label, each only a handful of postings:
    - **BIS** — the career site is JS-rendered with no usable API, so we read the UNjobs mirror at `unjobs.org/offices/bis_182320855689338880`. Titles arrive as `Role, City, Country` and get split back apart. BIS also cycles Senior Software / Data / Data & Analytics Engineer roles through this board.
    - **Swiss TPH** — Umantis at `jobs.swisstph.ch/Jobs/All`. Site chrome is German even on English postings, so `isEnglishUrl` would reject everything; `contentChecked: true` and the title filters decide instead.
    - **Basel Institute on Governance** — `baselgovernance.org/about/careers`. Each vacancy is a linked PDF, so there's no HTML description to language-check. The `rounded-3xl` card class is what separates vacancy tiles from the annual-report PDF link.
    - These boards use UN-flavoured titles ("Information Management Officer", "ICT Specialist") that `SOFTWARE_TITLE_RE` alone drops, so `INTL_TECH_RE` widens the net **for these sources only**. Boards this small can afford the noise; missing a rare opening can't.
  - **itjobs.ch** is swept Switzerland-wide (no `location` param) — the board is small, ~210 jobs over 7 pages, and locking it to Basel returned only ~9. It 404s one page past the end instead of serving an empty list. Each item embeds its full `description`, so the DE/FR check runs in-process via `isEnglishDescription` rather than costing a page fetch per job. Use `job_location.name` for location — it's the English form; the bare `location` field is German.
- **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV. Writes both `search` (the query/source that found the job) and `location` (the posting's own location, as parsed off its card by `fetch.ts`/`fetch-other.ts` — no extra page fetch) to the CSV. Migrates old CSVs on startup: adds `source` if missing, then `location` (backfilled `""` for rows written before the column existed).
  - Every LinkedIn search and every `fetch-other.ts` source hits a different host, so they run **concurrently** (bounded: 3 at a time within each group, both groups started together) instead of one after another — cut a full `bun fetch` from ~16-20 min to well under 2 min. `existingLinks.add()` happens synchronously right after the `.has()` check and before any `await`, which is what makes concurrent workers safe against double-claiming the same link. Concurrency surfaced a real gap: LinkedIn started returning occasional 429s under parallel load, which `fetchJobs` used to treat as end-of-results (silently truncating that search) — `fetchJobs` now retries a 429 up to 3× with backoff (1s/2s/4s) before giving up.
- **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions.
- **`daily.ts`** — the notification layer on top of `index.ts`. Snapshots the CSV's links, shells out to `bun fetch`, then diffs to find what's new. Survivors are filtered through `matchesProfile` and grouped by area (Basel / Remote / Zürich / Bern) so the message scans on a phone. `results/sent.txt` is written **only after a successful send**, so a Telegram failure retries next run instead of silently losing jobs. A non-zero exit from `bun fetch` sends a failure notice rather than failing quietly.
  - The area grouping matches against the CSV's `location` column (the posting's real location) first, falling back to `search` (the query/source label) only when `location` is empty — i.e. for rows written before the `location` column existed. This fixed two real mislabels: a LinkedIn "Bern" sweep can surface a job actually based in Lausanne (search label ≠ real location), and `fetch-other.ts` sources (job-room.ch, itjobs.ch, stellen-basel.ch, datacareer.ch) write the source name into `search` instead of a city, which used to dump their jobs into "⚪️ Other" regardless of where they actually were.
- **`profile.ts`** — `matchesProfile(title, company, minScore = 4)`. Hard rejects run first (Microsoft stack, Java/C++/SAP, life-science, non-software functions, blocked companies Jobgether and ALLPS), then `SIGNALS` scores stack and seniority terms from `cv/polo.md`. This is a **second, stricter gate than the fetch filters**: `fetch.ts` decides what lands in the CSV, `profile.ts` decides what's worth a phone buzz. Widen `SOFTWARE_TITLE_RE` to catch more in the CSV; tune `SIGNALS`/`minScore` to change what gets sent.
- **`telegram.ts`** — reads `.env` directly (no dotenv dep). Accepts `TELEGRAM_BOT` or `TELEGRAM_TOKEN` for the token. Splits at 3900 chars on line boundaries (Telegram hard-caps at 4096) and sleeps 400 ms between parts to stay under the rate limit. Messages are HTML — run titles/companies through `escapeHtml`.

### Filter pipeline (applied in order)

1. **Switzerland location guard** (card HTML) — extracts `job-search-card__location` from each card; drops jobs where location doesn't contain "Switzerland". Prevents LinkedIn from returning global results when geoId is ignored on unauthenticated requests.
2. **`NON_ENGLISH_RE`** (title) — drops German gender markers, umlauts, `/in` suffix, French accents/markers. Runs inside `fetchJobs`.
3. **`JUNIOR_TITLE_RE`** (title) — drops junior/intern/trainee/werkstudent. Runs inside `fetchJobs`.
4. **`JAVA_TITLE_RE`** (title) — drops Java-titled roles. Runs inside `fetchJobs`.
5. **`BIO_DOMAIN_RE`** (title) — drops roles whose *domain* is biology/chemistry/the lab bench (bioinformatics, biostatistics, GMP/MSAT, genomics, pathology, drug discovery, bench Scientist titles). Filters on the **role, not the employer** — software/platform/data jobs at Roche, Novartis, Swiss TPH and sciCORE stay in scope. Runs inside `fetchJobs`.
6. **`SOFTWARE_TITLE_RE`** (title) — requires a relevant tech/role term. Skipped when `raw: true`. Runs inside `fetchJobs`.
6.5. **`GERMAN_TITLE_RE`** (title) — drops "(German-speaking)" / "deutschsprachig" titles without a description fetch. Runs inside `fetchJobs`.
7. **`isViableContent`** / **`isViableUrl`** (description page) — one fetch, two checks: (a) counts DE/FR indicator words, drops if ≥ 6; (b) `requiresGerman()` drops English postings that demand German above B1. Runs in `index.ts` only (not `recheck.ts`). Adds 350 ms delay per job. `isEnglishContent` / `isEnglishUrl` remain exported for `recheck.ts` and temp scripts.

**`requiresGerman(text)`** is deliberately more than a keyword match — Rafael is German B1 (TELC), so B1/"basic"/"a plus" is fine and B2+ is not. For each mention of German it takes the *nearest* qualifying marker, and a marker is ignored when another language name sits between it and the mention. Both rules come from real false results:
- *"Infrastructure skills are **a plus** … Good knowledge of **German**"* (Helvetia) — presence-based matching let an unrelated "a plus" 95 chars away cancel a real requirement.
- *"Fluent in **English**; **German** and/or French advantageous"* (albedis) — "Fluent" is nearer to "German" than "advantageous" is, but qualifies English.

Deduplication: job IDs are extracted from `results/jobs.csv` on startup; any ID already seen is skipped before content-checking.

### SearchConfig

`url` overrides all filter params. Otherwise build from: `keywords`, `geoId`/`location`, `workType`, `jobType`, `timePosted`, `seniority`.

Known Swiss geoIds:
- Basel city: `103383283`
- Basel Metropolitan Area: `90009885`
- Basel-Country: `107117530`
- Zürich city: `102436504`
- Zürich Metropolitan Area: `107814425`
- Zürich Area: `90009888`

Use `location: "City, Switzerland"` (no `geoId`) when the geoId is unknown — LinkedIn resolves by name.

**Auth warning:** Do NOT send the `li_at` cookie in job search requests (`/jobs/search/`). With the cookie, LinkedIn serves the React SPA instead of server-rendered HTML, breaking the card parser entirely. The cookie is only useful for individual profile pages.

## Searching any city / mode temporarily

Write a throwaway script in `scripts/` (e.g. `scripts/bern-temp.ts`) — **never touch `config.ts`** unless the user explicitly asks to save the change. Delete temp scripts after the session unless the user asks to keep them.

Common patterns:

```ts
// City search (onsite + hybrid)
import { KEYWORDS } from "../config.ts";
import { fetchJobs } from "../fetch.ts";
import type { SearchConfig } from "../config.ts";

const config: SearchConfig = {
  label: "Bern", keywords: KEYWORDS, location: "Bern, Switzerland",
  workType: ["onsite", "hybrid"], timePosted: "month",
};
const jobs = await fetchJobs(config);
for (const j of jobs) console.log(j.title, j.company, j.link);
```

```ts
// Remote-only across all Switzerland
const config: SearchConfig = {
  label: "Switzerland Remote", keywords: KEYWORDS, location: "Switzerland",
  workType: ["remote"], timePosted: "month",
};
```

```ts
// Description-level keyword check (e.g. Ruby/Rails)
const RUBY_RE = /\bruby\b|\brails\b/i;
// fetch each job page, test RUBY_RE against the description text
```

Extra filters for temp scripts (import from `../fetch.ts`):
- `JAVA_TITLE_RE` — drop Java roles
- `JUNIOR_TITLE_RE` — drop junior/intern
- `isEnglishContent(id)` — drop DE/FR description content
- Add your own inline RE for EMEA, .NET, etc.

## Ranking jobs against the CV

Read `cv/polo.md` for the full profile. Key signals:

- **Seniority**: Senior / Staff / Principal / Lead / Architect — weight titles accordingly
- **Core stack**: Python · TypeScript · Ruby on Rails · React · Node.js
- **Domain strengths**: platform engineering · data pipelines · AI/LLM integrations · DevOps (Docker, Kubernetes, CI/CD, AWS)
- **Background edges**: fintech/CBDC (Moedax), civic tech (Tactical Tech / GFOSS), independent consulting (ExtraPolo, 40+ projects)
- **Languages**: English (full) · German B1 TELC certified · Spanish (full) — **drop** roles requiring German above B1; `requiresGerman()` does this automatically at fetch time
- **Hard no: biological domains** — drop bioinformatics, computational biology, biostatistics, lab/bench Scientist, GMP/MSAT, clinical/assay and drug-discovery-science roles; he has no life-science background. Software roles *at* pharma companies are still strong targets
- **Hard no: Microsoft stack** — drop .NET/C#, Azure-first, Windows/SCCM/Intune, SharePoint, Dynamics, Power Platform roles entirely; don't present them even as weak fits
- **Location**: based in Basel, open to Basel / Bern / Zürich; EU citizen, no permit needed

## Output format

**Always** present jobs as a markdown table — never as a bullet list or prose: `#`, `Title`, `Company`, `Canton`, `Why it fits`, `Link`.

`Canton` is mandatory on every table, including single-city searches and the list of dropped/rejected jobs (there `Why dropped` replaces `Why it fits`). Derive it from the actual location on the posting, not from the search label — recruiters and multi-site companies often post under a different area than the job sits in. Use the two-letter code: BS, BL, ZH, BE, ZG, LU, VD, GE, TI, AG, SG, TG. For remote roles write `Remote` (add scope if known, e.g. `Remote (EU)`).

Links use just the job ID as anchor text: `[4192954199](https://www.linkedin.com/jobs/view/4192954199/)`

Some employers (Optiml, LocalBini seen repeatedly) repost the same role under a new job ID every few weeks rather than keeping one listing open. When several rows share the same title + company, show only the most recent job ID and drop the older duplicates from the table — they're the same opening, not new leads.

## Workflow: "find jobs in X city / remote / custom search"

1. Write a temp script in `scripts/` with the right `SearchConfig` (city, workType, custom keywords).
2. Apply extra inline filters as needed: EMEA, .NET, description-level keyword check.
3. Run `isEnglishContent` per job to drop DE/FR content.
4. Rank survivors against `cv/polo.md`, present as a table.
5. Delete temp scripts after the session unless asked to keep them.

## Scheduled automations

Both run as **launchd agents** — this machine has no cron at all, so don't reach for `crontab`.

| plist (`~/Library/LaunchAgents/`) | when | runs |
|---|---|---|
| `com.rafapolo.linkme-daily` | 08:00 + 18:00 | `bun run daily.ts` → `results/daily.log` |
| `ch.polo.scicore-watch` | 08:47 | `scripts/scicore-watch.sh` → `results/.scicore-watch/watch.log` |

If the Mac is asleep at the scheduled time, launchd fires the job on wake rather than skipping the day. `launchctl list | grep -E 'linkme|scicore'` to check they're loaded; `launchctl unload <plist>` to stop one.

**`scripts/scicore-watch.sh`** watches https://scicore.unibas.ch/about-scicore/open-positions/ for a software engineering role Rafael is waiting on. It extracts the "Join the team!" block (keeping each posting's `<a href>` so the alert is tappable), diffs it against `results/.scicore-watch/snapshot.txt`, and notifies **only when a newly added line matches `$WANTED` ("software")**. Every other change still updates the snapshot silently — that's what keeps the eventual real alert clean instead of tangled with months of sysadmin churn. Three consecutive fetch failures alert, so a broken watcher can't look like "no news".

Test the alert path without waiting for the page to change:

```bash
SCICORE_URL="file://$PWD/fixture.html" bash scripts/scicore-watch.sh
```

The whole extractor is a `python3 -c` program inside shell single quotes, so **it must not contain a literal apostrophe** — that's why the `<a>` quote class is built with `chr(39)` instead of written inline.

## LinkedIn session token

`.env` holds `LI_AT` (LinkedIn session cookie), plus `TELEGRAM_BOT` and `TELEGRAM_CHAT_ID` for the digest. It is gitignored.

`LI_AT` is the `li_at` cookie value. Use it to fetch authenticated LinkedIn pages via curl:

```bash
curl -s "https://www.linkedin.com/in/<slug>" \
  -H "Cookie: li_at=$(grep LI_AT .env | cut -d= -f2-)" \
  -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" \
  --compressed -o /tmp/profile.html
```

**What works:** Full HTML pages for profiles that don't 999-block (varies by profile). Extract data with Python regex on the HTML.

**What doesn't:** LinkedIn Voyager API returns 401/403 (CSRF issues). `ch.linkedin.com` slugs sometimes work when `www.linkedin.com` returns HTTP 999. Public profiles without login redirect to a 1530-byte JS stub — no useful data.

**Tip:** Profile data is embedded as escaped JSON in the HTML. Use `html.unescape()` + regex for `"headline"`, `"locationName"`, `defaultLocalizedName` fields.

## Workflow: "check if new keywords should be added"

1. Run `bun recheck`.
2. Identify false negatives — real software roles that got dropped.
3. Ignore generic frequency suggestions (`engineer`, `development`, `senior`).
4. Propose specific additions to `SOFTWARE_TITLE_RE` in `fetch.ts` or words to `NON_ENGLISH_RE`.