From 7276aded92025fe44e897cd2c060bb0e05b33313 Mon Sep 17 00:00:00 2001 From: rafapolo Date: Thu, 27 Aug 2026 09:28:58 +0200 Subject: [PATCH] docs: cover the digest layer and scheduled jobs Rewrite the README around the current pipeline: the twice-daily Telegram digest, the sciCORE watcher, Telegram setup, and the two-stage filter pipeline. CLAUDE.md gains the bun daily commands, the daily/profile/telegram architecture notes, and a scheduled-automations section covering both launchd agents. Records that fetch.ts gates what reaches the CSV while profile.ts gates what reaches the phone. --- CLAUDE.md | 37 ++++++++++- README.md | 183 +++++++++++++++++++++++++++++++++++------------------- 2 files changed, 154 insertions(+), 66 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 0b817a5..6344ab9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,6 +8,10 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co |---|---| | `bun fetch` | run all searches in `config.ts`, append new jobs to `results/jobs.csv` | | `bun recheck` | fetch in raw mode, list filtered-out jobs + word-frequency keyword suggestions | +| `bun daily` | fetch, then push new profile-matching jobs to Telegram | +| `bun daily --dry-run` | same, but print the message instead of sending | +| `bun daily --resend` | skip fetching, re-send today's jobs from the CSV (`--date=YYYY-MM-DD` to pick a day) | +| `bun run telegram.ts --setup` | print the chat ID for `.env` after messaging the bot | Temp scripts: `bun run scripts/-temp.ts` — never edit `config.ts` for one-off searches. @@ -19,7 +23,12 @@ fetch.ts # scraper, HTML parser, all filters fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs) index.ts # orchestrator: runs searches, writes CSV recheck.ts # raw-mode fetch, lists filtered-out jobs +daily.ts # digest: fetch → match against profile.ts → send to Telegram +profile.ts # title-level scoring; decides what's worth a notification +telegram.ts # Telegram delivery, chunking, --setup chat-ID discovery scripts/ # one-off search scripts; import from ../config.ts and ../fetch.ts +scripts/scicore-watch.sh # daily launchd watch on sciCORE's open-positions page +results/sent.txt # links already pushed to Telegram — never sent twice cv/polo.md # Rafael's profile — use for ranking jobs cv/ # CV files (html, pdf, odt) results/jobs.csv # output — one row per job @@ -43,6 +52,9 @@ plans/ # LinkedIn post drafts - **itjobs.ch** is swept Switzerland-wide (no `location` param) — the board is small, ~210 jobs over 7 pages, and locking it to Basel returned only ~9. It 404s one page past the end instead of serving an empty list. Each item embeds its full `description`, so the DE/FR check runs in-process via `isEnglishDescription` rather than costing a page fetch per job. Use `job_location.name` for location — it's the English form; the bare `location` field is German. - **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV. - **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions. +- **`daily.ts`** — the notification layer on top of `index.ts`. Snapshots the CSV's links, shells out to `bun fetch`, then diffs to find what's new. Survivors are filtered through `matchesProfile` and grouped by area (Basel / Remote / Zürich / Bern) so the message scans on a phone. `results/sent.txt` is written **only after a successful send**, so a Telegram failure retries next run instead of silently losing jobs. A non-zero exit from `bun fetch` sends a failure notice rather than failing quietly. +- **`profile.ts`** — `matchesProfile(title, company, minScore = 4)`. Hard rejects run first (Microsoft stack, Java/C++/SAP, life-science, non-software functions, blocked companies Jobgether and ALLPS), then `SIGNALS` scores stack and seniority terms from `cv/polo.md`. This is a **second, stricter gate than the fetch filters**: `fetch.ts` decides what lands in the CSV, `profile.ts` decides what's worth a phone buzz. Widen `SOFTWARE_TITLE_RE` to catch more in the CSV; tune `SIGNALS`/`minScore` to change what gets sent. +- **`telegram.ts`** — reads `.env` directly (no dotenv dep). Accepts `TELEGRAM_BOT` or `TELEGRAM_TOKEN` for the token. Splits at 3900 chars on line boundaries (Telegram hard-caps at 4096) and sleeps 400 ms between parts to stay under the rate limit. Messages are HTML — run titles/companies through `escapeHtml`. ### Filter pipeline (applied in order) @@ -146,9 +158,32 @@ Links use just the job ID as anchor text: `[4192954199](https://www.linkedin.com 4. Rank survivors against `cv/polo.md`, present as a table. 5. Delete temp scripts after the session unless asked to keep them. +## Scheduled automations + +Both run as **launchd agents** — this machine has no cron at all, so don't reach for `crontab`. + +| plist (`~/Library/LaunchAgents/`) | when | runs | +|---|---|---| +| `com.rafapolo.linkme-daily` | 08:00 + 18:00 | `bun run daily.ts` → `results/daily.log` | +| `ch.polo.scicore-watch` | 08:47 | `scripts/scicore-watch.sh` → `results/.scicore-watch/watch.log` | + +If the Mac is asleep at the scheduled time, launchd fires the job on wake rather than skipping the day. `launchctl list | grep -E 'linkme|scicore'` to check they're loaded; `launchctl unload ` to stop one. + +**`scripts/scicore-watch.sh`** watches https://scicore.unibas.ch/about-scicore/open-positions/ for a software engineering role Rafael is waiting on. It extracts the "Join the team!" block (keeping each posting's `` so the alert is tappable), diffs it against `results/.scicore-watch/snapshot.txt`, and notifies **only when a newly added line matches `$WANTED` ("software")**. Every other change still updates the snapshot silently — that's what keeps the eventual real alert clean instead of tangled with months of sysadmin churn. Three consecutive fetch failures alert, so a broken watcher can't look like "no news". + +Test the alert path without waiting for the page to change: + +```bash +SCICORE_URL="file://$PWD/fixture.html" bash scripts/scicore-watch.sh +``` + +The whole extractor is a `python3 -c` program inside shell single quotes, so **it must not contain a literal apostrophe** — that's why the `` quote class is built with `chr(39)` instead of written inline. + ## LinkedIn session token -`.env` contains `LI_AT` — the LinkedIn session cookie (`li_at` value). Use it to fetch authenticated LinkedIn pages via curl: +`.env` holds `LI_AT` (LinkedIn session cookie), plus `TELEGRAM_BOT` and `TELEGRAM_CHAT_ID` for the digest. It is gitignored. + +`LI_AT` is the `li_at` cookie value. Use it to fetch authenticated LinkedIn pages via curl: ```bash curl -s "https://www.linkedin.com/in/" \ diff --git a/README.md b/README.md index b3e3400..a4fe768 100644 --- a/README.md +++ b/README.md @@ -1,98 +1,151 @@ # linkme -Scrapes LinkedIn job listings into a local CSV. Run it daily, delete rows you've applied to. +A personal job-hunting pipeline for the Swiss market. It scrapes LinkedIn (plus a handful of Swiss boards), filters out everything that doesn't fit the profile in `cv/polo.md`, appends what survives to a CSV, and pushes the good ones to Telegram twice a day. -## Usage +Two things run on their own once installed: + +| automation | when | what it does | +|---|---|---| +| **daily digest** | 08:00 + 18:00 | fetch every search, push new profile-matching jobs to Telegram | +| **sciCORE watch** | 08:47 | alert if sciCORE's open-positions page lists a *software* role | + +## Quick start ```bash -bun fetch # scrape and append new jobs to results/jobs.csv -bun recheck # show jobs currently filtered out + keyword suggestions +bun fetch # scrape all searches, append new jobs to results/jobs.csv +bun daily # fetch, then Telegram the new jobs that match the profile +bun recheck # show what the filters are currently dropping + keyword suggestions ``` -No setup needed. Results are appended to `results/jobs.csv` — reruns are safe, already-seen jobs are skipped by ID. +`bun fetch` needs no setup — reruns are safe, already-seen jobs are skipped by ID. Telegram needs the two-line setup below. + +## Telegram setup + +```bash +# 1. Message @BotFather → /newbot → copy the token into .env +echo 'TELEGRAM_BOT=' >> .env + +# 2. Send your bot any message ("hi"), then discover your chat ID +bun run telegram.ts --setup # prints TELEGRAM_CHAT_ID=... — paste it into .env +``` + +Then `bun daily --dry-run` prints the digest instead of sending it, and `bun daily --resend` re-sends today's jobs from the CSV without re-fetching. + +`.env` also holds `LI_AT`, a LinkedIn session cookie used only for fetching individual *profile* pages. Never send it with a job search — with the cookie LinkedIn serves a React SPA instead of server-rendered HTML and the card parser gets nothing. `.env` is gitignored. ## Structure ``` -config.ts # search keywords and areas -fetch.ts # scraper, HTML parser, filters -fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch) -index.ts # orchestrator: runs searches, writes CSV -recheck.ts # raw-mode fetch, lists filtered-out jobs -scripts/ # one-off search scripts (temp, not in config.ts) -cv/ # polo.md (profile) + CV files -results/ # jobs.csv output -jobs/ # notes on specific job leads -plans/ # LinkedIn post drafts +config.ts # keyword families, search areas, the sweep() generator +fetch.ts # LinkedIn scraper, HTML parser, all title/content filters +fetch-other.ts # extra sources: itjobs.ch, job-room.ch, stellen-basel.ch, + # datacareer.ch (403, currently dead), Basel intl orgs +index.ts # orchestrator: run searches → dedupe → content-check → CSV +recheck.ts # raw-mode fetch: lists what the filters dropped +daily.ts # digest: fetch, match against the profile, send to Telegram +profile.ts # title-level scoring — decides what's worth a notification +telegram.ts # Telegram delivery + --setup chat-ID discovery +scripts/ # one-off searches (*-temp.ts) and scicore-watch.sh +cv/polo.md # the profile everything is ranked against +results/jobs.csv # output, one row per job +results/sent.txt # links already pushed — never sent twice ``` +## How a job gets through + +Two stages. Cheap title filters first, then one page fetch per survivor. + +**Title filters** (inside `fetchJobs`, no network cost): + +1. **Switzerland guard** — the card's location must say Switzerland. LinkedIn ignores `geoId` on unauthenticated requests and will happily return global results. +2. `NON_ENGLISH_RE` — German gender markers (`m/w/d`, `/in`), umlauts, French `H/F` and accents. +3. `JUNIOR_TITLE_RE` — junior, intern, trainee, werkstudent, entry-level. +4. `JAVA_TITLE_RE` — Java-titled roles. +5. `BIO_DOMAIN_RE` — bioinformatics, biostatistics, genomics, GMP/MSAT, bench-scientist titles. Filters on the **role, not the employer**: software and data jobs at Roche, Novartis or Swiss TPH stay in. +6. `SOFTWARE_TITLE_RE` — requires a real tech/role term. Skipped in `raw` mode, which is what `bun recheck` uses. +7. `GERMAN_TITLE_RE` — "(German-speaking)", "deutschsprachig". + +**Content check** (`index.ts` only, one fetch + 350 ms per job): `isViableContent` counts German/French indicator words and drops at ≥ 6, then `requiresGerman()` drops English postings demanding German above B1. + +That last one is deliberately more than a keyword match. For each mention of German it takes the *nearest* qualifying marker, and ignores a marker when another language name sits between it and the mention — both rules came from real false results, like *"Fluent in English; German and/or French advantageous"*, where "Fluent" sits closer to "German" than "advantageous" does but qualifies English. + +**Profile match** (`profile.ts`, digest only) scores the title against the CV's stack and seniority signals, needing 4+ points to notify. Hard rejects encode standing decisions: Microsoft stack, Java/C++/SAP, life-science domains, non-software functions, and the blocked companies Jobgether (an aggregator, not an employer) and ALLPS (postings require German despite passing the English filter). + +Jobs are deduped globally by ID against `results/jobs.csv` on startup, so nothing is ever fetched or sent twice. + ## Output -`results/jobs.csv` — one row per job: +`results/jobs.csv` — `date_found`, `posted_at`, `search`, `title`, `company`, `link`, `source`. Open it in any spreadsheet and delete rows as you apply or dismiss. -| column | description | -|---|---| -| `date_found` | date you ran the fetch | -| `posted_at` | date the job was posted | -| `search` | which search config matched it | -| `title` | job title | -| `company` | hiring company | -| `link` | canonical job URL | -| `source` | site the job came from | +## Configuring searches -Open in any spreadsheet app. Delete rows as you apply or dismiss. - -## How it works - -- **Paginated scraping** — fetches up to 6 pages per search (150 jobs) with a 400 ms delay between pages -- **Card-boundary-safe parsing** — each job card is isolated before extracting fields -- **Language filter** — drops German (`m/w/d`, umlauts, gender markers) and French (`H/F`, accents) titles -- **Software-role filter** — requires a relevant term in the title (software, developer, data, AI, backend, frontend, devops, platform, architect, tech lead, typescript, python, rails, node.js, ML, cloud, security, SRE, etc.) -- **Seniority filter** — drops junior, intern, trainee, apprentice, werkstudent, entry-level -- **Global deduplication** — IDs indexed from existing CSV on startup; already-seen jobs are never re-added -- **30-day recency** — only jobs posted in the last month - -## recheck - -`bun recheck` fetches all searches in raw mode (bypassing the software-role filter) and lists every job that would be filtered out — useful for spotting false negatives. Also prints a word-frequency table of filtered titles to help decide what to add to `SOFTWARE_TITLE_RE`. - -## Temporary searches - -Write a throwaway script in `scripts/` instead of editing `config.ts`: - -```bash -bun run scripts/bern-temp.ts -``` - -Scripts in `scripts/` import from `../config.ts` and `../fetch.ts`. - -## Configure searches - -Edit `config.ts`: +`config.ts` holds four keyword families as separate OR-strings — `KEYWORDS`, `PLATFORM_KEYWORDS`, `RESEARCH_KEYWORDS`, `FIELD_KEYWORDS`. They're kept apart on purpose: LinkedIn ranks a focused query far better than one 50-term disjunction. `sweep()` generates one search per area per family (Research skips Bern; Platform and Field add a Switzerland-wide remote pass). ```ts -export const KEYWORDS = "Software Engineer OR Data Engineer OR ..."; - export const searches: SearchConfig[] = [ { label: "Basel (city)", keywords: KEYWORDS, geoId: "103383283", - location: "Basel", workType: ["onsite", "hybrid"], timePosted: "month", }, ]; ``` +`SearchConfig` also takes `jobType`, `seniority`, and `location` (use `"City, Switzerland"` when the geoId is unknown — LinkedIn resolves by name). A `url:` field overrides all filter params, so you can paste a LinkedIn search URL directly. + Known Swiss geoIds: -- `103383283` — Basel (city) -- `90009885` — Basel Metropolitan Area -- `107117530` — Basel-Country -- `102436504` — Zürich (city) -- `107814425` — Zürich Metropolitan Area -- `90009888` — Zürich Area -You can also paste any LinkedIn search URL directly as `url:` instead of building from filters. +| area | geoId | +|---|---| +| Basel (city) | `103383283` | +| Basel Metropolitan Area | `90009885` | +| Basel-Country | `107117530` | +| Zürich (city) | `102436504` | +| Zürich Metropolitan Area | `107814425` | +| Zürich Area | `90009888` | -`SearchConfig` also supports `jobType` (full-time, part-time, contract, temporary, internship) and `seniority` (internship, entry, associate, mid-senior, director, executive). +## One-off searches + +Don't edit `config.ts` for a single search — write a throwaway in `scripts/`: + +```ts +import { KEYWORDS } from "../config.ts"; +import { fetchJobs } from "../fetch.ts"; + +const jobs = await fetchJobs({ + label: "Luzern", keywords: KEYWORDS, + location: "Luzern, Switzerland", + workType: ["onsite", "hybrid"], timePosted: "month", +}); +for (const j of jobs) console.log(j.title, j.company, j.link); +``` + +```bash +bun run scripts/luzern-temp.ts +``` + +`*-temp.ts` scripts are disposable — delete them when the search is done. + +## Scheduling + +Both automations are launchd agents, not cron — this machine has no cron at all. + +```bash +launchctl list | grep -E 'linkme|scicore' # are they loaded? +launchctl unload ~/Library/LaunchAgents/com.rafapolo.linkme-daily.plist # stop the digest +launchctl unload ~/Library/LaunchAgents/ch.polo.scicore-watch.plist # stop the watcher +``` + +If the Mac is asleep at the scheduled time, launchd runs the job once it wakes rather than skipping the day. + +Logs: `results/daily.log` for the digest, `results/.scicore-watch/watch.log` for the watcher. The watcher logs every run — `no change`, `change ignored`, or the diff it sent — so silence on your phone can always be told apart from a broken script. Test its alert path without waiting for the real page to change: + +```bash +SCICORE_URL="file://$PWD/fixture.html" bash scripts/scicore-watch.sh +``` + +## recheck + +`bun recheck` re-runs every search with `SOFTWARE_TITLE_RE` bypassed and lists each job the filters would drop, plus a word-frequency table of those titles. Use it to spot false negatives; ignore the generic suggestions (`engineer`, `development`, `senior`) and look for real role terms worth adding to the regex.