docs: cover the digest layer and scheduled jobs
Rewrite the README around the current pipeline: the twice-daily Telegram digest, the sciCORE watcher, Telegram setup, and the two-stage filter pipeline. CLAUDE.md gains the bun daily commands, the daily/profile/telegram architecture notes, and a scheduled-automations section covering both launchd agents. Records that fetch.ts gates what reaches the CSV while profile.ts gates what reaches the phone.
This commit is contained in:
@@ -8,6 +8,10 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
|---|---|
|
||||
| `bun fetch` | run all searches in `config.ts`, append new jobs to `results/jobs.csv` |
|
||||
| `bun recheck` | fetch in raw mode, list filtered-out jobs + word-frequency keyword suggestions |
|
||||
| `bun daily` | fetch, then push new profile-matching jobs to Telegram |
|
||||
| `bun daily --dry-run` | same, but print the message instead of sending |
|
||||
| `bun daily --resend` | skip fetching, re-send today's jobs from the CSV (`--date=YYYY-MM-DD` to pick a day) |
|
||||
| `bun run telegram.ts --setup` | print the chat ID for `.env` after messaging the bot |
|
||||
|
||||
Temp scripts: `bun run scripts/<name>-temp.ts` — never edit `config.ts` for one-off searches.
|
||||
|
||||
@@ -19,7 +23,12 @@ fetch.ts # scraper, HTML parser, all filters
|
||||
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs)
|
||||
index.ts # orchestrator: runs searches, writes CSV
|
||||
recheck.ts # raw-mode fetch, lists filtered-out jobs
|
||||
daily.ts # digest: fetch → match against profile.ts → send to Telegram
|
||||
profile.ts # title-level scoring; decides what's worth a notification
|
||||
telegram.ts # Telegram delivery, chunking, --setup chat-ID discovery
|
||||
scripts/ # one-off search scripts; import from ../config.ts and ../fetch.ts
|
||||
scripts/scicore-watch.sh # daily launchd watch on sciCORE's open-positions page
|
||||
results/sent.txt # links already pushed to Telegram — never sent twice
|
||||
cv/polo.md # Rafael's profile — use for ranking jobs
|
||||
cv/ # CV files (html, pdf, odt)
|
||||
results/jobs.csv # output — one row per job
|
||||
@@ -43,6 +52,9 @@ plans/ # LinkedIn post drafts
|
||||
- **itjobs.ch** is swept Switzerland-wide (no `location` param) — the board is small, ~210 jobs over 7 pages, and locking it to Basel returned only ~9. It 404s one page past the end instead of serving an empty list. Each item embeds its full `description`, so the DE/FR check runs in-process via `isEnglishDescription` rather than costing a page fetch per job. Use `job_location.name` for location — it's the English form; the bare `location` field is German.
|
||||
- **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV.
|
||||
- **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions.
|
||||
- **`daily.ts`** — the notification layer on top of `index.ts`. Snapshots the CSV's links, shells out to `bun fetch`, then diffs to find what's new. Survivors are filtered through `matchesProfile` and grouped by area (Basel / Remote / Zürich / Bern) so the message scans on a phone. `results/sent.txt` is written **only after a successful send**, so a Telegram failure retries next run instead of silently losing jobs. A non-zero exit from `bun fetch` sends a failure notice rather than failing quietly.
|
||||
- **`profile.ts`** — `matchesProfile(title, company, minScore = 4)`. Hard rejects run first (Microsoft stack, Java/C++/SAP, life-science, non-software functions, blocked companies Jobgether and ALLPS), then `SIGNALS` scores stack and seniority terms from `cv/polo.md`. This is a **second, stricter gate than the fetch filters**: `fetch.ts` decides what lands in the CSV, `profile.ts` decides what's worth a phone buzz. Widen `SOFTWARE_TITLE_RE` to catch more in the CSV; tune `SIGNALS`/`minScore` to change what gets sent.
|
||||
- **`telegram.ts`** — reads `.env` directly (no dotenv dep). Accepts `TELEGRAM_BOT` or `TELEGRAM_TOKEN` for the token. Splits at 3900 chars on line boundaries (Telegram hard-caps at 4096) and sleeps 400 ms between parts to stay under the rate limit. Messages are HTML — run titles/companies through `escapeHtml`.
|
||||
|
||||
### Filter pipeline (applied in order)
|
||||
|
||||
@@ -146,9 +158,32 @@ Links use just the job ID as anchor text: `[4192954199](https://www.linkedin.com
|
||||
4. Rank survivors against `cv/polo.md`, present as a table.
|
||||
5. Delete temp scripts after the session unless asked to keep them.
|
||||
|
||||
## Scheduled automations
|
||||
|
||||
Both run as **launchd agents** — this machine has no cron at all, so don't reach for `crontab`.
|
||||
|
||||
| plist (`~/Library/LaunchAgents/`) | when | runs |
|
||||
|---|---|---|
|
||||
| `com.rafapolo.linkme-daily` | 08:00 + 18:00 | `bun run daily.ts` → `results/daily.log` |
|
||||
| `ch.polo.scicore-watch` | 08:47 | `scripts/scicore-watch.sh` → `results/.scicore-watch/watch.log` |
|
||||
|
||||
If the Mac is asleep at the scheduled time, launchd fires the job on wake rather than skipping the day. `launchctl list | grep -E 'linkme|scicore'` to check they're loaded; `launchctl unload <plist>` to stop one.
|
||||
|
||||
**`scripts/scicore-watch.sh`** watches https://scicore.unibas.ch/about-scicore/open-positions/ for a software engineering role Rafael is waiting on. It extracts the "Join the team!" block (keeping each posting's `<a href>` so the alert is tappable), diffs it against `results/.scicore-watch/snapshot.txt`, and notifies **only when a newly added line matches `$WANTED` ("software")**. Every other change still updates the snapshot silently — that's what keeps the eventual real alert clean instead of tangled with months of sysadmin churn. Three consecutive fetch failures alert, so a broken watcher can't look like "no news".
|
||||
|
||||
Test the alert path without waiting for the page to change:
|
||||
|
||||
```bash
|
||||
SCICORE_URL="file://$PWD/fixture.html" bash scripts/scicore-watch.sh
|
||||
```
|
||||
|
||||
The whole extractor is a `python3 -c` program inside shell single quotes, so **it must not contain a literal apostrophe** — that's why the `<a>` quote class is built with `chr(39)` instead of written inline.
|
||||
|
||||
## LinkedIn session token
|
||||
|
||||
`.env` contains `LI_AT` — the LinkedIn session cookie (`li_at` value). Use it to fetch authenticated LinkedIn pages via curl:
|
||||
`.env` holds `LI_AT` (LinkedIn session cookie), plus `TELEGRAM_BOT` and `TELEGRAM_CHAT_ID` for the digest. It is gitignored.
|
||||
|
||||
`LI_AT` is the `li_at` cookie value. Use it to fetch authenticated LinkedIn pages via curl:
|
||||
|
||||
```bash
|
||||
curl -s "https://www.linkedin.com/in/<slug>" \
|
||||
|
||||
Reference in New Issue
Block a user