docs: cover the digest layer and scheduled jobs
Rewrite the README around the current pipeline: the twice-daily Telegram digest, the sciCORE watcher, Telegram setup, and the two-stage filter pipeline. CLAUDE.md gains the bun daily commands, the daily/profile/telegram architecture notes, and a scheduled-automations section covering both launchd agents. Records that fetch.ts gates what reaches the CSV while profile.ts gates what reaches the phone.
This commit is contained in:
@@ -1,98 +1,151 @@
|
||||
# linkme
|
||||
|
||||
Scrapes LinkedIn job listings into a local CSV. Run it daily, delete rows you've applied to.
|
||||
A personal job-hunting pipeline for the Swiss market. It scrapes LinkedIn (plus a handful of Swiss boards), filters out everything that doesn't fit the profile in `cv/polo.md`, appends what survives to a CSV, and pushes the good ones to Telegram twice a day.
|
||||
|
||||
## Usage
|
||||
Two things run on their own once installed:
|
||||
|
||||
| automation | when | what it does |
|
||||
|---|---|---|
|
||||
| **daily digest** | 08:00 + 18:00 | fetch every search, push new profile-matching jobs to Telegram |
|
||||
| **sciCORE watch** | 08:47 | alert if sciCORE's open-positions page lists a *software* role |
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
bun fetch # scrape and append new jobs to results/jobs.csv
|
||||
bun recheck # show jobs currently filtered out + keyword suggestions
|
||||
bun fetch # scrape all searches, append new jobs to results/jobs.csv
|
||||
bun daily # fetch, then Telegram the new jobs that match the profile
|
||||
bun recheck # show what the filters are currently dropping + keyword suggestions
|
||||
```
|
||||
|
||||
No setup needed. Results are appended to `results/jobs.csv` — reruns are safe, already-seen jobs are skipped by ID.
|
||||
`bun fetch` needs no setup — reruns are safe, already-seen jobs are skipped by ID. Telegram needs the two-line setup below.
|
||||
|
||||
## Telegram setup
|
||||
|
||||
```bash
|
||||
# 1. Message @BotFather → /newbot → copy the token into .env
|
||||
echo 'TELEGRAM_BOT=<token>' >> .env
|
||||
|
||||
# 2. Send your bot any message ("hi"), then discover your chat ID
|
||||
bun run telegram.ts --setup # prints TELEGRAM_CHAT_ID=... — paste it into .env
|
||||
```
|
||||
|
||||
Then `bun daily --dry-run` prints the digest instead of sending it, and `bun daily --resend` re-sends today's jobs from the CSV without re-fetching.
|
||||
|
||||
`.env` also holds `LI_AT`, a LinkedIn session cookie used only for fetching individual *profile* pages. Never send it with a job search — with the cookie LinkedIn serves a React SPA instead of server-rendered HTML and the card parser gets nothing. `.env` is gitignored.
|
||||
|
||||
## Structure
|
||||
|
||||
```
|
||||
config.ts # search keywords and areas
|
||||
fetch.ts # scraper, HTML parser, filters
|
||||
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch)
|
||||
index.ts # orchestrator: runs searches, writes CSV
|
||||
recheck.ts # raw-mode fetch, lists filtered-out jobs
|
||||
scripts/ # one-off search scripts (temp, not in config.ts)
|
||||
cv/ # polo.md (profile) + CV files
|
||||
results/ # jobs.csv output
|
||||
jobs/ # notes on specific job leads
|
||||
plans/ # LinkedIn post drafts
|
||||
config.ts # keyword families, search areas, the sweep() generator
|
||||
fetch.ts # LinkedIn scraper, HTML parser, all title/content filters
|
||||
fetch-other.ts # extra sources: itjobs.ch, job-room.ch, stellen-basel.ch,
|
||||
# datacareer.ch (403, currently dead), Basel intl orgs
|
||||
index.ts # orchestrator: run searches → dedupe → content-check → CSV
|
||||
recheck.ts # raw-mode fetch: lists what the filters dropped
|
||||
daily.ts # digest: fetch, match against the profile, send to Telegram
|
||||
profile.ts # title-level scoring — decides what's worth a notification
|
||||
telegram.ts # Telegram delivery + --setup chat-ID discovery
|
||||
scripts/ # one-off searches (*-temp.ts) and scicore-watch.sh
|
||||
cv/polo.md # the profile everything is ranked against
|
||||
results/jobs.csv # output, one row per job
|
||||
results/sent.txt # links already pushed — never sent twice
|
||||
```
|
||||
|
||||
## How a job gets through
|
||||
|
||||
Two stages. Cheap title filters first, then one page fetch per survivor.
|
||||
|
||||
**Title filters** (inside `fetchJobs`, no network cost):
|
||||
|
||||
1. **Switzerland guard** — the card's location must say Switzerland. LinkedIn ignores `geoId` on unauthenticated requests and will happily return global results.
|
||||
2. `NON_ENGLISH_RE` — German gender markers (`m/w/d`, `/in`), umlauts, French `H/F` and accents.
|
||||
3. `JUNIOR_TITLE_RE` — junior, intern, trainee, werkstudent, entry-level.
|
||||
4. `JAVA_TITLE_RE` — Java-titled roles.
|
||||
5. `BIO_DOMAIN_RE` — bioinformatics, biostatistics, genomics, GMP/MSAT, bench-scientist titles. Filters on the **role, not the employer**: software and data jobs at Roche, Novartis or Swiss TPH stay in.
|
||||
6. `SOFTWARE_TITLE_RE` — requires a real tech/role term. Skipped in `raw` mode, which is what `bun recheck` uses.
|
||||
7. `GERMAN_TITLE_RE` — "(German-speaking)", "deutschsprachig".
|
||||
|
||||
**Content check** (`index.ts` only, one fetch + 350 ms per job): `isViableContent` counts German/French indicator words and drops at ≥ 6, then `requiresGerman()` drops English postings demanding German above B1.
|
||||
|
||||
That last one is deliberately more than a keyword match. For each mention of German it takes the *nearest* qualifying marker, and ignores a marker when another language name sits between it and the mention — both rules came from real false results, like *"Fluent in English; German and/or French advantageous"*, where "Fluent" sits closer to "German" than "advantageous" does but qualifies English.
|
||||
|
||||
**Profile match** (`profile.ts`, digest only) scores the title against the CV's stack and seniority signals, needing 4+ points to notify. Hard rejects encode standing decisions: Microsoft stack, Java/C++/SAP, life-science domains, non-software functions, and the blocked companies Jobgether (an aggregator, not an employer) and ALLPS (postings require German despite passing the English filter).
|
||||
|
||||
Jobs are deduped globally by ID against `results/jobs.csv` on startup, so nothing is ever fetched or sent twice.
|
||||
|
||||
## Output
|
||||
|
||||
`results/jobs.csv` — one row per job:
|
||||
`results/jobs.csv` — `date_found`, `posted_at`, `search`, `title`, `company`, `link`, `source`. Open it in any spreadsheet and delete rows as you apply or dismiss.
|
||||
|
||||
| column | description |
|
||||
|---|---|
|
||||
| `date_found` | date you ran the fetch |
|
||||
| `posted_at` | date the job was posted |
|
||||
| `search` | which search config matched it |
|
||||
| `title` | job title |
|
||||
| `company` | hiring company |
|
||||
| `link` | canonical job URL |
|
||||
| `source` | site the job came from |
|
||||
## Configuring searches
|
||||
|
||||
Open in any spreadsheet app. Delete rows as you apply or dismiss.
|
||||
|
||||
## How it works
|
||||
|
||||
- **Paginated scraping** — fetches up to 6 pages per search (150 jobs) with a 400 ms delay between pages
|
||||
- **Card-boundary-safe parsing** — each job card is isolated before extracting fields
|
||||
- **Language filter** — drops German (`m/w/d`, umlauts, gender markers) and French (`H/F`, accents) titles
|
||||
- **Software-role filter** — requires a relevant term in the title (software, developer, data, AI, backend, frontend, devops, platform, architect, tech lead, typescript, python, rails, node.js, ML, cloud, security, SRE, etc.)
|
||||
- **Seniority filter** — drops junior, intern, trainee, apprentice, werkstudent, entry-level
|
||||
- **Global deduplication** — IDs indexed from existing CSV on startup; already-seen jobs are never re-added
|
||||
- **30-day recency** — only jobs posted in the last month
|
||||
|
||||
## recheck
|
||||
|
||||
`bun recheck` fetches all searches in raw mode (bypassing the software-role filter) and lists every job that would be filtered out — useful for spotting false negatives. Also prints a word-frequency table of filtered titles to help decide what to add to `SOFTWARE_TITLE_RE`.
|
||||
|
||||
## Temporary searches
|
||||
|
||||
Write a throwaway script in `scripts/` instead of editing `config.ts`:
|
||||
|
||||
```bash
|
||||
bun run scripts/bern-temp.ts
|
||||
```
|
||||
|
||||
Scripts in `scripts/` import from `../config.ts` and `../fetch.ts`.
|
||||
|
||||
## Configure searches
|
||||
|
||||
Edit `config.ts`:
|
||||
`config.ts` holds four keyword families as separate OR-strings — `KEYWORDS`, `PLATFORM_KEYWORDS`, `RESEARCH_KEYWORDS`, `FIELD_KEYWORDS`. They're kept apart on purpose: LinkedIn ranks a focused query far better than one 50-term disjunction. `sweep()` generates one search per area per family (Research skips Bern; Platform and Field add a Switzerland-wide remote pass).
|
||||
|
||||
```ts
|
||||
export const KEYWORDS = "Software Engineer OR Data Engineer OR ...";
|
||||
|
||||
export const searches: SearchConfig[] = [
|
||||
{
|
||||
label: "Basel (city)",
|
||||
keywords: KEYWORDS,
|
||||
geoId: "103383283",
|
||||
location: "Basel",
|
||||
workType: ["onsite", "hybrid"],
|
||||
timePosted: "month",
|
||||
},
|
||||
];
|
||||
```
|
||||
|
||||
`SearchConfig` also takes `jobType`, `seniority`, and `location` (use `"City, Switzerland"` when the geoId is unknown — LinkedIn resolves by name). A `url:` field overrides all filter params, so you can paste a LinkedIn search URL directly.
|
||||
|
||||
Known Swiss geoIds:
|
||||
- `103383283` — Basel (city)
|
||||
- `90009885` — Basel Metropolitan Area
|
||||
- `107117530` — Basel-Country
|
||||
- `102436504` — Zürich (city)
|
||||
- `107814425` — Zürich Metropolitan Area
|
||||
- `90009888` — Zürich Area
|
||||
|
||||
You can also paste any LinkedIn search URL directly as `url:` instead of building from filters.
|
||||
| area | geoId |
|
||||
|---|---|
|
||||
| Basel (city) | `103383283` |
|
||||
| Basel Metropolitan Area | `90009885` |
|
||||
| Basel-Country | `107117530` |
|
||||
| Zürich (city) | `102436504` |
|
||||
| Zürich Metropolitan Area | `107814425` |
|
||||
| Zürich Area | `90009888` |
|
||||
|
||||
`SearchConfig` also supports `jobType` (full-time, part-time, contract, temporary, internship) and `seniority` (internship, entry, associate, mid-senior, director, executive).
|
||||
## One-off searches
|
||||
|
||||
Don't edit `config.ts` for a single search — write a throwaway in `scripts/`:
|
||||
|
||||
```ts
|
||||
import { KEYWORDS } from "../config.ts";
|
||||
import { fetchJobs } from "../fetch.ts";
|
||||
|
||||
const jobs = await fetchJobs({
|
||||
label: "Luzern", keywords: KEYWORDS,
|
||||
location: "Luzern, Switzerland",
|
||||
workType: ["onsite", "hybrid"], timePosted: "month",
|
||||
});
|
||||
for (const j of jobs) console.log(j.title, j.company, j.link);
|
||||
```
|
||||
|
||||
```bash
|
||||
bun run scripts/luzern-temp.ts
|
||||
```
|
||||
|
||||
`*-temp.ts` scripts are disposable — delete them when the search is done.
|
||||
|
||||
## Scheduling
|
||||
|
||||
Both automations are launchd agents, not cron — this machine has no cron at all.
|
||||
|
||||
```bash
|
||||
launchctl list | grep -E 'linkme|scicore' # are they loaded?
|
||||
launchctl unload ~/Library/LaunchAgents/com.rafapolo.linkme-daily.plist # stop the digest
|
||||
launchctl unload ~/Library/LaunchAgents/ch.polo.scicore-watch.plist # stop the watcher
|
||||
```
|
||||
|
||||
If the Mac is asleep at the scheduled time, launchd runs the job once it wakes rather than skipping the day.
|
||||
|
||||
Logs: `results/daily.log` for the digest, `results/.scicore-watch/watch.log` for the watcher. The watcher logs every run — `no change`, `change ignored`, or the diff it sent — so silence on your phone can always be told apart from a broken script. Test its alert path without waiting for the real page to change:
|
||||
|
||||
```bash
|
||||
SCICORE_URL="file://$PWD/fixture.html" bash scripts/scicore-watch.sh
|
||||
```
|
||||
|
||||
## recheck
|
||||
|
||||
`bun recheck` re-runs every search with `SOFTWARE_TITLE_RE` bypassed and lists each job the filters would drop, plus a word-frequency table of those titles. Use it to spot false negatives; ignore the generic suggestions (`engineer`, `development`, `senior`) and look for real role terms worth adding to the regex.
|
||||
|
||||
Reference in New Issue
Block a user