docs: document all eight sources in the readme
The readme listed the extra sources inline in the structure block and still described datacareer.ch as "403, currently dead". Replace that with a Sources table covering LinkedIn plus the eight boards in fetch-other.ts, with scope and the parsing quirk that matters for each. Write up the two that were dead: datacareer.ch's inverted User-Agent rule, and basel-jobs.ch's DataDome cookie — what gets through, what doesn't, how to mint, and why a launchd run degrades instead of failing. Also correct two things that had drifted: requiresGerman now covers coordination, German-language phrasing and spec-sheet language lines, and the CSV gained a location column that the Output section never mentioned.
This commit is contained in:
@@ -15,6 +15,8 @@ Two things run on their own once installed:
|
|||||||
bun fetch # scrape all searches, append new jobs to results/jobs.csv
|
bun fetch # scrape all searches, append new jobs to results/jobs.csv
|
||||||
bun daily # fetch, then Telegram the new jobs that match the profile
|
bun daily # fetch, then Telegram the new jobs that match the profile
|
||||||
bun recheck # show what the filters are currently dropping + keyword suggestions
|
bun recheck # show what the filters are currently dropping + keyword suggestions
|
||||||
|
|
||||||
|
bun run mint-datadome # refresh the basel-jobs.ch cookie (opens a Chrome window)
|
||||||
```
|
```
|
||||||
|
|
||||||
`bun fetch` needs no setup — reruns are safe, already-seen jobs are skipped by ID, and every search/source runs concurrently (a full run is well under 2 minutes). Telegram needs the two-line setup below.
|
`bun fetch` needs no setup — reruns are safe, already-seen jobs are skipped by ID, and every search/source runs concurrently (a full run is well under 2 minutes). Telegram needs the two-line setup below.
|
||||||
@@ -38,9 +40,7 @@ Then `bun daily --dry-run` prints the digest instead of sending it, and `bun dai
|
|||||||
```
|
```
|
||||||
config.ts # keyword families, search areas, the sweep() generator
|
config.ts # keyword families, search areas, the sweep() generator
|
||||||
fetch.ts # LinkedIn scraper, HTML parser, all title/content filters
|
fetch.ts # LinkedIn scraper, HTML parser, all title/content filters
|
||||||
fetch-other.ts # extra sources: itjobs.ch, job-room.ch, stellen-basel.ch,
|
fetch-other.ts # the eight non-LinkedIn sources (see Sources below)
|
||||||
# jobbasel.ch, englishjobsearch.ch, Basel intl orgs,
|
|
||||||
# datacareer.ch (403, currently dead)
|
|
||||||
index.ts # orchestrator: run searches → dedupe → content-check → CSV
|
index.ts # orchestrator: run searches → dedupe → content-check → CSV
|
||||||
recheck.ts # raw-mode fetch: lists what the filters dropped
|
recheck.ts # raw-mode fetch: lists what the filters dropped
|
||||||
daily.ts # digest: fetch, match against the profile, send to Telegram
|
daily.ts # digest: fetch, match against the profile, send to Telegram
|
||||||
@@ -52,6 +52,36 @@ results/jobs.csv # output, one row per job
|
|||||||
results/sent.txt # links already pushed — never sent twice
|
results/sent.txt # links already pushed — never sent twice
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Sources
|
||||||
|
|
||||||
|
LinkedIn is swept by `config.ts`'s search families. Everything else lives in `fetch-other.ts` and runs concurrently with it — a full `bun fetch` stays under two minutes.
|
||||||
|
|
||||||
|
| source | scope | notes |
|
||||||
|
|---|---|---|
|
||||||
|
| **LinkedIn** | 23 searches | keyword families × areas, generated by `sweep()` |
|
||||||
|
| **job-room.ch** | Switzerland | the federal employment service board (RAV); by far the largest feed |
|
||||||
|
| **basel-jobs.ch** | Basel region | behind DataDome — needs a cookie, see below |
|
||||||
|
| **jobbasel.ch** | Basel region | React SPA, but `/job/suche-{slug}` is server-rendered with a `window.__PRELOADED_STATE__` JSON blob |
|
||||||
|
| **englishjobsearch.ch** | BS + BL cantons | any page accepts `?format=markdown` and returns clean server-rendered markdown |
|
||||||
|
| **itjobs.ch** | Switzerland | small (~210 jobs); locking it to Basel returned only ~9, so it's swept nationally |
|
||||||
|
| **datacareer.ch** | Basel | 8 jobs on one page, no pagination |
|
||||||
|
| **stellen-basel.ch** | Basel region | WordPress REST API |
|
||||||
|
| **Basel intl orgs** | Basel | BIS (via the UNjobs mirror), Swiss TPH, Basel Institute on Governance |
|
||||||
|
|
||||||
|
Two of these were dead until recently and are worth knowing about:
|
||||||
|
|
||||||
|
**datacareer.ch** spent months returning 403 on every request. The cause was our own User-Agent — its nginx inverts the usual bot rule and blocks browser-like UAs while serving honest ones (no UA, `curl`, Googlebot). It gets `PLAIN_HEADERS` instead of the shared spoofed-Chrome ones.
|
||||||
|
|
||||||
|
**basel-jobs.ch** sits behind DataDome. Plain `fetch` gets 403 on any UA, headless Chrome over CDP gets 403, and Playwright's own `launch()` gets 403 even headful — but a real Chrome launched normally with `--remote-debugging-port` and attached to afterwards gets through. Clearance then lives entirely in a `datadome` cookie that replays fine over plain `fetch`, so a browser is only ever needed to *mint*, never to scrape:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bun run mint-datadome # opens a real Chrome window for ~10s, caches the cookie
|
||||||
|
```
|
||||||
|
|
||||||
|
The cache is `results/.datadome.json` (gitignored). Because minting needs a visible window, a launchd run can't do it — if the cookie goes stale the source logs `unavailable` and yields nothing while every other source continues, so the digest never fails over it. Set `LINKME_NO_BROWSER=1` to suppress minting entirely.
|
||||||
|
|
||||||
|
It's scraped in three passes, all of which cover different sets: the `informatik-jobs` category is a curated taxonomy rather than the whole board, and the two search modes query different indexes — `?terms=X&websearch=1` finds role words and postings filed outside IT, while the bare `?terms=X` is the only mode that matches stack terms (`react`, `frontend` and `backend` return literally zero under `websearch=1`).
|
||||||
|
|
||||||
## How a job gets through
|
## How a job gets through
|
||||||
|
|
||||||
Two stages. Cheap title filters first, then one page fetch per survivor.
|
Two stages. Cheap title filters first, then one page fetch per survivor.
|
||||||
@@ -68,7 +98,11 @@ Two stages. Cheap title filters first, then one page fetch per survivor.
|
|||||||
|
|
||||||
**Content check**: `isViableContent`/`isViableUrl` (LinkedIn, one page fetch + 350 ms per job in `index.ts`) and `isViableDescription` (`fetch-other.ts` sources that already carry the description inline, no extra fetch) all run the same two checks — DE/FR indicator words dropped at ≥ 6, then `requiresGerman()` drops English postings demanding German above B1.
|
**Content check**: `isViableContent`/`isViableUrl` (LinkedIn, one page fetch + 350 ms per job in `index.ts`) and `isViableDescription` (`fetch-other.ts` sources that already carry the description inline, no extra fetch) all run the same two checks — DE/FR indicator words dropped at ≥ 6, then `requiresGerman()` drops English postings demanding German above B1.
|
||||||
|
|
||||||
That last one is deliberately more than a keyword match. For each mention of German it takes the *nearest* qualifying marker, and ignores a marker when another language name sits between it and the mention — both rules came from real false results, like *"Fluent in English; German and/or French advantageous"*, where "Fluent" sits closer to "German" than "advantageous" does but qualifies English.
|
That last one is deliberately more than a keyword match, because the optional/required boundary is where every bug in it has been. Rafael is German B1, so *"a plus"* / *"advantageous"* / *"B1"* must survive and *"fluent"* / *"C1"* / *"sehr gutes Deutsch"* must not.
|
||||||
|
|
||||||
|
For each mention of German it takes the *nearest* qualifying marker, and ignores a marker when another language name sits between it and the mention — *"Fluent in English; German and/or French advantageous"* qualifies English, not German. But that rule has an exception: when the other language is **coordinated** with German the marker governs both, so *"Excellent communication skills in both English and German"* is correctly a hard requirement. Coordination means `and`/`und`/`sowie`/`&` — explicitly not `and/or`, since *"Very good English and/or German"* is satisfiable by English alone.
|
||||||
|
|
||||||
|
Two more cases it handles: German-language phrasing (*"Sehr gutes Deutsch in Wort und Schrift"*, which for a long time matched nothing at all), and spec-sheet lines like *"Languages: German & English"* that have no adjective to bind to — there the `Languages:` label itself counts, though still only if no optional marker sits closer.
|
||||||
|
|
||||||
**Profile match** (`profile.ts`, digest only) scores the title against the CV's stack and seniority signals, needing 4+ points to notify. Hard rejects encode standing decisions: Microsoft stack, Java/C++/SAP, life-science domains, non-software functions, and the blocked companies Jobgether (an aggregator, not an employer) and ALLPS (postings require German despite passing the English filter).
|
**Profile match** (`profile.ts`, digest only) scores the title against the CV's stack and seniority signals, needing 4+ points to notify. Hard rejects encode standing decisions: Microsoft stack, Java/C++/SAP, life-science domains, non-software functions, and the blocked companies Jobgether (an aggregator, not an employer) and ALLPS (postings require German despite passing the English filter).
|
||||||
|
|
||||||
@@ -76,7 +110,7 @@ Jobs are deduped globally by ID against `results/jobs.csv` on startup, so nothin
|
|||||||
|
|
||||||
## Output
|
## Output
|
||||||
|
|
||||||
`results/jobs.csv` — `date_found`, `posted_at`, `search`, `title`, `company`, `link`, `source`. Open it in any spreadsheet and delete rows as you apply or dismiss.
|
`results/jobs.csv` — `date_found`, `posted_at`, `search`, `title`, `company`, `link`, `source`, `location`. `search` is the query or source that found the job; `location` is the posting's own location, which is what the digest groups on (a "Bern" sweep can surface a job actually based in Lausanne). Open it in any spreadsheet and delete rows as you apply or dismiss.
|
||||||
|
|
||||||
## Configuring searches
|
## Configuring searches
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user