Compare commits

..
10 Commits
Author SHA1 Message Date
rodado 3875f13713 docs: update README sources table and drop retired sciCORE watcher references 2026-09-18 11:50:39 +02:00
rodado 288b8bdcd1 feat: add sciCORE and Swisscom as job sources
sciCORE now runs inside bun fetch/daily via fetchSciCore instead of its
own standalone launchd watcher. Swisscom's Workday career site is added
as a new source (fetchSwisscom), pulled via its public CXS JSON API.

Also fixes a false positive in requiresGerman(): Swisscom's stock line
"proficiency in German is considered an advantage" was being flagged as
requiring German because the noun "proficiency" sat closer to the
mention than the real optional marker "advantage". A local-predicate
override now lets the words immediately following "German is/are ..."
decide the verdict instead of losing to raw character distance.
2026-09-18 11:49:49 +02:00
rodado 9cbc525cf5 fix: dedupe digest against all previously-sent jobs, not just this run 2026-09-14 20:12:09 +02:00
rodado 29b00cd91e fix: dedupe the digest's matched jobs across links/providers
englishjobsearch.ch listed the same posting twice under different
canton pages, once as 'Swiss AviationSoftware' and once as 'Swiss-AS' --
different clickout links and slightly different company text for the
same opening, so it showed up twice in one digest. Matched jobs are now
deduped on normalized title+company, plus a title-only pass for titles
specific enough (5+ words) to make a same-source coincidental match
implausible.
2026-09-14 12:25:06 +02:00
rodado d401dd5f9a feat: dedupe the Ignored (German/French) digest count across runs and sources
Dropped jobs never reach the CSV, so nothing remembered them -- the same
posting got re-flagged and re-counted every single bun daily run (5x/day),
and separately every time it resurfaced under a different link/provider
(e.g. Consult & Pepper's recurring Senior Fullstack Software Engineer
reposted on job-room.ch every few days and again on LinkedIn). A persisted
results/.lang-dropped-seen.json now keys on normalized title+company so
each underlying job only counts once, ever.
2026-09-14 12:12:49 +02:00
rodado 0249800278 fix: stop letting unreadable or truncated descriptions slip past the German filter
isViableContent/isViableUrl defaulted to keep a job when its description
couldn't be fetched at all -- that's what let a fully-German HRM
Systems/Consult & Pepper posting ("In Deutsch bist du verhandlungssicher")
through on 2026-09-10. They now report retry:true instead, and index.ts
leaves the job out of the CSV so a fresh fetch judges it next run rather
than defaulting to keep.

job-room.ch's search API also returns a different keyword-anchored teaser
per hit for the same ad, and only the last one was kept -- a genuinely
German posting could slip through whenever the last keyword's teaser
happened to read as English. Every teaser variant is now collected and
checked together.
2026-09-14 12:12:43 +02:00
rodadoandClaude Sonnet 5 c7b86e06f4 fix: back off longer on 429s in description fetch
fetchDescription and isEnglishContent retried rate-limited (429) requests
after only 350/700/1050ms, too short under concurrent load, so retries kept
hitting 429, exhausted, and fell back to the "can't read it, so keep"
default. That let an all-German CLEEVEN posting (job 4465075383) through
undetected and out to Telegram. Match the 1s/2s/4s backoff already used for
listing-page 429 retries.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011zgfzuFy3bAbs8GTh9apFc
2026-09-10 12:15:58 +02:00
rodadoandClaude Sonnet 5 821bd19ff7 feat: add Switzerland-wide remote sweep for core keywords
Core KEYWORDS only ran onsite/hybrid against Basel/Bern/Zürich; a manual
remote fullstack search turned up viable postings (PAS01, xorlab, Proton,
SCOR) that none of the existing searches covered.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011zgfzuFy3bAbs8GTh9apFc
2026-09-10 12:15:46 +02:00
rodado a72a31f387 docs: adjust linkme-daily schedule to 9am/12pm/3pm/5pm/8pm 2026-09-10 10:44:03 +02:00
rodado e36bcdeb3a docs: update linkme-daily schedule to every 6h in CLAUDE.md 2026-09-10 10:42:37 +02:00
7 changed files with 344 additions and 45 deletions
+10 -15
View File
@@ -21,7 +21,7 @@ Temp scripts: `bun run scripts/<name>-temp.ts` — never edit `config.ts` for on
``` ```
config.ts # search keywords and areas config.ts # search keywords and areas
fetch.ts # scraper, HTML parser, all filters fetch.ts # scraper, HTML parser, all filters
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch) fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com)
index.ts # orchestrator: runs searches, writes CSV index.ts # orchestrator: runs searches, writes CSV
recheck.ts # raw-mode fetch, lists filtered-out jobs recheck.ts # raw-mode fetch, lists filtered-out jobs
daily.ts # digest: fetch → match against profile.ts → send to Telegram daily.ts # digest: fetch → match against profile.ts → send to Telegram
@@ -43,7 +43,7 @@ plans/ # LinkedIn post drafts
- Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform) and `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform and Field add a Switzerland-wide remote pass. - Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform) and `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform and Field add a Switzerland-wide remote pass.
- `RESEARCH_KEYWORDS` deliberately contains no bioinformatics/computational-biology terms — see `BIO_DOMAIN_RE`. - `RESEARCH_KEYWORDS` deliberately contains no bioinformatics/computational-biology terms — see `BIO_DOMAIN_RE`.
- **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`. - **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`.
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch). Imported by `index.ts`. - **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com). Imported by `index.ts`.
- **datacareer.ch** inverts the usual bot rule: its nginx **403s browser-like User-Agents** and serves honest ones (no UA, `curl/8.0`, Googlebot) fine. The spoofed Chrome UA in the shared `HEADERS` was the entire cause of the long-standing 403 — it uses `PLAIN_HEADERS` instead. Small board: Basel returns **8 jobs on one page with no pagination**, so `?page=2` legitimately yields 0 items. - **datacareer.ch** inverts the usual bot rule: its nginx **403s browser-like User-Agents** and serves honest ones (no UA, `curl/8.0`, Googlebot) fine. The spoofed Chrome UA in the shared `HEADERS` was the entire cause of the long-standing 403 — it uses `PLAIN_HEADERS` instead. Small board: Basel returns **8 jobs on one page with no pagination**, so `?page=2` legitimately yields 0 items.
- **job-room.ch** carries its language requirement as a **structured field** (`languageSkills[]`, e.g. `{languageIsoCode: "de", spokenLevel: "PROFICIENT"}`), entirely separate from the free-text `jobDescriptions[].description` — the UI renders it as "Deutsch (Fliessend)". `requiresGerman()`/`isViableDescription()` only ever scan the prose, so this field slipped through unchecked until 2026-09-04: measured against 1843 ads, 69 required PROFICIENT-level German and the prose check caught only 3 of them incidentally. `jobRoomRequiresGerman()` now checks the structured field directly (`spokenLevel`/`writtenLevel === "PROFICIENT"`) before the description-based check runs. Levels seen: `NONE`, `BASIC`, `INTERMEDIATE`, `PROFICIENT` — only `PROFICIENT` (fluent/C1+) is dropped, matching the B1 cutoff used everywhere else. - **job-room.ch** carries its language requirement as a **structured field** (`languageSkills[]`, e.g. `{languageIsoCode: "de", spokenLevel: "PROFICIENT"}`), entirely separate from the free-text `jobDescriptions[].description` — the UI renders it as "Deutsch (Fliessend)". `requiresGerman()`/`isViableDescription()` only ever scan the prose, so this field slipped through unchecked until 2026-09-04: measured against 1843 ads, 69 required PROFICIENT-level German and the prose check caught only 3 of them incidentally. `jobRoomRequiresGerman()` now checks the structured field directly (`spokenLevel`/`writtenLevel === "PROFICIENT"`) before the description-based check runs. Levels seen: `NONE`, `BASIC`, `INTERMEDIATE`, `PROFICIENT` — only `PROFICIENT` (fluent/C1+) is dropped, matching the B1 cutoff used everywhere else.
- **jobbasel.ch** looks like a JS-only React/Vite SPA (no jobs in the homepage's raw HTML), but its search route is server-rendered: `GET /job/suche-{slug}` returns full HTML with a `window.__PRELOADED_STATE__` blob containing the matched vacancies as JSON — no headless browser needed. Pagination is a URL suffix (`-seite-{n}`); a bare `?page=` query param is silently ignored. Swept with the same fixed keyword list as job-room.ch (no keyword-union endpoint exists) and deduped by vacancy id. Each item carries its full German description, so the DE/FR check runs in-process. - **jobbasel.ch** looks like a JS-only React/Vite SPA (no jobs in the homepage's raw HTML), but its search route is server-rendered: `GET /job/suche-{slug}` returns full HTML with a `window.__PRELOADED_STATE__` blob containing the matched vacancies as JSON — no headless browser needed. Pagination is a URL suffix (`-seite-{n}`); a bare `?page=` query param is silently ignored. Swept with the same fixed keyword list as job-room.ch (no keyword-union endpoint exists) and deduped by vacancy id. Each item carries its full German description, so the DE/FR check runs in-process.
@@ -82,6 +82,10 @@ plans/ # LinkedIn post drafts
- **Basel Institute on Governance** — `baselgovernance.org/about/careers`. Each vacancy is a linked PDF, so there's no HTML description to language-check. The `rounded-3xl` card class is what separates vacancy tiles from the annual-report PDF link. - **Basel Institute on Governance** — `baselgovernance.org/about/careers`. Each vacancy is a linked PDF, so there's no HTML description to language-check. The `rounded-3xl` card class is what separates vacancy tiles from the annual-report PDF link.
- These boards use UN-flavoured titles ("Information Management Officer", "ICT Specialist") that `SOFTWARE_TITLE_RE` alone drops, so `INTL_TECH_RE` widens the net **for these sources only**. Boards this small can afford the noise; missing a rare opening can't. - These boards use UN-flavoured titles ("Information Management Officer", "ICT Specialist") that `SOFTWARE_TITLE_RE` alone drops, so `INTL_TECH_RE` widens the net **for these sources only**. Boards this small can afford the noise; missing a rare opening can't.
- **itjobs.ch** is swept Switzerland-wide (no `location` param) — the board is small, ~210 jobs over 7 pages, and locking it to Basel returned only ~9. It 404s one page past the end instead of serving an empty list. Each item embeds its full `description`, so the DE/FR check runs in-process via `isEnglishDescription` rather than costing a page fetch per job. Use `job_location.name` for location — it's the English form; the bare `location` field is German. - **itjobs.ch** is swept Switzerland-wide (no `location` param) — the board is small, ~210 jobs over 7 pages, and locking it to Basel returned only ~9. It 404s one page past the end instead of serving an empty list. Each item embeds its full `description`, so the DE/FR check runs in-process via `isEnglishDescription` rather than costing a page fetch per job. Use `job_location.name` for location — it's the English form; the bare `location` field is German.
- **scicore.unibas.ch** (`fetchSciCore`) replaces the standalone `scripts/scicore-watch.sh` launchd watcher (removed 2026-09-16) — sciCORE now runs inside `bun fetch`/`bun daily` like every other source instead of its own 08:47 agent. The open-positions page is a plain-text announcement list, not a real job board: each `<li>` under `<h1>Join the team!</h1>` is either an active posting linked to `careers.unibas.ch` or a closed/in-review notice with no link — items matching `deadline has passed`/`interview process is (still) ongoing` are skipped since they're not open applications. `contentChecked: true` because the listing itself is the only English text available; there's no separate description page to fetch. Title still has to clear `SOFTWARE_TITLE_RE`, same as every other source — this is deliberately broader than the old watcher's "software"-only match (it also catches "security", matching the IT Security Engineer role Rafael prepped for — see `project_scicore_security_report` in memory) but still drops pure sysadmin titles like "Linux Systems Administrator".
- **swisscom.com** (`fetchSwisscom`, added 2026-09-18) — Swisscom's own Workday career site (`swisscom.wd103.myworkdayjobs.com/SwisscomExternalCareers`). Small board (~47 openings, Switzerland-wide), so no keyword sweep — the full listing is pulled and filtered locally, like itjobs.ch. The listing page itself is a bare JS shell with no jobs in the raw HTML; postings live behind the standard Workday CXS JSON API (`POST /wday/cxs/swisscom/SwisscomExternalCareers/jobs`). `total` on the first page (offset 0) is authoritative — later pages report `total: 0` (a tenant quirk), and an offset past the true total silently wraps around and re-serves page 1 instead of an empty list, so the loop's stop condition is the first page's total, never a later one's. Listing rows carry no description, so each title-filtered candidate costs one extra fetch against the job detail endpoint (`/job{externalPath}`), which returns the full HTML description — the DE/FR + `requiresGerman` check runs on that in-process (`contentChecked: true`).
Swisscom mostly hires in German — of 47 postings, only ~16 clear the English/software title filters, and nearly all of those turned out to be written in German too; only genuinely-English, German-optional roles survive. Adding this source surfaced a real bug in `requiresGerman()`: the stock line *"Fluency in English is required; proficiency in German is considered an advantage"* had the noun "proficiency" (a required-marker word) sitting right next to "German", closer than the real optional-marker "advantage" — nearest() picked the wrong one and wrongly flagged the posting as requiring German. Fixed by adding a local-predicate override (`GERMAN_LOCAL_PREDICATE_RE`): when a copula ("is"/"are", optionally "considered"/"deemed") directly follows the German mention, that immediate predicate decides the verdict outright instead of competing on raw character distance with markers elsewhere in the window — "German is considered an advantage" resolves as optional regardless of what else is nearby. It only fires when a copula immediately follows the mention, so *"Work proficiency in German, English, and French"* (no copula after "German", a genuine hard requirement — Senior Operation Architect (FR/GER/ENG)) still falls through to the general nearest-marker logic and is correctly caught as required. Re-check both cases (and the pre-existing Helvetia/albedis/Karakun/J. Safra Sarasin/BBF/spec-sheet cases in the paragraph above) before touching `requiresGerman` again.
- **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV. Writes both `search` (the query/source that found the job) and `location` (the posting's own location, as parsed off its card by `fetch.ts`/`fetch-other.ts` — no extra page fetch) to the CSV. Migrates old CSVs on startup: adds `source` if missing, then `location` (backfilled `""` for rows written before the column existed). - **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV. Writes both `search` (the query/source that found the job) and `location` (the posting's own location, as parsed off its card by `fetch.ts`/`fetch-other.ts` — no extra page fetch) to the CSV. Migrates old CSVs on startup: adds `source` if missing, then `location` (backfilled `""` for rows written before the column existed).
- Every LinkedIn search and every `fetch-other.ts` source hits a different host, so they run **concurrently** (bounded: 3 at a time within each group, both groups started together) instead of one after another — cut a full `bun fetch` from ~16-20 min to well under 2 min. `existingLinks.add()` happens synchronously right after the `.has()` check and before any `await`, which is what makes concurrent workers safe against double-claiming the same link. Concurrency surfaced a real gap: LinkedIn started returning occasional 429s under parallel load, which `fetchJobs` used to treat as end-of-results (silently truncating that search) — `fetchJobs` now retries a 429 up to 3× with backoff (1s/2s/4s) before giving up. - Every LinkedIn search and every `fetch-other.ts` source hits a different host, so they run **concurrently** (bounded: 3 at a time within each group, both groups started together) instead of one after another — cut a full `bun fetch` from ~16-20 min to well under 2 min. `existingLinks.add()` happens synchronously right after the `.has()` check and before any `await`, which is what makes concurrent workers safe against double-claiming the same link. Concurrency surfaced a real gap: LinkedIn started returning occasional 429s under parallel load, which `fetchJobs` used to treat as end-of-results (silently truncating that search) — `fetchJobs` now retries a 429 up to 3× with backoff (1s/2s/4s) before giving up.
- **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions. - **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions.
@@ -203,24 +207,15 @@ Some employers (Optiml, LocalBini seen repeatedly) repost the same role under a
## Scheduled automations ## Scheduled automations
Both run as **launchd agents** — this machine has no cron at all, so don't reach for `crontab`. Runs as a **launchd agent** — this machine has no cron at all, so don't reach for `crontab`.
| plist (`~/Library/LaunchAgents/`) | when | runs | | plist (`~/Library/LaunchAgents/`) | when | runs |
|---|---|---| |---|---|---|
| `com.rafapolo.linkme-daily` | 08:00 + 18:00 | `bun run daily.ts` → `results/daily.log` | | `com.rafapolo.linkme-daily` | 09:00 / 12:00 / 15:00 / 17:00 / 20:00 | `bun run daily.ts` → `results/daily.log` |
| `ch.polo.scicore-watch` | 08:47 | `scripts/scicore-watch.sh` → `results/.scicore-watch/watch.log` |
If the Mac is asleep at the scheduled time, launchd fires the job on wake rather than skipping the day. `launchctl list | grep -E 'linkme|scicore'` to check they're loaded; `launchctl unload <plist>` to stop one. If the Mac is asleep at the scheduled time, launchd fires the job on wake rather than skipping the day. `launchctl list | grep linkme` to check it's loaded; `launchctl unload <plist>` to stop it.
**`scripts/scicore-watch.sh`** watches https://scicore.unibas.ch/about-scicore/open-positions/ for a software engineering role Rafael is waiting on. It extracts the "Join the team!" block (keeping each posting's `<a href>` so the alert is tappable), diffs it against `results/.scicore-watch/snapshot.txt`, and notifies **only when a newly added line matches `$WANTED` ("software")**. Every other change still updates the snapshot silently — that's what keeps the eventual real alert clean instead of tangled with months of sysadmin churn. Three consecutive fetch failures alert, so a broken watcher can't look like "no news". **`scripts/scicore-watch.sh`** is retired (2026-09-16) — sciCORE is now swept by `fetchSciCore` in `fetch-other.ts` as part of `bun fetch`/`bun daily`, see the `fetch-other.ts` entry above. Its `ch.polo.scicore-watch` launchd agent (08:47) was unloaded and its plist removed; the script and `results/.scicore-watch/` (snapshot + log) are left in the repo for reference/history, not run anymore.
Test the alert path without waiting for the page to change:
```bash
SCICORE_URL="file://$PWD/fixture.html" bash scripts/scicore-watch.sh
```
The whole extractor is a `python3 -c` program inside shell single quotes, so **it must not contain a literal apostrophe** — that's why the `<a>` quote class is built with `chr(39)` instead of written inline.
## LinkedIn session token ## LinkedIn session token
+13 -15
View File
@@ -1,13 +1,14 @@
# linkme # linkme
A personal job-hunting pipeline for the Swiss market. It scrapes LinkedIn (plus a handful of Swiss boards), filters out everything that doesn't fit the profile in `cv/polo.md`, appends what survives to a CSV, and pushes the good ones to Telegram twice a day. A personal job-hunting pipeline for the Swiss market. It scrapes LinkedIn (plus a handful of Swiss boards), filters out everything that doesn't fit the profile in `cv/polo.md`, appends what survives to a CSV, and pushes the good ones to Telegram throughout the day.
Two things run on their own once installed: One thing runs on its own once installed:
| automation | when | what it does | | automation | when | what it does |
|---|---|---| |---|---|---|
| **daily digest** | 08:00 + 18:00 | fetch every search, push new profile-matching jobs to Telegram | | **daily digest** | 09:00 / 12:00 / 15:00 / 17:00 / 20:00 | fetch every search + source, push new profile-matching jobs to Telegram |
| **sciCORE watch** | 08:47 | alert if sciCORE's open-positions page lists a *software* role |
sciCORE used to have its own separate 08:47 watcher; it's now just another source swept inside `bun fetch`/`bun daily` (see Sources below).
## Quick start ## Quick start
@@ -40,13 +41,13 @@ Then `bun daily --dry-run` prints the digest instead of sending it, and `bun dai
``` ```
config.ts # keyword families, search areas, the sweep() generator config.ts # keyword families, search areas, the sweep() generator
fetch.ts # LinkedIn scraper, HTML parser, all title/content filters fetch.ts # LinkedIn scraper, HTML parser, all title/content filters
fetch-other.ts # the eight non-LinkedIn sources (see Sources below) fetch-other.ts # the ten non-LinkedIn sources (see Sources below)
index.ts # orchestrator: run searches → dedupe → content-check → CSV index.ts # orchestrator: run searches → dedupe → content-check → CSV
recheck.ts # raw-mode fetch: lists what the filters dropped recheck.ts # raw-mode fetch: lists what the filters dropped
daily.ts # digest: fetch, match against the profile, send to Telegram daily.ts # digest: fetch, match against the profile, send to Telegram
profile.ts # title-level scoring — decides what's worth a notification profile.ts # title-level scoring — decides what's worth a notification
telegram.ts # Telegram delivery + --setup chat-ID discovery telegram.ts # Telegram delivery + --setup chat-ID discovery
scripts/ # one-off searches (*-temp.ts) and scicore-watch.sh scripts/ # one-off searches (*-temp.ts); scicore-watch.sh kept for history, retired
cv/polo.md # the profile everything is ranked against cv/polo.md # the profile everything is ranked against
results/jobs.csv # output, one row per job results/jobs.csv # output, one row per job
results/sent.txt # links already pushed — never sent twice results/sent.txt # links already pushed — never sent twice
@@ -67,6 +68,8 @@ LinkedIn is swept by `config.ts`'s search families. Everything else lives in `fe
| **datacareer.ch** | Basel | 8 jobs on one page, no pagination | | **datacareer.ch** | Basel | 8 jobs on one page, no pagination |
| **stellen-basel.ch** | Basel region | WordPress REST API | | **stellen-basel.ch** | Basel region | WordPress REST API |
| **Basel intl orgs** | Basel | BIS (via the UNjobs mirror), Swiss TPH, Basel Institute on Governance | | **Basel intl orgs** | Basel | BIS (via the UNjobs mirror), Swiss TPH, Basel Institute on Governance |
| **scicore.unibas.ch** | Basel | sciCORE's open-positions announcement list; used to be its own separate 08:47 watcher, now just another source |
| **swisscom.com** | Switzerland | Swisscom's own Workday career site, pulled via its public CXS JSON API; small board and mostly German-language, so few survive the filters |
Two of these were dead until recently and are worth knowing about: Two of these were dead until recently and are worth knowing about:
@@ -165,21 +168,16 @@ bun run scripts/luzern-temp.ts
## Scheduling ## Scheduling
Both automations are launchd agents, not cron — this machine has no cron at all. The digest runs as a launchd agent, not cron — this machine has no cron at all.
```bash ```bash
launchctl list | grep -E 'linkme|scicore' # are they loaded? launchctl list | grep linkme # is it loaded?
launchctl unload ~/Library/LaunchAgents/com.rafapolo.linkme-daily.plist # stop the digest launchctl unload ~/Library/LaunchAgents/com.rafapolo.linkme-daily.plist # stop it
launchctl unload ~/Library/LaunchAgents/ch.polo.scicore-watch.plist # stop the watcher
``` ```
If the Mac is asleep at the scheduled time, launchd runs the job once it wakes rather than skipping the day. If the Mac is asleep at the scheduled time, launchd runs the job once it wakes rather than skipping the day.
Logs: `results/daily.log` for the digest, `results/.scicore-watch/watch.log` for the watcher. The watcher logs every run — `no change`, `change ignored`, or the diff it sent — so silence on your phone can always be told apart from a broken script. Test its alert path without waiting for the real page to change: Logs: `results/daily.log`.
```bash
SCICORE_URL="file://$PWD/fixture.html" bash scripts/scicore-watch.sh
```
## recheck ## recheck
+12
View File
@@ -168,6 +168,18 @@ export const searches: SearchConfig[] = [
workType: ["onsite", "hybrid"], workType: ["onsite", "hybrid"],
timePosted: "month", timePosted: "month",
}, },
{
// Core KEYWORDS previously only ran onsite/hybrid against the three named
// cities — a manual "Fullstack Developer" remote-CH search on 2026-09-10
// turned up 21 viable postings (PAS01, xorlab, Proton, SCOR...) that no
// existing search covered, since Platform/Field are the only families with
// a Switzerland-wide remote sweep. Adding the same sweep for core KEYWORDS.
label: "Switzerland Remote",
keywords: KEYWORDS,
location: "Switzerland",
workType: ["remote"],
timePosted: "month",
},
]; ];
// --- Role-family sweeps --------------------------------------------------- // --- Role-family sweeps ---------------------------------------------------
+49 -1
View File
@@ -62,7 +62,55 @@ const langDropped = await Bun.file(LANG_DROPPED)
.catch(() => 0); .catch(() => 0);
// Only leads that actually match the CV (profile.ts) and haven't been sent before. // Only leads that actually match the CV (profile.ts) and haven't been sent before.
const matched = fresh.filter((r) => !sent.has(r.link) && matchesProfile(r.title, r.company).ok); const preDedupe = fresh.filter((r) => !sent.has(r.link) && matchesProfile(r.title, r.company).ok);
// Sources can surface the same underlying opening twice under different
// links/providers — e.g. englishjobsearch.ch listing the same posting on two
// canton pages with the company rendered as "Swiss AviationSoftware" on one
// and "Swiss-AS" on the other. Exact title+company catches the common case
// (same employer, identical spelling); the title-only pass catches spelling
// drift too, but only for titles specific enough (5+ words) that a
// coincidental match between two genuinely different postings is implausible.
const normTitle = (title: string) =>
title
.replace(/[([][^)\]]*[)\]]/g, "") // drop "(all genders)", "(m/w/d)", "(a)", "80-100%" noise
.replace(/[^a-z0-9 ]/gi, " ")
.replace(/\s+/g, " ")
.trim()
.toLowerCase();
const normCompany = (company: string) =>
company
.replace(/\b(ag|gmbh|ltd|llc|inc|sa|se|plc|co)\b\.?/gi, "")
.replace(/[^a-z0-9]/gi, "")
.toLowerCase();
const seenExact = new Set<string>();
const seenTitleOnly = new Set<string>();
// Seed both sets from everything already sent (any past day), keyed the same
// way — not just today's batch. Without this, a job re-dedupes only against
// jobs sent in the *same run*: LinkedIn reposts the same opening under a new
// job ID every few weeks (Optiml, LocalBini, hexagonab, nicollcurtin, etc. —
// confirmed repeats in sent.txt across different days), each repost lands
// under a fresh link that passes the `sent.has(link)` check untouched, and it
// got renotified as if new. `all` (the full CSV) still has every past row, so
// each previously-sent link resolves back to a title+company to seed with.
for (const r of all) {
if (!sent.has(r.link)) continue;
const t = normTitle(r.title);
seenExact.add(`${t}|${normCompany(r.company)}`);
if (t.split(" ").filter(Boolean).length >= 5) seenTitleOnly.add(t);
}
const matched = preDedupe.filter((r) => {
const t = normTitle(r.title);
const exactKey = `${t}|${normCompany(r.company)}`;
if (seenExact.has(exactKey)) return false;
const specific = t.split(" ").filter(Boolean).length >= 5;
if (specific && seenTitleOnly.has(t)) return false;
seenExact.add(exactKey);
if (specific) seenTitleOnly.add(t);
return true;
});
console.log(`${fresh.length} new · ${matched.length} match profile · ${fresh.length - matched.length} filtered or already sent`); console.log(`${fresh.length} new · ${matched.length} match profile · ${fresh.length - matched.length} filtered or already sent`);
if (!matched.length) { if (!matched.length) {
+159 -3
View File
@@ -219,6 +219,101 @@ export async function fetchItJobsCh(): Promise<Job[]> {
return allJobs; return allJobs;
} }
// swisscom.com — Swisscom's own Workday career site. Small board (~47 openings,
// Switzerland-wide), so no keyword sweep — pull the full listing and filter
// locally, like itjobs.ch. The listing page itself is a bare JS shell (no jobs
// in the raw HTML); postings live behind the standard Workday CXS JSON API:
// POST /wday/cxs/swisscom/SwisscomExternalCareers/jobs {appliedFacets, limit, offset, searchText}
// `total` on the first page (offset 0) is authoritative — on later pages it
// comes back as 0 (a quirk of this tenant), and an offset past the true total
// silently wraps around and re-serves page 1 instead of an empty list. So the
// loop's stopping condition is the first page's total, never a later one's.
// Listing rows carry no description, so each title-filtered candidate costs
// one extra fetch against the job detail endpoint (`/job{externalPath}`),
// which returns the full HTML description — the DE/FR + requiresGerman check
// runs on that here, same as job-room.ch/jobbasel.ch.
const SWISSCOM_API_BASE = "https://swisscom.wd103.myworkdayjobs.com/wday/cxs/swisscom/SwisscomExternalCareers";
interface SwisscomListing {
title: string;
externalPath: string;
postedOn: string;
bulletFields: string[];
}
function parseSwisscomPostedOn(text: string): string {
const iso = (d: Date) => d.toISOString().slice(0, 10);
const now = new Date();
if (/posted today/i.test(text)) return iso(now);
if (/posted yesterday/i.test(text)) {
const d = new Date(now);
d.setDate(d.getDate() - 1);
return iso(d);
}
const m = text.match(/posted (\d+) days? ago/i);
if (m) {
const d = new Date(now);
d.setDate(d.getDate() - Number(m[1]));
return iso(d);
}
return ""; // "Posted 30+ Days Ago" — no exact date available
}
export async function fetchSwisscom(): Promise<Job[]> {
const listings: SwisscomListing[] = [];
let total = Infinity;
for (let offset = 0; offset < total; offset += 20) {
if (offset > 0) await Bun.sleep(400);
const res = await fetch(`${SWISSCOM_API_BASE}/jobs`, {
method: "POST",
headers: { ...HEADERS, "Content-Type": "application/json" },
body: JSON.stringify({ appliedFacets: {}, limit: 20, offset, searchText: "" }),
});
if (!res.ok) { console.log(` Swisscom … failed (${res.status})`); break; }
const data = (await res.json()) as { total: number; jobPostings: SwisscomListing[] };
if (offset === 0) total = data.total;
if (!data.jobPostings.length) break;
listings.push(...data.jobPostings);
}
const jobs: Job[] = [];
let kept = 0;
let langDropped = 0;
for (const item of listings) {
const title = decodeHtml(item.title);
if (!applyFilters(title)) continue;
if (MS_STACK_RE.test(title)) continue;
await Bun.sleep(300);
const detailRes = await fetch(`${SWISSCOM_API_BASE}${item.externalPath}`, { headers: HEADERS });
if (!detailRes.ok) continue;
const detail = (await detailRes.json()) as {
jobPostingInfo?: { location?: string; jobDescription?: string; externalUrl?: string };
};
const info = detail.jobPostingInfo;
if (!isViableDescription(info?.jobDescription).ok) { langDropped++; continue; }
jobs.push({
id: `sc-${item.bulletFields?.[0] ?? item.externalPath}`,
title,
company: "Swisscom",
location: info?.location ?? "Switzerland",
postedAt: parseSwisscomPostedOn(item.postedOn),
link: info?.externalUrl ?? `https://swisscom.wd103.myworkdayjobs.com/SwisscomExternalCareers${item.externalPath}`,
search: "swisscom.com",
source: "swisscom.com",
contentChecked: true, // isViableDescription already ran on the real description
});
kept++;
}
console.log(` Swisscom … ${listings.length} items, ${kept} kept, ${langDropped} non-English`);
return jobs;
}
// job-room.ch — the federal employment service board (RAV / Arbeitsvermittlung). // job-room.ch — the federal employment service board (RAV / Arbeitsvermittlung).
// Roles under the Stellenmeldepflicht must be posted here before anywhere else, // Roles under the Stellenmeldepflicht must be posted here before anywhere else,
// so it often carries a job days ahead of LinkedIn. // so it often carries a job days ahead of LinkedIn.
@@ -291,6 +386,16 @@ function stripTracking(url: string): string {
export async function fetchJobRoom(): Promise<Job[]> { export async function fetchJobRoom(): Promise<Job[]> {
const ads = new Map<string, JobRoomAd>(); const ads = new Map<string, JobRoomAd>();
// The search API returns a short keyword-anchored teaser per hit, not the
// full posting — the same ad re-appears across keyword sweeps with a
// different teaser each time, and `ads.set()` above only keeps the latest.
// A genuinely German ad (Consult & Pepper's recurring "Senior Fullstack
// Software Engineer" — full text: "In Deutsch bist du verhandlungssicher")
// slipped through repeatedly because whichever keyword ran last happened to
// anchor on an English-reading fragment of it. Collecting every variant and
// checking them together means one clean-looking teaser can no longer
// override another teaser that actually shows the German.
const descVariants = new Map<string, Set<string>>();
for (const keyword of JOBROOM_KEYWORDS) { for (const keyword of JOBROOM_KEYWORDS) {
process.stdout.write(` ${keyword} … `); process.stdout.write(` ${keyword} … `);
@@ -313,7 +418,11 @@ export async function fetchJobRoom(): Promise<Job[]> {
const hits = (await res.json()) as Array<{ jobAdvertisement?: JobRoomAd }>; const hits = (await res.json()) as Array<{ jobAdvertisement?: JobRoomAd }>;
for (const h of hits) { for (const h of hits) {
if (h.jobAdvertisement?.id) ads.set(h.jobAdvertisement.id, h.jobAdvertisement); const id = h.jobAdvertisement?.id;
if (!id) continue;
ads.set(id, h.jobAdvertisement!);
const d = h.jobAdvertisement!.jobContent?.jobDescriptions?.[0]?.description;
if (d) (descVariants.get(id) ?? descVariants.set(id, new Set()).get(id)!).add(d);
} }
total += hits.length; total += hits.length;
if (hits.length < 100) break; if (hits.length < 100) break;
@@ -341,8 +450,11 @@ export async function fetchJobRoom(): Promise<Job[]> {
// A handful of ads ship without a description. isViableDescription passes // A handful of ads ship without a description. isViableDescription passes
// those through by design, so leave contentChecked unset and let index.ts // those through by design, so leave contentChecked unset and let index.ts
// fall back to fetching the page. // fall back to fetching the page. When we do have variants, join every
const body = desc.description ?? ""; // one seen across keyword sweeps rather than just the last teaser stored
// on `ad` (see comment above) — more text only adds detection signal, it
// never removes any.
const body = [...(descVariants.get(ad.id) ?? [])].join(" \n ") || desc.description || "";
const hasBody = body.trim().length > 40; const hasBody = body.trim().length > 40;
if (hasBody && !isViableDescription(body).ok) { langDropped++; continue; } if (hasBody && !isViableDescription(body).ok) { langDropped++; continue; }
@@ -521,6 +633,50 @@ export async function fetchBaselIntlOrgs(): Promise<Job[]> {
return jobs; return jobs;
} }
// sciCORE's open-positions page (formerly polled separately by
// scripts/scicore-watch.sh) is a plain-text announcement list, not a job
// board: each <li> is either an active posting linked to careers.unibas.ch
// or a closed/in-review notice with no link at all. Skip the latter — "the
// interview process is still ongoing, but the submission deadline has
// passed" is the site's own wording for "not accepting applications" — since
// they're not something Rafael could still apply to.
export async function fetchSciCore(): Promise<Job[]> {
const jobs: Job[] = [];
const url = "https://scicore.unibas.ch/about-scicore/open-positions/";
const res = await fetch(url, { headers: HEADERS });
if (!res.ok) { console.log(` sciCORE … failed (${res.status})`); return jobs; }
const html = await res.text();
const sectionM = html.match(/<h1>Join the team!<\/h1>([\s\S]*?)<footer/i);
const items = sectionM ? [...sectionM[1].matchAll(/<li[^>]*>([\s\S]*?)<\/li>/gi)] : [];
let kept = 0;
for (const [, raw] of items) {
if (/deadline has passed|interview process is (?:still\s+)?ongoing/i.test(raw)) continue;
const linkM = raw.match(/<a\s[^>]*href="([^"]+)"[^>]*>([\s\S]*?)<\/a>/i);
const title = decodeHtml((linkM ? linkM[2] : raw).replace(/<[^>]+>/g, " "));
if (!title || !applyFilters(title)) continue;
const link = linkM ? linkM[1] : url;
jobs.push({
id: `scicore-${link.replace(/\W+/g, "").slice(-24)}`,
title,
company: "sciCORE",
location: "Basel, Switzerland",
postedAt: "",
link,
search: "scicore.unibas.ch",
source: "scicore.unibas.ch",
contentChecked: true, // listing is site-authored English; no separate description to fetch
});
kept++;
}
console.log(` sciCORE … ${items.length} items, ${kept} kept`);
return jobs;
}
// jobbasel.ch — the page shell is a React/Vite SPA (no jobs in the raw HTML // jobbasel.ch — the page shell is a React/Vite SPA (no jobs in the raw HTML
// on the homepage), but the search results route is server-rendered: fetching // on the homepage), but the search results route is server-rendered: fetching
// `/job/suche-{slug}` returns full HTML with a `window.__PRELOADED_STATE__` // `/job/suche-{slug}` returns full HTML with a `window.__PRELOADED_STATE__`
+54 -8
View File
@@ -200,6 +200,22 @@ const GERMAN_OPTIONAL_RE =
const GERMAN_REQUIRED_RE = const GERMAN_REQUIRED_RE =
/\b(?:required|require[sd]?|must|mandatory|essential|necessary|fluen\w*|proficien\w*|excellent|strong|very\s+good|good\s+(?:knowledge|command|level)|native|mother\s*tongue|business\s+level|negotiat\w*|B2|C1|C2|verhandlungssicher\w*|fliessend\w*|fließend\w*|muttersprach\w*|sehr\s+gut\w*|gut\w*\s+Deutschkenntnisse|Deutschkenntnisse|voraussetzung|zwingend|erforderlich)\b/i; /\b(?:required|require[sd]?|must|mandatory|essential|necessary|fluen\w*|proficien\w*|excellent|strong|very\s+good|good\s+(?:knowledge|command|level)|native|mother\s*tongue|business\s+level|negotiat\w*|B2|C1|C2|verhandlungssicher\w*|fliessend\w*|fließend\w*|muttersprach\w*|sehr\s+gut\w*|gut\w*\s+Deutschkenntnisse|Deutschkenntnisse|voraussetzung|zwingend|erforderlich)\b/i;
/**
* "German is <verdict>" / "proficiency in German is considered <verdict>" —
* the copula directly after the mention names the mention's own predicate,
* which should decide the case outright rather than compete on raw distance
* with unrelated markers elsewhere in the window. Swisscom's stock line
* "Fluency in English is required; proficiency in German is considered an
* advantage" put a required-marker word ("proficiency") right next to
* "German" and an optional-marker word ("advantage") further away — nearest()
* picked "proficiency" and wrongly called it required. The predicate right
* after the mention ("is considered an advantage") is what actually decides.
* Only fires when a copula immediately follows; "Work proficiency in German,
* English and French" (no copula after "German") correctly falls through to
* the general nearest-marker logic below.
*/
const GERMAN_LOCAL_PREDICATE_RE = /^\s*,?\s*(?:is|are)\s+(?:considered\s+|deemed\s+)?/i;
/** /**
* Spec-sheet style language line ("Languages: German & English"), common on * Spec-sheet style language line ("Languages: German & English"), common on
* contractor postings. There is no adjective for the nearest-marker logic to * contractor postings. There is no adjective for the nearest-marker logic to
@@ -261,6 +277,16 @@ export function requiresGerman(text: string): boolean {
for (const m of flat.matchAll(GERMAN_MENTION_RE)) { for (const m of flat.matchAll(GERMAN_MENTION_RE)) {
const i = m.index ?? 0; const i = m.index ?? 0;
// The mention's own predicate ("German is <verdict>") overrides distance.
const after = flat.slice(i + m[0].length, i + m[0].length + 60);
const predicateM = after.match(GERMAN_LOCAL_PREDICATE_RE);
if (predicateM) {
const rest = after.slice(predicateM[0].length);
if (new RegExp(`^${GERMAN_OPTIONAL_RE.source}`, "i").test(rest)) continue;
if (new RegExp(`^${GERMAN_REQUIRED_RE.source}`, "i").test(rest)) return true;
}
const from = Math.max(0, i - 110); const from = Math.max(0, i - 110);
const window = flat.slice(from, i + 110); const window = flat.slice(from, i + 110);
const at = i - from; // mention's offset inside the window const at = i - from; // mention's offset inside the window
@@ -275,7 +301,13 @@ export function requiresGerman(text: string): boolean {
export async function fetchDescription(id: string): Promise<string | null> { export async function fetchDescription(id: string): Promise<string | null> {
for (let attempt = 0; attempt < 3; attempt++) { for (let attempt = 0; attempt < 3; attempt++) {
try { try {
await Bun.sleep(350 * (attempt + 1)); // 429s need the same backoff as the listing-page retry (1s/2s/4s) — the
// old 350/700/1050ms wait was too short under concurrent load, so this
// exhausted its retries while still rate-limited and fell through to the
// "can't read it, so keep" default below, letting German postings (e.g.
// CLEEVEN, job 4465075383, all-German with zero English content) slip
// past isViableContent entirely on 2026-09-10.
await Bun.sleep(1000 * 2 ** attempt);
const res = await fetch(`https://www.linkedin.com/jobs/view/${id}/`, { headers: HEADERS }); const res = await fetch(`https://www.linkedin.com/jobs/view/${id}/`, { headers: HEADERS });
const html = await res.text(); const html = await res.text();
const m = html.match(/class="show-more-less-html__markup[^"]*">([\s\S]{0,8000}?)<\/div>/); const m = html.match(/class="show-more-less-html__markup[^"]*">([\s\S]{0,8000}?)<\/div>/);
@@ -289,10 +321,21 @@ export async function fetchDescription(id: string): Promise<string | null> {
return null; return null;
} }
/** English-language AND not demanding German above B1. */ /**
export async function isViableContent(id: string): Promise<{ ok: boolean; reason?: string }> { * English-language AND not demanding German above B1.
*
* `retry: true` means the description couldn't be read at all (rate-limited,
* stub page) — index.ts leaves the job out of the CSV entirely rather than
* defaulting to "keep", so it's picked up fresh and re-checked next run
* instead of ever being judged sight-unseen. The old default-to-keep here is
* what let a fully-German HRM Systems/Consult & Pepper posting ("In Deutsch
* bist du verhandlungssicher") through on 2026-09-10 — same failure mode as
* the CLEEVEN case above, just on the "keep" side of isViableContent instead
* of isEnglishContent.
*/
export async function isViableContent(id: string): Promise<{ ok: boolean; reason?: string; retry?: boolean }> {
const text = await fetchDescription(id); const text = await fetchDescription(id);
if (text === null) return { ok: true }; // keep when we genuinely can't read it if (text === null) return { ok: false, reason: "unreadable — retrying next run", retry: true };
if (hasNonEnglishUmlaut(text)) return { ok: false, reason: "non-English (umlaut)" }; if (hasNonEnglishUmlaut(text)) return { ok: false, reason: "non-English (umlaut)" };
const de = (text.match(DE_CONTENT_RE) ?? []).length; const de = (text.match(DE_CONTENT_RE) ?? []).length;
const fr = (text.match(FR_CONTENT_RE) ?? []).length; const fr = (text.match(FR_CONTENT_RE) ?? []).length;
@@ -307,7 +350,8 @@ export async function isEnglishContent(id: string): Promise<boolean> {
// false "English" result. Back off and try again before falling back. // false "English" result. Back off and try again before falling back.
for (let attempt = 0; attempt < 3; attempt++) { for (let attempt = 0; attempt < 3; attempt++) {
try { try {
await Bun.sleep(350 * (attempt + 1)); // Same 1s/2s/4s backoff as fetchDescription — see its comment.
await Bun.sleep(1000 * 2 ** attempt);
const res = await fetch(`https://www.linkedin.com/jobs/view/${id}/`, { headers: HEADERS }); const res = await fetch(`https://www.linkedin.com/jobs/view/${id}/`, { headers: HEADERS });
const html = await res.text(); const html = await res.text();
const m = html.match(/class="show-more-less-html__markup[^"]*">([\s\S]{0,8000}?)<\/div>/); const m = html.match(/class="show-more-less-html__markup[^"]*">([\s\S]{0,8000}?)<\/div>/);
@@ -337,11 +381,13 @@ export async function isEnglishUrl(url: string): Promise<boolean> {
} }
/** As isViableContent, but for an arbitrary job URL (non-LinkedIn sources). */ /** As isViableContent, but for an arbitrary job URL (non-LinkedIn sources). */
export async function isViableUrl(url: string): Promise<{ ok: boolean; reason?: string }> { export async function isViableUrl(url: string): Promise<{ ok: boolean; reason?: string; retry?: boolean }> {
try { try {
await Bun.sleep(350); await Bun.sleep(350);
const res = await fetch(url, { headers: HEADERS }); const res = await fetch(url, { headers: HEADERS });
if (!res.ok) return { ok: true }; // Same reasoning as isViableContent: an unreadable page shouldn't default
// to "keep" — defer it and let next run's fresh fetch actually judge it.
if (!res.ok) return { ok: false, reason: "unreadable — retrying next run", retry: true };
const text = (await res.text()).replace(/<[^>]+>/g, " "); const text = (await res.text()).replace(/<[^>]+>/g, " ");
if (hasNonEnglishUmlaut(text)) return { ok: false, reason: "non-English (umlaut)" }; if (hasNonEnglishUmlaut(text)) return { ok: false, reason: "non-English (umlaut)" };
const de = (text.match(DE_CONTENT_RE) ?? []).length; const de = (text.match(DE_CONTENT_RE) ?? []).length;
@@ -350,7 +396,7 @@ export async function isViableUrl(url: string): Promise<{ ok: boolean; reason?:
if (requiresGerman(text)) return { ok: false, reason: "requires German > B1" }; if (requiresGerman(text)) return { ok: false, reason: "requires German > B1" };
return { ok: true }; return { ok: true };
} catch { } catch {
return { ok: true }; return { ok: false, reason: "unreadable — retrying next run", retry: true };
} }
} }
+47 -3
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env bun #!/usr/bin/env bun
import { searches } from "./config.ts"; import { searches } from "./config.ts";
import { fetchJobs, isViableContent, isViableUrl } from "./fetch.ts"; import { fetchJobs, isViableContent, isViableUrl } from "./fetch.ts";
import { fetchDataCareer, fetchStellenBasel, fetchItJobsCh, fetchJobRoom, fetchBaselIntlOrgs, fetchJobBasel, fetchEnglishJobSearch, fetchBaselJobs } from "./fetch-other.ts"; import { fetchDataCareer, fetchStellenBasel, fetchItJobsCh, fetchJobRoom, fetchBaselIntlOrgs, fetchJobBasel, fetchEnglishJobSearch, fetchBaselJobs, fetchSciCore, fetchSwisscom } from "./fetch-other.ts";
const CSV = "./results/jobs.csv"; const CSV = "./results/jobs.csv";
const OLD_HEADER = "date_found,posted_at,search,title,company,link"; const OLD_HEADER = "date_found,posted_at,search,title,company,link";
@@ -62,6 +62,36 @@ const date = new Date().toISOString().slice(0, 10);
const newRows: string[] = []; const newRows: string[] = [];
let totalLangDropped = 0; // jobs dropped for German/French content — never reach the CSV, so daily.ts can't see them on its own let totalLangDropped = 0; // jobs dropped for German/French content — never reach the CSV, so daily.ts can't see them on its own
// Persisted across runs so the Telegram digest's "Ignored (German/French)"
// count reflects genuinely new drops, not the same posting getting re-flagged
// every run (dropped jobs never reach the CSV, so nothing else remembers
// them) or the same underlying job counted again each time it resurfaces
// under a different link/provider (e.g. Consult & Pepper's "Senior Fullstack
// Software Engineer" reposted on job-room.ch every few days, and separately
// on LinkedIn) — keyed on normalized title+company rather than the link.
const LANG_SEEN = "./results/.lang-dropped-seen.json";
const langSeen = new Set<string>(
await Bun.file(LANG_SEEN)
.json()
.catch(() => [])
);
function langKey(title: string, company: string): string {
const t = title
.replace(/[([][^)\]]*[)\]]/g, "") // drop "(all genders)", "(m/w/d)", "(a)", "(80 – 100%)" noise
.replace(/\s+/g, " ")
.trim()
.toLowerCase();
return `${t}|${company.trim().toLowerCase()}`;
}
// Only counts (and remembers) a drop the first time this title+company is
// seen; still runs for every dropped job so `langSeen` stays complete.
function countLangDrop(title: string, company: string): void {
const key = langKey(title, company);
if (langSeen.has(key)) return;
langSeen.add(key);
totalLangDropped++;
}
function csvCell(val: string): string { function csvCell(val: string): string {
return `"${val.replace(/"/g, '""')}"`; return `"${val.replace(/"/g, '""')}"`;
} }
@@ -115,10 +145,17 @@ async function processLinkedInSearch(config: (typeof searches)[number]): Promise
// One fetch, two checks: written in German/French, or written in English // One fetch, two checks: written in German/French, or written in English
// but demanding German above B1. // but demanding German above B1.
const viable = await isViableContent(job.id); const viable = await isViableContent(job.id);
if (viable.retry) {
// Couldn't read the description at all — leave it out of the CSV so
// next run's fresh fetch judges it properly, instead of defaulting to
// "keep" and risking a German-requiring posting slipping through.
console.log(viable.reason);
continue;
}
if (!viable.ok) { if (!viable.ok) {
console.log(`dropped (${viable.reason})`); console.log(`dropped (${viable.reason})`);
langDropped++; langDropped++;
totalLangDropped++; countLangDrop(job.title, job.company);
continue; continue;
} }
console.log("ok"); console.log("ok");
@@ -141,6 +178,8 @@ const extraSources: Array<{ label: string; fetch: () => ReturnType<typeof fetchD
{ label: "jobbasel.ch", fetch: fetchJobBasel }, { label: "jobbasel.ch", fetch: fetchJobBasel },
{ label: "englishjobsearch.ch", fetch: fetchEnglishJobSearch }, { label: "englishjobsearch.ch", fetch: fetchEnglishJobSearch },
{ label: "basel-jobs.ch", fetch: fetchBaselJobs }, { label: "basel-jobs.ch", fetch: fetchBaselJobs },
{ label: "scicore.unibas.ch", fetch: fetchSciCore },
{ label: "swisscom.com", fetch: fetchSwisscom },
]; ];
async function processExtraSource({ label, fetch: fetchFn }: (typeof extraSources)[number]): Promise<void> { async function processExtraSource({ label, fetch: fetchFn }: (typeof extraSources)[number]): Promise<void> {
@@ -167,10 +206,14 @@ async function processExtraSource({ label, fetch: fetchFn }: (typeof extraSource
if (!job.contentChecked) { if (!job.contentChecked) {
process.stdout.write(` [${label}] checking content: ${job.title} … `); process.stdout.write(` [${label}] checking content: ${job.title} … `);
const viable = await isViableUrl(job.link); const viable = await isViableUrl(job.link);
if (viable.retry) {
console.log(viable.reason);
continue;
}
if (!viable.ok) { if (!viable.ok) {
console.log(`dropped (${viable.reason})`); console.log(`dropped (${viable.reason})`);
langDropped++; langDropped++;
totalLangDropped++; countLangDrop(job.title, job.company);
continue; continue;
} }
console.log("ok"); console.log("ok");
@@ -198,6 +241,7 @@ await Promise.all([
// without listing rejects that were never even parsed into a Row. // without listing rejects that were never even parsed into a Row.
const LANG_DROPPED = "./results/.lang-dropped.json"; const LANG_DROPPED = "./results/.lang-dropped.json";
await Bun.write(LANG_DROPPED, JSON.stringify({ date, count: totalLangDropped })); await Bun.write(LANG_DROPPED, JSON.stringify({ date, count: totalLangDropped }));
await Bun.write(LANG_SEEN, JSON.stringify([...langSeen]));
if (newRows.length === 0) { if (newRows.length === 0) {
console.log("\nNo new jobs."); console.log("\nNo new jobs.");