Compare commits
10
Commits
d2aa61c8ad
...
3875f13713
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3875f13713 | ||
|
|
288b8bdcd1 | ||
|
|
9cbc525cf5 | ||
|
|
29b00cd91e | ||
|
|
d401dd5f9a | ||
|
|
0249800278 | ||
|
|
c7b86e06f4 | ||
|
|
821bd19ff7 | ||
|
|
a72a31f387 | ||
|
|
e36bcdeb3a |
@@ -21,7 +21,7 @@ Temp scripts: `bun run scripts/<name>-temp.ts` — never edit `config.ts` for on
|
|||||||
```
|
```
|
||||||
config.ts # search keywords and areas
|
config.ts # search keywords and areas
|
||||||
fetch.ts # scraper, HTML parser, all filters
|
fetch.ts # scraper, HTML parser, all filters
|
||||||
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch)
|
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com)
|
||||||
index.ts # orchestrator: runs searches, writes CSV
|
index.ts # orchestrator: runs searches, writes CSV
|
||||||
recheck.ts # raw-mode fetch, lists filtered-out jobs
|
recheck.ts # raw-mode fetch, lists filtered-out jobs
|
||||||
daily.ts # digest: fetch → match against profile.ts → send to Telegram
|
daily.ts # digest: fetch → match against profile.ts → send to Telegram
|
||||||
@@ -43,7 +43,7 @@ plans/ # LinkedIn post drafts
|
|||||||
- Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform) and `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform and Field add a Switzerland-wide remote pass.
|
- Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform) and `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform and Field add a Switzerland-wide remote pass.
|
||||||
- `RESEARCH_KEYWORDS` deliberately contains no bioinformatics/computational-biology terms — see `BIO_DOMAIN_RE`.
|
- `RESEARCH_KEYWORDS` deliberately contains no bioinformatics/computational-biology terms — see `BIO_DOMAIN_RE`.
|
||||||
- **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`.
|
- **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`.
|
||||||
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch). Imported by `index.ts`.
|
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com). Imported by `index.ts`.
|
||||||
- **datacareer.ch** inverts the usual bot rule: its nginx **403s browser-like User-Agents** and serves honest ones (no UA, `curl/8.0`, Googlebot) fine. The spoofed Chrome UA in the shared `HEADERS` was the entire cause of the long-standing 403 — it uses `PLAIN_HEADERS` instead. Small board: Basel returns **8 jobs on one page with no pagination**, so `?page=2` legitimately yields 0 items.
|
- **datacareer.ch** inverts the usual bot rule: its nginx **403s browser-like User-Agents** and serves honest ones (no UA, `curl/8.0`, Googlebot) fine. The spoofed Chrome UA in the shared `HEADERS` was the entire cause of the long-standing 403 — it uses `PLAIN_HEADERS` instead. Small board: Basel returns **8 jobs on one page with no pagination**, so `?page=2` legitimately yields 0 items.
|
||||||
- **job-room.ch** carries its language requirement as a **structured field** (`languageSkills[]`, e.g. `{languageIsoCode: "de", spokenLevel: "PROFICIENT"}`), entirely separate from the free-text `jobDescriptions[].description` — the UI renders it as "Deutsch (Fliessend)". `requiresGerman()`/`isViableDescription()` only ever scan the prose, so this field slipped through unchecked until 2026-09-04: measured against 1843 ads, 69 required PROFICIENT-level German and the prose check caught only 3 of them incidentally. `jobRoomRequiresGerman()` now checks the structured field directly (`spokenLevel`/`writtenLevel === "PROFICIENT"`) before the description-based check runs. Levels seen: `NONE`, `BASIC`, `INTERMEDIATE`, `PROFICIENT` — only `PROFICIENT` (fluent/C1+) is dropped, matching the B1 cutoff used everywhere else.
|
- **job-room.ch** carries its language requirement as a **structured field** (`languageSkills[]`, e.g. `{languageIsoCode: "de", spokenLevel: "PROFICIENT"}`), entirely separate from the free-text `jobDescriptions[].description` — the UI renders it as "Deutsch (Fliessend)". `requiresGerman()`/`isViableDescription()` only ever scan the prose, so this field slipped through unchecked until 2026-09-04: measured against 1843 ads, 69 required PROFICIENT-level German and the prose check caught only 3 of them incidentally. `jobRoomRequiresGerman()` now checks the structured field directly (`spokenLevel`/`writtenLevel === "PROFICIENT"`) before the description-based check runs. Levels seen: `NONE`, `BASIC`, `INTERMEDIATE`, `PROFICIENT` — only `PROFICIENT` (fluent/C1+) is dropped, matching the B1 cutoff used everywhere else.
|
||||||
- **jobbasel.ch** looks like a JS-only React/Vite SPA (no jobs in the homepage's raw HTML), but its search route is server-rendered: `GET /job/suche-{slug}` returns full HTML with a `window.__PRELOADED_STATE__` blob containing the matched vacancies as JSON — no headless browser needed. Pagination is a URL suffix (`-seite-{n}`); a bare `?page=` query param is silently ignored. Swept with the same fixed keyword list as job-room.ch (no keyword-union endpoint exists) and deduped by vacancy id. Each item carries its full German description, so the DE/FR check runs in-process.
|
- **jobbasel.ch** looks like a JS-only React/Vite SPA (no jobs in the homepage's raw HTML), but its search route is server-rendered: `GET /job/suche-{slug}` returns full HTML with a `window.__PRELOADED_STATE__` blob containing the matched vacancies as JSON — no headless browser needed. Pagination is a URL suffix (`-seite-{n}`); a bare `?page=` query param is silently ignored. Swept with the same fixed keyword list as job-room.ch (no keyword-union endpoint exists) and deduped by vacancy id. Each item carries its full German description, so the DE/FR check runs in-process.
|
||||||
@@ -82,6 +82,10 @@ plans/ # LinkedIn post drafts
|
|||||||
- **Basel Institute on Governance** — `baselgovernance.org/about/careers`. Each vacancy is a linked PDF, so there's no HTML description to language-check. The `rounded-3xl` card class is what separates vacancy tiles from the annual-report PDF link.
|
- **Basel Institute on Governance** — `baselgovernance.org/about/careers`. Each vacancy is a linked PDF, so there's no HTML description to language-check. The `rounded-3xl` card class is what separates vacancy tiles from the annual-report PDF link.
|
||||||
- These boards use UN-flavoured titles ("Information Management Officer", "ICT Specialist") that `SOFTWARE_TITLE_RE` alone drops, so `INTL_TECH_RE` widens the net **for these sources only**. Boards this small can afford the noise; missing a rare opening can't.
|
- These boards use UN-flavoured titles ("Information Management Officer", "ICT Specialist") that `SOFTWARE_TITLE_RE` alone drops, so `INTL_TECH_RE` widens the net **for these sources only**. Boards this small can afford the noise; missing a rare opening can't.
|
||||||
- **itjobs.ch** is swept Switzerland-wide (no `location` param) — the board is small, ~210 jobs over 7 pages, and locking it to Basel returned only ~9. It 404s one page past the end instead of serving an empty list. Each item embeds its full `description`, so the DE/FR check runs in-process via `isEnglishDescription` rather than costing a page fetch per job. Use `job_location.name` for location — it's the English form; the bare `location` field is German.
|
- **itjobs.ch** is swept Switzerland-wide (no `location` param) — the board is small, ~210 jobs over 7 pages, and locking it to Basel returned only ~9. It 404s one page past the end instead of serving an empty list. Each item embeds its full `description`, so the DE/FR check runs in-process via `isEnglishDescription` rather than costing a page fetch per job. Use `job_location.name` for location — it's the English form; the bare `location` field is German.
|
||||||
|
- **scicore.unibas.ch** (`fetchSciCore`) replaces the standalone `scripts/scicore-watch.sh` launchd watcher (removed 2026-09-16) — sciCORE now runs inside `bun fetch`/`bun daily` like every other source instead of its own 08:47 agent. The open-positions page is a plain-text announcement list, not a real job board: each `<li>` under `<h1>Join the team!</h1>` is either an active posting linked to `careers.unibas.ch` or a closed/in-review notice with no link — items matching `deadline has passed`/`interview process is (still) ongoing` are skipped since they're not open applications. `contentChecked: true` because the listing itself is the only English text available; there's no separate description page to fetch. Title still has to clear `SOFTWARE_TITLE_RE`, same as every other source — this is deliberately broader than the old watcher's "software"-only match (it also catches "security", matching the IT Security Engineer role Rafael prepped for — see `project_scicore_security_report` in memory) but still drops pure sysadmin titles like "Linux Systems Administrator".
|
||||||
|
- **swisscom.com** (`fetchSwisscom`, added 2026-09-18) — Swisscom's own Workday career site (`swisscom.wd103.myworkdayjobs.com/SwisscomExternalCareers`). Small board (~47 openings, Switzerland-wide), so no keyword sweep — the full listing is pulled and filtered locally, like itjobs.ch. The listing page itself is a bare JS shell with no jobs in the raw HTML; postings live behind the standard Workday CXS JSON API (`POST /wday/cxs/swisscom/SwisscomExternalCareers/jobs`). `total` on the first page (offset 0) is authoritative — later pages report `total: 0` (a tenant quirk), and an offset past the true total silently wraps around and re-serves page 1 instead of an empty list, so the loop's stop condition is the first page's total, never a later one's. Listing rows carry no description, so each title-filtered candidate costs one extra fetch against the job detail endpoint (`/job{externalPath}`), which returns the full HTML description — the DE/FR + `requiresGerman` check runs on that in-process (`contentChecked: true`).
|
||||||
|
|
||||||
|
Swisscom mostly hires in German — of 47 postings, only ~16 clear the English/software title filters, and nearly all of those turned out to be written in German too; only genuinely-English, German-optional roles survive. Adding this source surfaced a real bug in `requiresGerman()`: the stock line *"Fluency in English is required; proficiency in German is considered an advantage"* had the noun "proficiency" (a required-marker word) sitting right next to "German", closer than the real optional-marker "advantage" — nearest() picked the wrong one and wrongly flagged the posting as requiring German. Fixed by adding a local-predicate override (`GERMAN_LOCAL_PREDICATE_RE`): when a copula ("is"/"are", optionally "considered"/"deemed") directly follows the German mention, that immediate predicate decides the verdict outright instead of competing on raw character distance with markers elsewhere in the window — "German is considered an advantage" resolves as optional regardless of what else is nearby. It only fires when a copula immediately follows the mention, so *"Work proficiency in German, English, and French"* (no copula after "German", a genuine hard requirement — Senior Operation Architect (FR/GER/ENG)) still falls through to the general nearest-marker logic and is correctly caught as required. Re-check both cases (and the pre-existing Helvetia/albedis/Karakun/J. Safra Sarasin/BBF/spec-sheet cases in the paragraph above) before touching `requiresGerman` again.
|
||||||
- **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV. Writes both `search` (the query/source that found the job) and `location` (the posting's own location, as parsed off its card by `fetch.ts`/`fetch-other.ts` — no extra page fetch) to the CSV. Migrates old CSVs on startup: adds `source` if missing, then `location` (backfilled `""` for rows written before the column existed).
|
- **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV. Writes both `search` (the query/source that found the job) and `location` (the posting's own location, as parsed off its card by `fetch.ts`/`fetch-other.ts` — no extra page fetch) to the CSV. Migrates old CSVs on startup: adds `source` if missing, then `location` (backfilled `""` for rows written before the column existed).
|
||||||
- Every LinkedIn search and every `fetch-other.ts` source hits a different host, so they run **concurrently** (bounded: 3 at a time within each group, both groups started together) instead of one after another — cut a full `bun fetch` from ~16-20 min to well under 2 min. `existingLinks.add()` happens synchronously right after the `.has()` check and before any `await`, which is what makes concurrent workers safe against double-claiming the same link. Concurrency surfaced a real gap: LinkedIn started returning occasional 429s under parallel load, which `fetchJobs` used to treat as end-of-results (silently truncating that search) — `fetchJobs` now retries a 429 up to 3× with backoff (1s/2s/4s) before giving up.
|
- Every LinkedIn search and every `fetch-other.ts` source hits a different host, so they run **concurrently** (bounded: 3 at a time within each group, both groups started together) instead of one after another — cut a full `bun fetch` from ~16-20 min to well under 2 min. `existingLinks.add()` happens synchronously right after the `.has()` check and before any `await`, which is what makes concurrent workers safe against double-claiming the same link. Concurrency surfaced a real gap: LinkedIn started returning occasional 429s under parallel load, which `fetchJobs` used to treat as end-of-results (silently truncating that search) — `fetchJobs` now retries a 429 up to 3× with backoff (1s/2s/4s) before giving up.
|
||||||
- **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions.
|
- **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions.
|
||||||
@@ -203,24 +207,15 @@ Some employers (Optiml, LocalBini seen repeatedly) repost the same role under a
|
|||||||
|
|
||||||
## Scheduled automations
|
## Scheduled automations
|
||||||
|
|
||||||
Both run as **launchd agents** — this machine has no cron at all, so don't reach for `crontab`.
|
Runs as a **launchd agent** — this machine has no cron at all, so don't reach for `crontab`.
|
||||||
|
|
||||||
| plist (`~/Library/LaunchAgents/`) | when | runs |
|
| plist (`~/Library/LaunchAgents/`) | when | runs |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `com.rafapolo.linkme-daily` | 08:00 + 18:00 | `bun run daily.ts` → `results/daily.log` |
|
| `com.rafapolo.linkme-daily` | 09:00 / 12:00 / 15:00 / 17:00 / 20:00 | `bun run daily.ts` → `results/daily.log` |
|
||||||
| `ch.polo.scicore-watch` | 08:47 | `scripts/scicore-watch.sh` → `results/.scicore-watch/watch.log` |
|
|
||||||
|
|
||||||
If the Mac is asleep at the scheduled time, launchd fires the job on wake rather than skipping the day. `launchctl list | grep -E 'linkme|scicore'` to check they're loaded; `launchctl unload <plist>` to stop one.
|
If the Mac is asleep at the scheduled time, launchd fires the job on wake rather than skipping the day. `launchctl list | grep linkme` to check it's loaded; `launchctl unload <plist>` to stop it.
|
||||||
|
|
||||||
**`scripts/scicore-watch.sh`** watches https://scicore.unibas.ch/about-scicore/open-positions/ for a software engineering role Rafael is waiting on. It extracts the "Join the team!" block (keeping each posting's `<a href>` so the alert is tappable), diffs it against `results/.scicore-watch/snapshot.txt`, and notifies **only when a newly added line matches `$WANTED` ("software")**. Every other change still updates the snapshot silently — that's what keeps the eventual real alert clean instead of tangled with months of sysadmin churn. Three consecutive fetch failures alert, so a broken watcher can't look like "no news".
|
**`scripts/scicore-watch.sh`** is retired (2026-09-16) — sciCORE is now swept by `fetchSciCore` in `fetch-other.ts` as part of `bun fetch`/`bun daily`, see the `fetch-other.ts` entry above. Its `ch.polo.scicore-watch` launchd agent (08:47) was unloaded and its plist removed; the script and `results/.scicore-watch/` (snapshot + log) are left in the repo for reference/history, not run anymore.
|
||||||
|
|
||||||
Test the alert path without waiting for the page to change:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
SCICORE_URL="file://$PWD/fixture.html" bash scripts/scicore-watch.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
The whole extractor is a `python3 -c` program inside shell single quotes, so **it must not contain a literal apostrophe** — that's why the `<a>` quote class is built with `chr(39)` instead of written inline.
|
|
||||||
|
|
||||||
## LinkedIn session token
|
## LinkedIn session token
|
||||||
|
|
||||||
|
|||||||
@@ -1,13 +1,14 @@
|
|||||||
# linkme
|
# linkme
|
||||||
|
|
||||||
A personal job-hunting pipeline for the Swiss market. It scrapes LinkedIn (plus a handful of Swiss boards), filters out everything that doesn't fit the profile in `cv/polo.md`, appends what survives to a CSV, and pushes the good ones to Telegram twice a day.
|
A personal job-hunting pipeline for the Swiss market. It scrapes LinkedIn (plus a handful of Swiss boards), filters out everything that doesn't fit the profile in `cv/polo.md`, appends what survives to a CSV, and pushes the good ones to Telegram throughout the day.
|
||||||
|
|
||||||
Two things run on their own once installed:
|
One thing runs on its own once installed:
|
||||||
|
|
||||||
| automation | when | what it does |
|
| automation | when | what it does |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| **daily digest** | 08:00 + 18:00 | fetch every search, push new profile-matching jobs to Telegram |
|
| **daily digest** | 09:00 / 12:00 / 15:00 / 17:00 / 20:00 | fetch every search + source, push new profile-matching jobs to Telegram |
|
||||||
| **sciCORE watch** | 08:47 | alert if sciCORE's open-positions page lists a *software* role |
|
|
||||||
|
sciCORE used to have its own separate 08:47 watcher; it's now just another source swept inside `bun fetch`/`bun daily` (see Sources below).
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
@@ -40,13 +41,13 @@ Then `bun daily --dry-run` prints the digest instead of sending it, and `bun dai
|
|||||||
```
|
```
|
||||||
config.ts # keyword families, search areas, the sweep() generator
|
config.ts # keyword families, search areas, the sweep() generator
|
||||||
fetch.ts # LinkedIn scraper, HTML parser, all title/content filters
|
fetch.ts # LinkedIn scraper, HTML parser, all title/content filters
|
||||||
fetch-other.ts # the eight non-LinkedIn sources (see Sources below)
|
fetch-other.ts # the ten non-LinkedIn sources (see Sources below)
|
||||||
index.ts # orchestrator: run searches → dedupe → content-check → CSV
|
index.ts # orchestrator: run searches → dedupe → content-check → CSV
|
||||||
recheck.ts # raw-mode fetch: lists what the filters dropped
|
recheck.ts # raw-mode fetch: lists what the filters dropped
|
||||||
daily.ts # digest: fetch, match against the profile, send to Telegram
|
daily.ts # digest: fetch, match against the profile, send to Telegram
|
||||||
profile.ts # title-level scoring — decides what's worth a notification
|
profile.ts # title-level scoring — decides what's worth a notification
|
||||||
telegram.ts # Telegram delivery + --setup chat-ID discovery
|
telegram.ts # Telegram delivery + --setup chat-ID discovery
|
||||||
scripts/ # one-off searches (*-temp.ts) and scicore-watch.sh
|
scripts/ # one-off searches (*-temp.ts); scicore-watch.sh kept for history, retired
|
||||||
cv/polo.md # the profile everything is ranked against
|
cv/polo.md # the profile everything is ranked against
|
||||||
results/jobs.csv # output, one row per job
|
results/jobs.csv # output, one row per job
|
||||||
results/sent.txt # links already pushed — never sent twice
|
results/sent.txt # links already pushed — never sent twice
|
||||||
@@ -67,6 +68,8 @@ LinkedIn is swept by `config.ts`'s search families. Everything else lives in `fe
|
|||||||
| **datacareer.ch** | Basel | 8 jobs on one page, no pagination |
|
| **datacareer.ch** | Basel | 8 jobs on one page, no pagination |
|
||||||
| **stellen-basel.ch** | Basel region | WordPress REST API |
|
| **stellen-basel.ch** | Basel region | WordPress REST API |
|
||||||
| **Basel intl orgs** | Basel | BIS (via the UNjobs mirror), Swiss TPH, Basel Institute on Governance |
|
| **Basel intl orgs** | Basel | BIS (via the UNjobs mirror), Swiss TPH, Basel Institute on Governance |
|
||||||
|
| **scicore.unibas.ch** | Basel | sciCORE's open-positions announcement list; used to be its own separate 08:47 watcher, now just another source |
|
||||||
|
| **swisscom.com** | Switzerland | Swisscom's own Workday career site, pulled via its public CXS JSON API; small board and mostly German-language, so few survive the filters |
|
||||||
|
|
||||||
Two of these were dead until recently and are worth knowing about:
|
Two of these were dead until recently and are worth knowing about:
|
||||||
|
|
||||||
@@ -165,21 +168,16 @@ bun run scripts/luzern-temp.ts
|
|||||||
|
|
||||||
## Scheduling
|
## Scheduling
|
||||||
|
|
||||||
Both automations are launchd agents, not cron — this machine has no cron at all.
|
The digest runs as a launchd agent, not cron — this machine has no cron at all.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
launchctl list | grep -E 'linkme|scicore' # are they loaded?
|
launchctl list | grep linkme # is it loaded?
|
||||||
launchctl unload ~/Library/LaunchAgents/com.rafapolo.linkme-daily.plist # stop the digest
|
launchctl unload ~/Library/LaunchAgents/com.rafapolo.linkme-daily.plist # stop it
|
||||||
launchctl unload ~/Library/LaunchAgents/ch.polo.scicore-watch.plist # stop the watcher
|
|
||||||
```
|
```
|
||||||
|
|
||||||
If the Mac is asleep at the scheduled time, launchd runs the job once it wakes rather than skipping the day.
|
If the Mac is asleep at the scheduled time, launchd runs the job once it wakes rather than skipping the day.
|
||||||
|
|
||||||
Logs: `results/daily.log` for the digest, `results/.scicore-watch/watch.log` for the watcher. The watcher logs every run — `no change`, `change ignored`, or the diff it sent — so silence on your phone can always be told apart from a broken script. Test its alert path without waiting for the real page to change:
|
Logs: `results/daily.log`.
|
||||||
|
|
||||||
```bash
|
|
||||||
SCICORE_URL="file://$PWD/fixture.html" bash scripts/scicore-watch.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
## recheck
|
## recheck
|
||||||
|
|
||||||
|
|||||||
@@ -168,6 +168,18 @@ export const searches: SearchConfig[] = [
|
|||||||
workType: ["onsite", "hybrid"],
|
workType: ["onsite", "hybrid"],
|
||||||
timePosted: "month",
|
timePosted: "month",
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
// Core KEYWORDS previously only ran onsite/hybrid against the three named
|
||||||
|
// cities — a manual "Fullstack Developer" remote-CH search on 2026-09-10
|
||||||
|
// turned up 21 viable postings (PAS01, xorlab, Proton, SCOR...) that no
|
||||||
|
// existing search covered, since Platform/Field are the only families with
|
||||||
|
// a Switzerland-wide remote sweep. Adding the same sweep for core KEYWORDS.
|
||||||
|
label: "Switzerland Remote",
|
||||||
|
keywords: KEYWORDS,
|
||||||
|
location: "Switzerland",
|
||||||
|
workType: ["remote"],
|
||||||
|
timePosted: "month",
|
||||||
|
},
|
||||||
];
|
];
|
||||||
|
|
||||||
// --- Role-family sweeps ---------------------------------------------------
|
// --- Role-family sweeps ---------------------------------------------------
|
||||||
|
|||||||
@@ -62,7 +62,55 @@ const langDropped = await Bun.file(LANG_DROPPED)
|
|||||||
.catch(() => 0);
|
.catch(() => 0);
|
||||||
|
|
||||||
// Only leads that actually match the CV (profile.ts) and haven't been sent before.
|
// Only leads that actually match the CV (profile.ts) and haven't been sent before.
|
||||||
const matched = fresh.filter((r) => !sent.has(r.link) && matchesProfile(r.title, r.company).ok);
|
const preDedupe = fresh.filter((r) => !sent.has(r.link) && matchesProfile(r.title, r.company).ok);
|
||||||
|
|
||||||
|
// Sources can surface the same underlying opening twice under different
|
||||||
|
// links/providers — e.g. englishjobsearch.ch listing the same posting on two
|
||||||
|
// canton pages with the company rendered as "Swiss AviationSoftware" on one
|
||||||
|
// and "Swiss-AS" on the other. Exact title+company catches the common case
|
||||||
|
// (same employer, identical spelling); the title-only pass catches spelling
|
||||||
|
// drift too, but only for titles specific enough (5+ words) that a
|
||||||
|
// coincidental match between two genuinely different postings is implausible.
|
||||||
|
const normTitle = (title: string) =>
|
||||||
|
title
|
||||||
|
.replace(/[([][^)\]]*[)\]]/g, "") // drop "(all genders)", "(m/w/d)", "(a)", "80-100%" noise
|
||||||
|
.replace(/[^a-z0-9 ]/gi, " ")
|
||||||
|
.replace(/\s+/g, " ")
|
||||||
|
.trim()
|
||||||
|
.toLowerCase();
|
||||||
|
const normCompany = (company: string) =>
|
||||||
|
company
|
||||||
|
.replace(/\b(ag|gmbh|ltd|llc|inc|sa|se|plc|co)\b\.?/gi, "")
|
||||||
|
.replace(/[^a-z0-9]/gi, "")
|
||||||
|
.toLowerCase();
|
||||||
|
const seenExact = new Set<string>();
|
||||||
|
const seenTitleOnly = new Set<string>();
|
||||||
|
|
||||||
|
// Seed both sets from everything already sent (any past day), keyed the same
|
||||||
|
// way — not just today's batch. Without this, a job re-dedupes only against
|
||||||
|
// jobs sent in the *same run*: LinkedIn reposts the same opening under a new
|
||||||
|
// job ID every few weeks (Optiml, LocalBini, hexagonab, nicollcurtin, etc. —
|
||||||
|
// confirmed repeats in sent.txt across different days), each repost lands
|
||||||
|
// under a fresh link that passes the `sent.has(link)` check untouched, and it
|
||||||
|
// got renotified as if new. `all` (the full CSV) still has every past row, so
|
||||||
|
// each previously-sent link resolves back to a title+company to seed with.
|
||||||
|
for (const r of all) {
|
||||||
|
if (!sent.has(r.link)) continue;
|
||||||
|
const t = normTitle(r.title);
|
||||||
|
seenExact.add(`${t}|${normCompany(r.company)}`);
|
||||||
|
if (t.split(" ").filter(Boolean).length >= 5) seenTitleOnly.add(t);
|
||||||
|
}
|
||||||
|
|
||||||
|
const matched = preDedupe.filter((r) => {
|
||||||
|
const t = normTitle(r.title);
|
||||||
|
const exactKey = `${t}|${normCompany(r.company)}`;
|
||||||
|
if (seenExact.has(exactKey)) return false;
|
||||||
|
const specific = t.split(" ").filter(Boolean).length >= 5;
|
||||||
|
if (specific && seenTitleOnly.has(t)) return false;
|
||||||
|
seenExact.add(exactKey);
|
||||||
|
if (specific) seenTitleOnly.add(t);
|
||||||
|
return true;
|
||||||
|
});
|
||||||
console.log(`${fresh.length} new · ${matched.length} match profile · ${fresh.length - matched.length} filtered or already sent`);
|
console.log(`${fresh.length} new · ${matched.length} match profile · ${fresh.length - matched.length} filtered or already sent`);
|
||||||
|
|
||||||
if (!matched.length) {
|
if (!matched.length) {
|
||||||
|
|||||||
+159
-3
@@ -219,6 +219,101 @@ export async function fetchItJobsCh(): Promise<Job[]> {
|
|||||||
return allJobs;
|
return allJobs;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// swisscom.com — Swisscom's own Workday career site. Small board (~47 openings,
|
||||||
|
// Switzerland-wide), so no keyword sweep — pull the full listing and filter
|
||||||
|
// locally, like itjobs.ch. The listing page itself is a bare JS shell (no jobs
|
||||||
|
// in the raw HTML); postings live behind the standard Workday CXS JSON API:
|
||||||
|
// POST /wday/cxs/swisscom/SwisscomExternalCareers/jobs {appliedFacets, limit, offset, searchText}
|
||||||
|
// `total` on the first page (offset 0) is authoritative — on later pages it
|
||||||
|
// comes back as 0 (a quirk of this tenant), and an offset past the true total
|
||||||
|
// silently wraps around and re-serves page 1 instead of an empty list. So the
|
||||||
|
// loop's stopping condition is the first page's total, never a later one's.
|
||||||
|
// Listing rows carry no description, so each title-filtered candidate costs
|
||||||
|
// one extra fetch against the job detail endpoint (`/job{externalPath}`),
|
||||||
|
// which returns the full HTML description — the DE/FR + requiresGerman check
|
||||||
|
// runs on that here, same as job-room.ch/jobbasel.ch.
|
||||||
|
const SWISSCOM_API_BASE = "https://swisscom.wd103.myworkdayjobs.com/wday/cxs/swisscom/SwisscomExternalCareers";
|
||||||
|
|
||||||
|
interface SwisscomListing {
|
||||||
|
title: string;
|
||||||
|
externalPath: string;
|
||||||
|
postedOn: string;
|
||||||
|
bulletFields: string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseSwisscomPostedOn(text: string): string {
|
||||||
|
const iso = (d: Date) => d.toISOString().slice(0, 10);
|
||||||
|
const now = new Date();
|
||||||
|
if (/posted today/i.test(text)) return iso(now);
|
||||||
|
if (/posted yesterday/i.test(text)) {
|
||||||
|
const d = new Date(now);
|
||||||
|
d.setDate(d.getDate() - 1);
|
||||||
|
return iso(d);
|
||||||
|
}
|
||||||
|
const m = text.match(/posted (\d+) days? ago/i);
|
||||||
|
if (m) {
|
||||||
|
const d = new Date(now);
|
||||||
|
d.setDate(d.getDate() - Number(m[1]));
|
||||||
|
return iso(d);
|
||||||
|
}
|
||||||
|
return ""; // "Posted 30+ Days Ago" — no exact date available
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function fetchSwisscom(): Promise<Job[]> {
|
||||||
|
const listings: SwisscomListing[] = [];
|
||||||
|
let total = Infinity;
|
||||||
|
|
||||||
|
for (let offset = 0; offset < total; offset += 20) {
|
||||||
|
if (offset > 0) await Bun.sleep(400);
|
||||||
|
const res = await fetch(`${SWISSCOM_API_BASE}/jobs`, {
|
||||||
|
method: "POST",
|
||||||
|
headers: { ...HEADERS, "Content-Type": "application/json" },
|
||||||
|
body: JSON.stringify({ appliedFacets: {}, limit: 20, offset, searchText: "" }),
|
||||||
|
});
|
||||||
|
if (!res.ok) { console.log(` Swisscom … failed (${res.status})`); break; }
|
||||||
|
|
||||||
|
const data = (await res.json()) as { total: number; jobPostings: SwisscomListing[] };
|
||||||
|
if (offset === 0) total = data.total;
|
||||||
|
if (!data.jobPostings.length) break;
|
||||||
|
listings.push(...data.jobPostings);
|
||||||
|
}
|
||||||
|
|
||||||
|
const jobs: Job[] = [];
|
||||||
|
let kept = 0;
|
||||||
|
let langDropped = 0;
|
||||||
|
for (const item of listings) {
|
||||||
|
const title = decodeHtml(item.title);
|
||||||
|
if (!applyFilters(title)) continue;
|
||||||
|
if (MS_STACK_RE.test(title)) continue;
|
||||||
|
|
||||||
|
await Bun.sleep(300);
|
||||||
|
const detailRes = await fetch(`${SWISSCOM_API_BASE}${item.externalPath}`, { headers: HEADERS });
|
||||||
|
if (!detailRes.ok) continue;
|
||||||
|
|
||||||
|
const detail = (await detailRes.json()) as {
|
||||||
|
jobPostingInfo?: { location?: string; jobDescription?: string; externalUrl?: string };
|
||||||
|
};
|
||||||
|
const info = detail.jobPostingInfo;
|
||||||
|
if (!isViableDescription(info?.jobDescription).ok) { langDropped++; continue; }
|
||||||
|
|
||||||
|
jobs.push({
|
||||||
|
id: `sc-${item.bulletFields?.[0] ?? item.externalPath}`,
|
||||||
|
title,
|
||||||
|
company: "Swisscom",
|
||||||
|
location: info?.location ?? "Switzerland",
|
||||||
|
postedAt: parseSwisscomPostedOn(item.postedOn),
|
||||||
|
link: info?.externalUrl ?? `https://swisscom.wd103.myworkdayjobs.com/SwisscomExternalCareers${item.externalPath}`,
|
||||||
|
search: "swisscom.com",
|
||||||
|
source: "swisscom.com",
|
||||||
|
contentChecked: true, // isViableDescription already ran on the real description
|
||||||
|
});
|
||||||
|
kept++;
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(` Swisscom … ${listings.length} items, ${kept} kept, ${langDropped} non-English`);
|
||||||
|
return jobs;
|
||||||
|
}
|
||||||
|
|
||||||
// job-room.ch — the federal employment service board (RAV / Arbeitsvermittlung).
|
// job-room.ch — the federal employment service board (RAV / Arbeitsvermittlung).
|
||||||
// Roles under the Stellenmeldepflicht must be posted here before anywhere else,
|
// Roles under the Stellenmeldepflicht must be posted here before anywhere else,
|
||||||
// so it often carries a job days ahead of LinkedIn.
|
// so it often carries a job days ahead of LinkedIn.
|
||||||
@@ -291,6 +386,16 @@ function stripTracking(url: string): string {
|
|||||||
|
|
||||||
export async function fetchJobRoom(): Promise<Job[]> {
|
export async function fetchJobRoom(): Promise<Job[]> {
|
||||||
const ads = new Map<string, JobRoomAd>();
|
const ads = new Map<string, JobRoomAd>();
|
||||||
|
// The search API returns a short keyword-anchored teaser per hit, not the
|
||||||
|
// full posting — the same ad re-appears across keyword sweeps with a
|
||||||
|
// different teaser each time, and `ads.set()` above only keeps the latest.
|
||||||
|
// A genuinely German ad (Consult & Pepper's recurring "Senior Fullstack
|
||||||
|
// Software Engineer" — full text: "In Deutsch bist du verhandlungssicher")
|
||||||
|
// slipped through repeatedly because whichever keyword ran last happened to
|
||||||
|
// anchor on an English-reading fragment of it. Collecting every variant and
|
||||||
|
// checking them together means one clean-looking teaser can no longer
|
||||||
|
// override another teaser that actually shows the German.
|
||||||
|
const descVariants = new Map<string, Set<string>>();
|
||||||
|
|
||||||
for (const keyword of JOBROOM_KEYWORDS) {
|
for (const keyword of JOBROOM_KEYWORDS) {
|
||||||
process.stdout.write(` ${keyword} … `);
|
process.stdout.write(` ${keyword} … `);
|
||||||
@@ -313,7 +418,11 @@ export async function fetchJobRoom(): Promise<Job[]> {
|
|||||||
|
|
||||||
const hits = (await res.json()) as Array<{ jobAdvertisement?: JobRoomAd }>;
|
const hits = (await res.json()) as Array<{ jobAdvertisement?: JobRoomAd }>;
|
||||||
for (const h of hits) {
|
for (const h of hits) {
|
||||||
if (h.jobAdvertisement?.id) ads.set(h.jobAdvertisement.id, h.jobAdvertisement);
|
const id = h.jobAdvertisement?.id;
|
||||||
|
if (!id) continue;
|
||||||
|
ads.set(id, h.jobAdvertisement!);
|
||||||
|
const d = h.jobAdvertisement!.jobContent?.jobDescriptions?.[0]?.description;
|
||||||
|
if (d) (descVariants.get(id) ?? descVariants.set(id, new Set()).get(id)!).add(d);
|
||||||
}
|
}
|
||||||
total += hits.length;
|
total += hits.length;
|
||||||
if (hits.length < 100) break;
|
if (hits.length < 100) break;
|
||||||
@@ -341,8 +450,11 @@ export async function fetchJobRoom(): Promise<Job[]> {
|
|||||||
|
|
||||||
// A handful of ads ship without a description. isViableDescription passes
|
// A handful of ads ship without a description. isViableDescription passes
|
||||||
// those through by design, so leave contentChecked unset and let index.ts
|
// those through by design, so leave contentChecked unset and let index.ts
|
||||||
// fall back to fetching the page.
|
// fall back to fetching the page. When we do have variants, join every
|
||||||
const body = desc.description ?? "";
|
// one seen across keyword sweeps rather than just the last teaser stored
|
||||||
|
// on `ad` (see comment above) — more text only adds detection signal, it
|
||||||
|
// never removes any.
|
||||||
|
const body = [...(descVariants.get(ad.id) ?? [])].join(" \n ") || desc.description || "";
|
||||||
const hasBody = body.trim().length > 40;
|
const hasBody = body.trim().length > 40;
|
||||||
if (hasBody && !isViableDescription(body).ok) { langDropped++; continue; }
|
if (hasBody && !isViableDescription(body).ok) { langDropped++; continue; }
|
||||||
|
|
||||||
@@ -521,6 +633,50 @@ export async function fetchBaselIntlOrgs(): Promise<Job[]> {
|
|||||||
return jobs;
|
return jobs;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// sciCORE's open-positions page (formerly polled separately by
|
||||||
|
// scripts/scicore-watch.sh) is a plain-text announcement list, not a job
|
||||||
|
// board: each <li> is either an active posting linked to careers.unibas.ch
|
||||||
|
// or a closed/in-review notice with no link at all. Skip the latter — "the
|
||||||
|
// interview process is still ongoing, but the submission deadline has
|
||||||
|
// passed" is the site's own wording for "not accepting applications" — since
|
||||||
|
// they're not something Rafael could still apply to.
|
||||||
|
export async function fetchSciCore(): Promise<Job[]> {
|
||||||
|
const jobs: Job[] = [];
|
||||||
|
const url = "https://scicore.unibas.ch/about-scicore/open-positions/";
|
||||||
|
const res = await fetch(url, { headers: HEADERS });
|
||||||
|
if (!res.ok) { console.log(` sciCORE … failed (${res.status})`); return jobs; }
|
||||||
|
|
||||||
|
const html = await res.text();
|
||||||
|
const sectionM = html.match(/<h1>Join the team!<\/h1>([\s\S]*?)<footer/i);
|
||||||
|
const items = sectionM ? [...sectionM[1].matchAll(/<li[^>]*>([\s\S]*?)<\/li>/gi)] : [];
|
||||||
|
|
||||||
|
let kept = 0;
|
||||||
|
for (const [, raw] of items) {
|
||||||
|
if (/deadline has passed|interview process is (?:still\s+)?ongoing/i.test(raw)) continue;
|
||||||
|
|
||||||
|
const linkM = raw.match(/<a\s[^>]*href="([^"]+)"[^>]*>([\s\S]*?)<\/a>/i);
|
||||||
|
const title = decodeHtml((linkM ? linkM[2] : raw).replace(/<[^>]+>/g, " "));
|
||||||
|
if (!title || !applyFilters(title)) continue;
|
||||||
|
|
||||||
|
const link = linkM ? linkM[1] : url;
|
||||||
|
jobs.push({
|
||||||
|
id: `scicore-${link.replace(/\W+/g, "").slice(-24)}`,
|
||||||
|
title,
|
||||||
|
company: "sciCORE",
|
||||||
|
location: "Basel, Switzerland",
|
||||||
|
postedAt: "",
|
||||||
|
link,
|
||||||
|
search: "scicore.unibas.ch",
|
||||||
|
source: "scicore.unibas.ch",
|
||||||
|
contentChecked: true, // listing is site-authored English; no separate description to fetch
|
||||||
|
});
|
||||||
|
kept++;
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(` sciCORE … ${items.length} items, ${kept} kept`);
|
||||||
|
return jobs;
|
||||||
|
}
|
||||||
|
|
||||||
// jobbasel.ch — the page shell is a React/Vite SPA (no jobs in the raw HTML
|
// jobbasel.ch — the page shell is a React/Vite SPA (no jobs in the raw HTML
|
||||||
// on the homepage), but the search results route is server-rendered: fetching
|
// on the homepage), but the search results route is server-rendered: fetching
|
||||||
// `/job/suche-{slug}` returns full HTML with a `window.__PRELOADED_STATE__`
|
// `/job/suche-{slug}` returns full HTML with a `window.__PRELOADED_STATE__`
|
||||||
|
|||||||
@@ -200,6 +200,22 @@ const GERMAN_OPTIONAL_RE =
|
|||||||
const GERMAN_REQUIRED_RE =
|
const GERMAN_REQUIRED_RE =
|
||||||
/\b(?:required|require[sd]?|must|mandatory|essential|necessary|fluen\w*|proficien\w*|excellent|strong|very\s+good|good\s+(?:knowledge|command|level)|native|mother\s*tongue|business\s+level|negotiat\w*|B2|C1|C2|verhandlungssicher\w*|fliessend\w*|fließend\w*|muttersprach\w*|sehr\s+gut\w*|gut\w*\s+Deutschkenntnisse|Deutschkenntnisse|voraussetzung|zwingend|erforderlich)\b/i;
|
/\b(?:required|require[sd]?|must|mandatory|essential|necessary|fluen\w*|proficien\w*|excellent|strong|very\s+good|good\s+(?:knowledge|command|level)|native|mother\s*tongue|business\s+level|negotiat\w*|B2|C1|C2|verhandlungssicher\w*|fliessend\w*|fließend\w*|muttersprach\w*|sehr\s+gut\w*|gut\w*\s+Deutschkenntnisse|Deutschkenntnisse|voraussetzung|zwingend|erforderlich)\b/i;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* "German is <verdict>" / "proficiency in German is considered <verdict>" —
|
||||||
|
* the copula directly after the mention names the mention's own predicate,
|
||||||
|
* which should decide the case outright rather than compete on raw distance
|
||||||
|
* with unrelated markers elsewhere in the window. Swisscom's stock line
|
||||||
|
* "Fluency in English is required; proficiency in German is considered an
|
||||||
|
* advantage" put a required-marker word ("proficiency") right next to
|
||||||
|
* "German" and an optional-marker word ("advantage") further away — nearest()
|
||||||
|
* picked "proficiency" and wrongly called it required. The predicate right
|
||||||
|
* after the mention ("is considered an advantage") is what actually decides.
|
||||||
|
* Only fires when a copula immediately follows; "Work proficiency in German,
|
||||||
|
* English and French" (no copula after "German") correctly falls through to
|
||||||
|
* the general nearest-marker logic below.
|
||||||
|
*/
|
||||||
|
const GERMAN_LOCAL_PREDICATE_RE = /^\s*,?\s*(?:is|are)\s+(?:considered\s+|deemed\s+)?/i;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Spec-sheet style language line ("Languages: German & English"), common on
|
* Spec-sheet style language line ("Languages: German & English"), common on
|
||||||
* contractor postings. There is no adjective for the nearest-marker logic to
|
* contractor postings. There is no adjective for the nearest-marker logic to
|
||||||
@@ -261,6 +277,16 @@ export function requiresGerman(text: string): boolean {
|
|||||||
|
|
||||||
for (const m of flat.matchAll(GERMAN_MENTION_RE)) {
|
for (const m of flat.matchAll(GERMAN_MENTION_RE)) {
|
||||||
const i = m.index ?? 0;
|
const i = m.index ?? 0;
|
||||||
|
|
||||||
|
// The mention's own predicate ("German is <verdict>") overrides distance.
|
||||||
|
const after = flat.slice(i + m[0].length, i + m[0].length + 60);
|
||||||
|
const predicateM = after.match(GERMAN_LOCAL_PREDICATE_RE);
|
||||||
|
if (predicateM) {
|
||||||
|
const rest = after.slice(predicateM[0].length);
|
||||||
|
if (new RegExp(`^${GERMAN_OPTIONAL_RE.source}`, "i").test(rest)) continue;
|
||||||
|
if (new RegExp(`^${GERMAN_REQUIRED_RE.source}`, "i").test(rest)) return true;
|
||||||
|
}
|
||||||
|
|
||||||
const from = Math.max(0, i - 110);
|
const from = Math.max(0, i - 110);
|
||||||
const window = flat.slice(from, i + 110);
|
const window = flat.slice(from, i + 110);
|
||||||
const at = i - from; // mention's offset inside the window
|
const at = i - from; // mention's offset inside the window
|
||||||
@@ -275,7 +301,13 @@ export function requiresGerman(text: string): boolean {
|
|||||||
export async function fetchDescription(id: string): Promise<string | null> {
|
export async function fetchDescription(id: string): Promise<string | null> {
|
||||||
for (let attempt = 0; attempt < 3; attempt++) {
|
for (let attempt = 0; attempt < 3; attempt++) {
|
||||||
try {
|
try {
|
||||||
await Bun.sleep(350 * (attempt + 1));
|
// 429s need the same backoff as the listing-page retry (1s/2s/4s) — the
|
||||||
|
// old 350/700/1050ms wait was too short under concurrent load, so this
|
||||||
|
// exhausted its retries while still rate-limited and fell through to the
|
||||||
|
// "can't read it, so keep" default below, letting German postings (e.g.
|
||||||
|
// CLEEVEN, job 4465075383, all-German with zero English content) slip
|
||||||
|
// past isViableContent entirely on 2026-09-10.
|
||||||
|
await Bun.sleep(1000 * 2 ** attempt);
|
||||||
const res = await fetch(`https://www.linkedin.com/jobs/view/${id}/`, { headers: HEADERS });
|
const res = await fetch(`https://www.linkedin.com/jobs/view/${id}/`, { headers: HEADERS });
|
||||||
const html = await res.text();
|
const html = await res.text();
|
||||||
const m = html.match(/class="show-more-less-html__markup[^"]*">([\s\S]{0,8000}?)<\/div>/);
|
const m = html.match(/class="show-more-less-html__markup[^"]*">([\s\S]{0,8000}?)<\/div>/);
|
||||||
@@ -289,10 +321,21 @@ export async function fetchDescription(id: string): Promise<string | null> {
|
|||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** English-language AND not demanding German above B1. */
|
/**
|
||||||
export async function isViableContent(id: string): Promise<{ ok: boolean; reason?: string }> {
|
* English-language AND not demanding German above B1.
|
||||||
|
*
|
||||||
|
* `retry: true` means the description couldn't be read at all (rate-limited,
|
||||||
|
* stub page) — index.ts leaves the job out of the CSV entirely rather than
|
||||||
|
* defaulting to "keep", so it's picked up fresh and re-checked next run
|
||||||
|
* instead of ever being judged sight-unseen. The old default-to-keep here is
|
||||||
|
* what let a fully-German HRM Systems/Consult & Pepper posting ("In Deutsch
|
||||||
|
* bist du verhandlungssicher") through on 2026-09-10 — same failure mode as
|
||||||
|
* the CLEEVEN case above, just on the "keep" side of isViableContent instead
|
||||||
|
* of isEnglishContent.
|
||||||
|
*/
|
||||||
|
export async function isViableContent(id: string): Promise<{ ok: boolean; reason?: string; retry?: boolean }> {
|
||||||
const text = await fetchDescription(id);
|
const text = await fetchDescription(id);
|
||||||
if (text === null) return { ok: true }; // keep when we genuinely can't read it
|
if (text === null) return { ok: false, reason: "unreadable — retrying next run", retry: true };
|
||||||
if (hasNonEnglishUmlaut(text)) return { ok: false, reason: "non-English (umlaut)" };
|
if (hasNonEnglishUmlaut(text)) return { ok: false, reason: "non-English (umlaut)" };
|
||||||
const de = (text.match(DE_CONTENT_RE) ?? []).length;
|
const de = (text.match(DE_CONTENT_RE) ?? []).length;
|
||||||
const fr = (text.match(FR_CONTENT_RE) ?? []).length;
|
const fr = (text.match(FR_CONTENT_RE) ?? []).length;
|
||||||
@@ -307,7 +350,8 @@ export async function isEnglishContent(id: string): Promise<boolean> {
|
|||||||
// false "English" result. Back off and try again before falling back.
|
// false "English" result. Back off and try again before falling back.
|
||||||
for (let attempt = 0; attempt < 3; attempt++) {
|
for (let attempt = 0; attempt < 3; attempt++) {
|
||||||
try {
|
try {
|
||||||
await Bun.sleep(350 * (attempt + 1));
|
// Same 1s/2s/4s backoff as fetchDescription — see its comment.
|
||||||
|
await Bun.sleep(1000 * 2 ** attempt);
|
||||||
const res = await fetch(`https://www.linkedin.com/jobs/view/${id}/`, { headers: HEADERS });
|
const res = await fetch(`https://www.linkedin.com/jobs/view/${id}/`, { headers: HEADERS });
|
||||||
const html = await res.text();
|
const html = await res.text();
|
||||||
const m = html.match(/class="show-more-less-html__markup[^"]*">([\s\S]{0,8000}?)<\/div>/);
|
const m = html.match(/class="show-more-less-html__markup[^"]*">([\s\S]{0,8000}?)<\/div>/);
|
||||||
@@ -337,11 +381,13 @@ export async function isEnglishUrl(url: string): Promise<boolean> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/** As isViableContent, but for an arbitrary job URL (non-LinkedIn sources). */
|
/** As isViableContent, but for an arbitrary job URL (non-LinkedIn sources). */
|
||||||
export async function isViableUrl(url: string): Promise<{ ok: boolean; reason?: string }> {
|
export async function isViableUrl(url: string): Promise<{ ok: boolean; reason?: string; retry?: boolean }> {
|
||||||
try {
|
try {
|
||||||
await Bun.sleep(350);
|
await Bun.sleep(350);
|
||||||
const res = await fetch(url, { headers: HEADERS });
|
const res = await fetch(url, { headers: HEADERS });
|
||||||
if (!res.ok) return { ok: true };
|
// Same reasoning as isViableContent: an unreadable page shouldn't default
|
||||||
|
// to "keep" — defer it and let next run's fresh fetch actually judge it.
|
||||||
|
if (!res.ok) return { ok: false, reason: "unreadable — retrying next run", retry: true };
|
||||||
const text = (await res.text()).replace(/<[^>]+>/g, " ");
|
const text = (await res.text()).replace(/<[^>]+>/g, " ");
|
||||||
if (hasNonEnglishUmlaut(text)) return { ok: false, reason: "non-English (umlaut)" };
|
if (hasNonEnglishUmlaut(text)) return { ok: false, reason: "non-English (umlaut)" };
|
||||||
const de = (text.match(DE_CONTENT_RE) ?? []).length;
|
const de = (text.match(DE_CONTENT_RE) ?? []).length;
|
||||||
@@ -350,7 +396,7 @@ export async function isViableUrl(url: string): Promise<{ ok: boolean; reason?:
|
|||||||
if (requiresGerman(text)) return { ok: false, reason: "requires German > B1" };
|
if (requiresGerman(text)) return { ok: false, reason: "requires German > B1" };
|
||||||
return { ok: true };
|
return { ok: true };
|
||||||
} catch {
|
} catch {
|
||||||
return { ok: true };
|
return { ok: false, reason: "unreadable — retrying next run", retry: true };
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
#!/usr/bin/env bun
|
#!/usr/bin/env bun
|
||||||
import { searches } from "./config.ts";
|
import { searches } from "./config.ts";
|
||||||
import { fetchJobs, isViableContent, isViableUrl } from "./fetch.ts";
|
import { fetchJobs, isViableContent, isViableUrl } from "./fetch.ts";
|
||||||
import { fetchDataCareer, fetchStellenBasel, fetchItJobsCh, fetchJobRoom, fetchBaselIntlOrgs, fetchJobBasel, fetchEnglishJobSearch, fetchBaselJobs } from "./fetch-other.ts";
|
import { fetchDataCareer, fetchStellenBasel, fetchItJobsCh, fetchJobRoom, fetchBaselIntlOrgs, fetchJobBasel, fetchEnglishJobSearch, fetchBaselJobs, fetchSciCore, fetchSwisscom } from "./fetch-other.ts";
|
||||||
|
|
||||||
const CSV = "./results/jobs.csv";
|
const CSV = "./results/jobs.csv";
|
||||||
const OLD_HEADER = "date_found,posted_at,search,title,company,link";
|
const OLD_HEADER = "date_found,posted_at,search,title,company,link";
|
||||||
@@ -62,6 +62,36 @@ const date = new Date().toISOString().slice(0, 10);
|
|||||||
const newRows: string[] = [];
|
const newRows: string[] = [];
|
||||||
let totalLangDropped = 0; // jobs dropped for German/French content — never reach the CSV, so daily.ts can't see them on its own
|
let totalLangDropped = 0; // jobs dropped for German/French content — never reach the CSV, so daily.ts can't see them on its own
|
||||||
|
|
||||||
|
// Persisted across runs so the Telegram digest's "Ignored (German/French)"
|
||||||
|
// count reflects genuinely new drops, not the same posting getting re-flagged
|
||||||
|
// every run (dropped jobs never reach the CSV, so nothing else remembers
|
||||||
|
// them) or the same underlying job counted again each time it resurfaces
|
||||||
|
// under a different link/provider (e.g. Consult & Pepper's "Senior Fullstack
|
||||||
|
// Software Engineer" reposted on job-room.ch every few days, and separately
|
||||||
|
// on LinkedIn) — keyed on normalized title+company rather than the link.
|
||||||
|
const LANG_SEEN = "./results/.lang-dropped-seen.json";
|
||||||
|
const langSeen = new Set<string>(
|
||||||
|
await Bun.file(LANG_SEEN)
|
||||||
|
.json()
|
||||||
|
.catch(() => [])
|
||||||
|
);
|
||||||
|
function langKey(title: string, company: string): string {
|
||||||
|
const t = title
|
||||||
|
.replace(/[([][^)\]]*[)\]]/g, "") // drop "(all genders)", "(m/w/d)", "(a)", "(80 – 100%)" noise
|
||||||
|
.replace(/\s+/g, " ")
|
||||||
|
.trim()
|
||||||
|
.toLowerCase();
|
||||||
|
return `${t}|${company.trim().toLowerCase()}`;
|
||||||
|
}
|
||||||
|
// Only counts (and remembers) a drop the first time this title+company is
|
||||||
|
// seen; still runs for every dropped job so `langSeen` stays complete.
|
||||||
|
function countLangDrop(title: string, company: string): void {
|
||||||
|
const key = langKey(title, company);
|
||||||
|
if (langSeen.has(key)) return;
|
||||||
|
langSeen.add(key);
|
||||||
|
totalLangDropped++;
|
||||||
|
}
|
||||||
|
|
||||||
function csvCell(val: string): string {
|
function csvCell(val: string): string {
|
||||||
return `"${val.replace(/"/g, '""')}"`;
|
return `"${val.replace(/"/g, '""')}"`;
|
||||||
}
|
}
|
||||||
@@ -115,10 +145,17 @@ async function processLinkedInSearch(config: (typeof searches)[number]): Promise
|
|||||||
// One fetch, two checks: written in German/French, or written in English
|
// One fetch, two checks: written in German/French, or written in English
|
||||||
// but demanding German above B1.
|
// but demanding German above B1.
|
||||||
const viable = await isViableContent(job.id);
|
const viable = await isViableContent(job.id);
|
||||||
|
if (viable.retry) {
|
||||||
|
// Couldn't read the description at all — leave it out of the CSV so
|
||||||
|
// next run's fresh fetch judges it properly, instead of defaulting to
|
||||||
|
// "keep" and risking a German-requiring posting slipping through.
|
||||||
|
console.log(viable.reason);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
if (!viable.ok) {
|
if (!viable.ok) {
|
||||||
console.log(`dropped (${viable.reason})`);
|
console.log(`dropped (${viable.reason})`);
|
||||||
langDropped++;
|
langDropped++;
|
||||||
totalLangDropped++;
|
countLangDrop(job.title, job.company);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
console.log("ok");
|
console.log("ok");
|
||||||
@@ -141,6 +178,8 @@ const extraSources: Array<{ label: string; fetch: () => ReturnType<typeof fetchD
|
|||||||
{ label: "jobbasel.ch", fetch: fetchJobBasel },
|
{ label: "jobbasel.ch", fetch: fetchJobBasel },
|
||||||
{ label: "englishjobsearch.ch", fetch: fetchEnglishJobSearch },
|
{ label: "englishjobsearch.ch", fetch: fetchEnglishJobSearch },
|
||||||
{ label: "basel-jobs.ch", fetch: fetchBaselJobs },
|
{ label: "basel-jobs.ch", fetch: fetchBaselJobs },
|
||||||
|
{ label: "scicore.unibas.ch", fetch: fetchSciCore },
|
||||||
|
{ label: "swisscom.com", fetch: fetchSwisscom },
|
||||||
];
|
];
|
||||||
|
|
||||||
async function processExtraSource({ label, fetch: fetchFn }: (typeof extraSources)[number]): Promise<void> {
|
async function processExtraSource({ label, fetch: fetchFn }: (typeof extraSources)[number]): Promise<void> {
|
||||||
@@ -167,10 +206,14 @@ async function processExtraSource({ label, fetch: fetchFn }: (typeof extraSource
|
|||||||
if (!job.contentChecked) {
|
if (!job.contentChecked) {
|
||||||
process.stdout.write(` [${label}] checking content: ${job.title} … `);
|
process.stdout.write(` [${label}] checking content: ${job.title} … `);
|
||||||
const viable = await isViableUrl(job.link);
|
const viable = await isViableUrl(job.link);
|
||||||
|
if (viable.retry) {
|
||||||
|
console.log(viable.reason);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
if (!viable.ok) {
|
if (!viable.ok) {
|
||||||
console.log(`dropped (${viable.reason})`);
|
console.log(`dropped (${viable.reason})`);
|
||||||
langDropped++;
|
langDropped++;
|
||||||
totalLangDropped++;
|
countLangDrop(job.title, job.company);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
console.log("ok");
|
console.log("ok");
|
||||||
@@ -198,6 +241,7 @@ await Promise.all([
|
|||||||
// without listing rejects that were never even parsed into a Row.
|
// without listing rejects that were never even parsed into a Row.
|
||||||
const LANG_DROPPED = "./results/.lang-dropped.json";
|
const LANG_DROPPED = "./results/.lang-dropped.json";
|
||||||
await Bun.write(LANG_DROPPED, JSON.stringify({ date, count: totalLangDropped }));
|
await Bun.write(LANG_DROPPED, JSON.stringify({ date, count: totalLangDropped }));
|
||||||
|
await Bun.write(LANG_SEEN, JSON.stringify([...langSeen]));
|
||||||
|
|
||||||
if (newRows.length === 0) {
|
if (newRows.length === 0) {
|
||||||
console.log("\nNo new jobs.");
|
console.log("\nNo new jobs.");
|
||||||
|
|||||||
Reference in New Issue
Block a user