feat: add sciCORE and Swisscom as job sources

sciCORE now runs inside bun fetch/daily via fetchSciCore instead of its own standalone launchd watcher. Swisscom's Workday career site is added as a new source (fetchSwisscom), pulled via its public CXS JSON API. Also fixes a false positive in requiresGerman(): Swisscom's stock line "proficiency in German is considered an advantage" was being flagged as requiring German because the noun "proficiency" sat closer to the mention than the real optional marker "advantage". A local-predicate override now lets the words immediately following "German is/are ..." decide the verdict instead of losing to raw character distance.
2026-09-18 11:49:49 +02:00
parent 9cbc525cf5
commit 288b8bdcd1
4 changed files with 177 additions and 15 deletions
@@ -21,7 +21,7 @@ Temp scripts: `bun run scripts/<name>-temp.ts` — never edit `config.ts` for on
 ```
 config.ts          # search keywords and areas
 fetch.ts           # scraper, HTML parser, all filters
-fetch-other.ts     # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch)
+fetch-other.ts     # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com)
 index.ts           # orchestrator: runs searches, writes CSV
 recheck.ts         # raw-mode fetch, lists filtered-out jobs
 daily.ts           # digest: fetch → match against profile.ts → send to Telegram
@@ -43,7 +43,7 @@ plans/             # LinkedIn post drafts
  - Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform) and `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform and Field add a Switzerland-wide remote pass.
  - `RESEARCH_KEYWORDS` deliberately contains no bioinformatics/computational-biology terms — see `BIO_DOMAIN_RE`.
 - **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`.
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch). Imported by `index.ts`.
+- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com). Imported by `index.ts`.
  - **datacareer.ch** inverts the usual bot rule: its nginx **403s browser-like User-Agents** and serves honest ones (no UA, `curl/8.0`, Googlebot) fine. The spoofed Chrome UA in the shared `HEADERS` was the entire cause of the long-standing 403 — it uses `PLAIN_HEADERS` instead. Small board: Basel returns **8 jobs on one page with no pagination**, so `?page=2` legitimately yields 0 items.
  - **job-room.ch** carries its language requirement as a **structured field** (`languageSkills[]`, e.g. `{languageIsoCode: "de", spokenLevel: "PROFICIENT"}`), entirely separate from the free-text `jobDescriptions[].description` — the UI renders it as "Deutsch (Fliessend)". `requiresGerman()`/`isViableDescription()` only ever scan the prose, so this field slipped through unchecked until 2026-09-04: measured against 1843 ads, 69 required PROFICIENT-level German and the prose check caught only 3 of them incidentally. `jobRoomRequiresGerman()` now checks the structured field directly (`spokenLevel`/`writtenLevel === "PROFICIENT"`) before the description-based check runs. Levels seen: `NONE`, `BASIC`, `INTERMEDIATE`, `PROFICIENT` — only `PROFICIENT` (fluent/C1+) is dropped, matching the B1 cutoff used everywhere else.
  - **jobbasel.ch** looks like a JS-only React/Vite SPA (no jobs in the homepage's raw HTML), but its search route is server-rendered: `GET /job/suche-{slug}` returns full HTML with a `window.__PRELOADED_STATE__` blob containing the matched vacancies as JSON — no headless browser needed. Pagination is a URL suffix (`-seite-{n}`); a bare `?page=` query param is silently ignored. Swept with the same fixed keyword list as job-room.ch (no keyword-union endpoint exists) and deduped by vacancy id. Each item carries its full German description, so the DE/FR check runs in-process.
@@ -82,6 +82,10 @@ plans/             # LinkedIn post drafts
    - **Basel Institute on Governance** — `baselgovernance.org/about/careers`. Each vacancy is a linked PDF, so there's no HTML description to language-check. The `rounded-3xl` card class is what separates vacancy tiles from the annual-report PDF link.
    - These boards use UN-flavoured titles ("Information Management Officer", "ICT Specialist") that `SOFTWARE_TITLE_RE` alone drops, so `INTL_TECH_RE` widens the net **for these sources only**. Boards this small can afford the noise; missing a rare opening can't.
  - **itjobs.ch** is swept Switzerland-wide (no `location` param) — the board is small, ~210 jobs over 7 pages, and locking it to Basel returned only ~9. It 404s one page past the end instead of serving an empty list. Each item embeds its full `description`, so the DE/FR check runs in-process via `isEnglishDescription` rather than costing a page fetch per job. Use `job_location.name` for location — it's the English form; the bare `location` field is German.
+  - **scicore.unibas.ch** (`fetchSciCore`) replaces the standalone `scripts/scicore-watch.sh` launchd watcher (removed 2026-09-16) — sciCORE now runs inside `bun fetch`/`bun daily` like every other source instead of its own 08:47 agent. The open-positions page is a plain-text announcement list, not a real job board: each `<li>` under `<h1>Join the team!</h1>` is either an active posting linked to `careers.unibas.ch` or a closed/in-review notice with no link — items matching `deadline has passed`/`interview process is (still) ongoing` are skipped since they're not open applications. `contentChecked: true` because the listing itself is the only English text available; there's no separate description page to fetch. Title still has to clear `SOFTWARE_TITLE_RE`, same as every other source — this is deliberately broader than the old watcher's "software"-only match (it also catches "security", matching the IT Security Engineer role Rafael prepped for — see `project_scicore_security_report` in memory) but still drops pure sysadmin titles like "Linux Systems Administrator".
+  - **swisscom.com** (`fetchSwisscom`, added 2026-09-18) — Swisscom's own Workday career site (`swisscom.wd103.myworkdayjobs.com/SwisscomExternalCareers`). Small board (~47 openings, Switzerland-wide), so no keyword sweep — the full listing is pulled and filtered locally, like itjobs.ch. The listing page itself is a bare JS shell with no jobs in the raw HTML; postings live behind the standard Workday CXS JSON API (`POST /wday/cxs/swisscom/SwisscomExternalCareers/jobs`). `total` on the first page (offset 0) is authoritative — later pages report `total: 0` (a tenant quirk), and an offset past the true total silently wraps around and re-serves page 1 instead of an empty list, so the loop's stop condition is the first page's total, never a later one's. Listing rows carry no description, so each title-filtered candidate costs one extra fetch against the job detail endpoint (`/job{externalPath}`), which returns the full HTML description — the DE/FR + `requiresGerman` check runs on that in-process (`contentChecked: true`).
+
+    Swisscom mostly hires in German — of 47 postings, only ~16 clear the English/software title filters, and nearly all of those turned out to be written in German too; only genuinely-English, German-optional roles survive. Adding this source surfaced a real bug in `requiresGerman()`: the stock line *"Fluency in English is required; proficiency in German is considered an advantage"* had the noun "proficiency" (a required-marker word) sitting right next to "German", closer than the real optional-marker "advantage" — nearest() picked the wrong one and wrongly flagged the posting as requiring German. Fixed by adding a local-predicate override (`GERMAN_LOCAL_PREDICATE_RE`): when a copula ("is"/"are", optionally "considered"/"deemed") directly follows the German mention, that immediate predicate decides the verdict outright instead of competing on raw character distance with markers elsewhere in the window — "German is considered an advantage" resolves as optional regardless of what else is nearby. It only fires when a copula immediately follows the mention, so *"Work proficiency in German, English, and French"* (no copula after "German", a genuine hard requirement — Senior Operation Architect (FR/GER/ENG)) still falls through to the general nearest-marker logic and is correctly caught as required. Re-check both cases (and the pre-existing Helvetia/albedis/Karakun/J. Safra Sarasin/BBF/spec-sheet cases in the paragraph above) before touching `requiresGerman` again.
 - **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV. Writes both `search` (the query/source that found the job) and `location` (the posting's own location, as parsed off its card by `fetch.ts`/`fetch-other.ts` — no extra page fetch) to the CSV. Migrates old CSVs on startup: adds `source` if missing, then `location` (backfilled `""` for rows written before the column existed).
  - Every LinkedIn search and every `fetch-other.ts` source hits a different host, so they run **concurrently** (bounded: 3 at a time within each group, both groups started together) instead of one after another — cut a full `bun fetch` from ~16-20 min to well under 2 min. `existingLinks.add()` happens synchronously right after the `.has()` check and before any `await`, which is what makes concurrent workers safe against double-claiming the same link. Concurrency surfaced a real gap: LinkedIn started returning occasional 429s under parallel load, which `fetchJobs` used to treat as end-of-results (silently truncating that search) — `fetchJobs` now retries a 429 up to 3× with backoff (1s/2s/4s) before giving up.
 - **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions.
@@ -203,24 +207,15 @@ Some employers (Optiml, LocalBini seen repeatedly) repost the same role under a

 ## Scheduled automations

-Both run as **launchd agents** — this machine has no cron at all, so don't reach for `crontab`.
+Runs as a **launchd agent** — this machine has no cron at all, so don't reach for `crontab`.

 | plist (`~/Library/LaunchAgents/`) | when | runs |
 |---|---|---|
 | `com.rafapolo.linkme-daily` | 09:00 / 12:00 / 15:00 / 17:00 / 20:00 | `bun run daily.ts` → `results/daily.log` |
-| `ch.polo.scicore-watch` | 08:47 | `scripts/scicore-watch.sh` → `results/.scicore-watch/watch.log` |

-If the Mac is asleep at the scheduled time, launchd fires the job on wake rather than skipping the day. `launchctl list | grep -E 'linkme|scicore'` to check they're loaded; `launchctl unload <plist>` to stop one.
+If the Mac is asleep at the scheduled time, launchd fires the job on wake rather than skipping the day. `launchctl list | grep linkme` to check it's loaded; `launchctl unload <plist>` to stop it.

-**`scripts/scicore-watch.sh`** watches https://scicore.unibas.ch/about-scicore/open-positions/ for a software engineering role Rafael is waiting on. It extracts the "Join the team!" block (keeping each posting's `<a href>` so the alert is tappable), diffs it against `results/.scicore-watch/snapshot.txt`, and notifies **only when a newly added line matches `$WANTED` ("software")**. Every other change still updates the snapshot silently — that's what keeps the eventual real alert clean instead of tangled with months of sysadmin churn. Three consecutive fetch failures alert, so a broken watcher can't look like "no news".
-
-Test the alert path without waiting for the page to change:
-
-```bash
-SCICORE_URL="file://$PWD/fixture.html" bash scripts/scicore-watch.sh
-```
-
-The whole extractor is a `python3 -c` program inside shell single quotes, so **it must not contain a literal apostrophe** — that's why the `<a>` quote class is built with `chr(39)` instead of written inline.
+**`scripts/scicore-watch.sh`** is retired (2026-09-16) — sciCORE is now swept by `fetchSciCore` in `fetch-other.ts` as part of `bun fetch`/`bun daily`, see the `fetch-other.ts` entry above. Its `ch.polo.scicore-watch` launchd agent (08:47) was unloaded and its plist removed; the script and `results/.scicore-watch/` (snapshot + log) are left in the repo for reference/history, not run anymore.

 ## LinkedIn session token

@@ -219,6 +219,101 @@ export async function fetchItJobsCh(): Promise<Job[]> {
  return allJobs;
 }

+// swisscom.com — Swisscom's own Workday career site. Small board (~47 openings,
+// Switzerland-wide), so no keyword sweep — pull the full listing and filter
+// locally, like itjobs.ch. The listing page itself is a bare JS shell (no jobs
+// in the raw HTML); postings live behind the standard Workday CXS JSON API:
+//   POST /wday/cxs/swisscom/SwisscomExternalCareers/jobs {appliedFacets, limit, offset, searchText}
+// `total` on the first page (offset 0) is authoritative — on later pages it
+// comes back as 0 (a quirk of this tenant), and an offset past the true total
+// silently wraps around and re-serves page 1 instead of an empty list. So the
+// loop's stopping condition is the first page's total, never a later one's.
+// Listing rows carry no description, so each title-filtered candidate costs
+// one extra fetch against the job detail endpoint (`/job{externalPath}`),
+// which returns the full HTML description — the DE/FR + requiresGerman check
+// runs on that here, same as job-room.ch/jobbasel.ch.
+const SWISSCOM_API_BASE = "https://swisscom.wd103.myworkdayjobs.com/wday/cxs/swisscom/SwisscomExternalCareers";
+
+interface SwisscomListing {
+  title: string;
+  externalPath: string;
+  postedOn: string;
+  bulletFields: string[];
+}
+
+function parseSwisscomPostedOn(text: string): string {
+  const iso = (d: Date) => d.toISOString().slice(0, 10);
+  const now = new Date();
+  if (/posted today/i.test(text)) return iso(now);
+  if (/posted yesterday/i.test(text)) {
+    const d = new Date(now);
+    d.setDate(d.getDate() - 1);
+    return iso(d);
+  }
+  const m = text.match(/posted (\d+) days? ago/i);
+  if (m) {
+    const d = new Date(now);
+    d.setDate(d.getDate() - Number(m[1]));
+    return iso(d);
+  }
+  return ""; // "Posted 30+ Days Ago" — no exact date available
+}
+
+export async function fetchSwisscom(): Promise<Job[]> {
+  const listings: SwisscomListing[] = [];
+  let total = Infinity;
+
+  for (let offset = 0; offset < total; offset += 20) {
+    if (offset > 0) await Bun.sleep(400);
+    const res = await fetch(`${SWISSCOM_API_BASE}/jobs`, {
+      method: "POST",
+      headers: { ...HEADERS, "Content-Type": "application/json" },
+      body: JSON.stringify({ appliedFacets: {}, limit: 20, offset, searchText: "" }),
+    });
+    if (!res.ok) { console.log(`  Swisscom … failed (${res.status})`); break; }
+
+    const data = (await res.json()) as { total: number; jobPostings: SwisscomListing[] };
+    if (offset === 0) total = data.total;
+    if (!data.jobPostings.length) break;
+    listings.push(...data.jobPostings);
+  }
+
+  const jobs: Job[] = [];
+  let kept = 0;
+  let langDropped = 0;
+  for (const item of listings) {
+    const title = decodeHtml(item.title);
+    if (!applyFilters(title)) continue;
+    if (MS_STACK_RE.test(title)) continue;
+
+    await Bun.sleep(300);
+    const detailRes = await fetch(`${SWISSCOM_API_BASE}${item.externalPath}`, { headers: HEADERS });
+    if (!detailRes.ok) continue;
+
+    const detail = (await detailRes.json()) as {
+      jobPostingInfo?: { location?: string; jobDescription?: string; externalUrl?: string };
+    };
+    const info = detail.jobPostingInfo;
+    if (!isViableDescription(info?.jobDescription).ok) { langDropped++; continue; }
+
+    jobs.push({
+      id: `sc-${item.bulletFields?.[0] ?? item.externalPath}`,
+      title,
+      company: "Swisscom",
+      location: info?.location ?? "Switzerland",
+      postedAt: parseSwisscomPostedOn(item.postedOn),
+      link: info?.externalUrl ?? `https://swisscom.wd103.myworkdayjobs.com/SwisscomExternalCareers${item.externalPath}`,
+      search: "swisscom.com",
+      source: "swisscom.com",
+      contentChecked: true, // isViableDescription already ran on the real description
+    });
+    kept++;
+  }
+
+  console.log(`  Swisscom … ${listings.length} items, ${kept} kept, ${langDropped} non-English`);
+  return jobs;
+}
+
 // job-room.ch — the federal employment service board (RAV / Arbeitsvermittlung).
 // Roles under the Stellenmeldepflicht must be posted here before anywhere else,
 // so it often carries a job days ahead of LinkedIn.
@@ -538,6 +633,50 @@ export async function fetchBaselIntlOrgs(): Promise<Job[]> {
  return jobs;
 }

+// sciCORE's open-positions page (formerly polled separately by
+// scripts/scicore-watch.sh) is a plain-text announcement list, not a job
+// board: each <li> is either an active posting linked to careers.unibas.ch
+// or a closed/in-review notice with no link at all. Skip the latter — "the
+// interview process is still ongoing, but the submission deadline has
+// passed" is the site's own wording for "not accepting applications" — since
+// they're not something Rafael could still apply to.
+export async function fetchSciCore(): Promise<Job[]> {
+  const jobs: Job[] = [];
+  const url = "https://scicore.unibas.ch/about-scicore/open-positions/";
+  const res = await fetch(url, { headers: HEADERS });
+  if (!res.ok) { console.log(`  sciCORE … failed (${res.status})`); return jobs; }
+
+  const html = await res.text();
+  const sectionM = html.match(/<h1>Join the team!<\/h1>([\s\S]*?)<footer/i);
+  const items = sectionM ? [...sectionM[1].matchAll(/<li[^>]*>([\s\S]*?)<\/li>/gi)] : [];
+
+  let kept = 0;
+  for (const [, raw] of items) {
+    if (/deadline has passed|interview process is (?:still\s+)?ongoing/i.test(raw)) continue;
+
+    const linkM = raw.match(/<a\s[^>]*href="([^"]+)"[^>]*>([\s\S]*?)<\/a>/i);
+    const title = decodeHtml((linkM ? linkM[2] : raw).replace(/<[^>]+>/g, " "));
+    if (!title || !applyFilters(title)) continue;
+
+    const link = linkM ? linkM[1] : url;
+    jobs.push({
+      id: `scicore-${link.replace(/\W+/g, "").slice(-24)}`,
+      title,
+      company: "sciCORE",
+      location: "Basel, Switzerland",
+      postedAt: "",
+      link,
+      search: "scicore.unibas.ch",
+      source: "scicore.unibas.ch",
+      contentChecked: true, // listing is site-authored English; no separate description to fetch
+    });
+    kept++;
+  }
+
+  console.log(`  sciCORE … ${items.length} items, ${kept} kept`);
+  return jobs;
+}
+
 // jobbasel.ch — the page shell is a React/Vite SPA (no jobs in the raw HTML
 // on the homepage), but the search results route is server-rendered: fetching
 // `/job/suche-{slug}` returns full HTML with a `window.__PRELOADED_STATE__`
@@ -200,6 +200,22 @@ const GERMAN_OPTIONAL_RE =
 const GERMAN_REQUIRED_RE =
  /\b(?:required|require[sd]?|must|mandatory|essential|necessary|fluen\w*|proficien\w*|excellent|strong|very\s+good|good\s+(?:knowledge|command|level)|native|mother\s*tongue|business\s+level|negotiat\w*|B2|C1|C2|verhandlungssicher\w*|fliessend\w*|fließend\w*|muttersprach\w*|sehr\s+gut\w*|gut\w*\s+Deutschkenntnisse|Deutschkenntnisse|voraussetzung|zwingend|erforderlich)\b/i;

+/**
+ * "German is <verdict>" / "proficiency in German is considered <verdict>" —
+ * the copula directly after the mention names the mention's own predicate,
+ * which should decide the case outright rather than compete on raw distance
+ * with unrelated markers elsewhere in the window. Swisscom's stock line
+ * "Fluency in English is required; proficiency in German is considered an
+ * advantage" put a required-marker word ("proficiency") right next to
+ * "German" and an optional-marker word ("advantage") further away — nearest()
+ * picked "proficiency" and wrongly called it required. The predicate right
+ * after the mention ("is considered an advantage") is what actually decides.
+ * Only fires when a copula immediately follows; "Work proficiency in German,
+ * English and French" (no copula after "German") correctly falls through to
+ * the general nearest-marker logic below.
+ */
+const GERMAN_LOCAL_PREDICATE_RE = /^\s*,?\s*(?:is|are)\s+(?:considered\s+|deemed\s+)?/i;
+
 /**
 * Spec-sheet style language line ("Languages: German & English"), common on
 * contractor postings. There is no adjective for the nearest-marker logic to
@@ -261,6 +277,16 @@ export function requiresGerman(text: string): boolean {

  for (const m of flat.matchAll(GERMAN_MENTION_RE)) {
    const i = m.index ?? 0;
+
+    // The mention's own predicate ("German is <verdict>") overrides distance.
+    const after = flat.slice(i + m[0].length, i + m[0].length + 60);
+    const predicateM = after.match(GERMAN_LOCAL_PREDICATE_RE);
+    if (predicateM) {
+      const rest = after.slice(predicateM[0].length);
+      if (new RegExp(`^${GERMAN_OPTIONAL_RE.source}`, "i").test(rest)) continue;
+      if (new RegExp(`^${GERMAN_REQUIRED_RE.source}`, "i").test(rest)) return true;
+    }
+
    const from = Math.max(0, i - 110);
    const window = flat.slice(from, i + 110);
    const at = i - from; // mention's offset inside the window
@@ -1,7 +1,7 @@
 #!/usr/bin/env bun
 import { searches } from "./config.ts";
 import { fetchJobs, isViableContent, isViableUrl } from "./fetch.ts";
-import { fetchDataCareer, fetchStellenBasel, fetchItJobsCh, fetchJobRoom, fetchBaselIntlOrgs, fetchJobBasel, fetchEnglishJobSearch, fetchBaselJobs } from "./fetch-other.ts";
+import { fetchDataCareer, fetchStellenBasel, fetchItJobsCh, fetchJobRoom, fetchBaselIntlOrgs, fetchJobBasel, fetchEnglishJobSearch, fetchBaselJobs, fetchSciCore, fetchSwisscom } from "./fetch-other.ts";

 const CSV = "./results/jobs.csv";
 const OLD_HEADER = "date_found,posted_at,search,title,company,link";
@@ -178,6 +178,8 @@ const extraSources: Array<{ label: string; fetch: () => ReturnType<typeof fetchD
  { label: "jobbasel.ch", fetch: fetchJobBasel },
  { label: "englishjobsearch.ch", fetch: fetchEnglishJobSearch },
  { label: "basel-jobs.ch", fetch: fetchBaselJobs },
+  { label: "scicore.unibas.ch", fetch: fetchSciCore },
+  { label: "swisscom.com", fetch: fetchSwisscom },
 ];

 async function processExtraSource({ label, fetch: fetchFn }: (typeof extraSources)[number]): Promise<void> {