feat: add novartis.com and more-jobs.ch as job sources
more-jobs.ch lists foreign postings tagged as CH, so drop locations naming another country/city or a 5-digit postcode, and dedupe its reposts by title+company.
This commit is contained in:
@@ -21,7 +21,7 @@ Temp scripts: `bun run scripts/<name>-temp.ts` — never edit `config.ts` for on
|
|||||||
```
|
```
|
||||||
config.ts # search keywords and areas
|
config.ts # search keywords and areas
|
||||||
fetch.ts # scraper, HTML parser, all filters
|
fetch.ts # scraper, HTML parser, all filters
|
||||||
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com)
|
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch)
|
||||||
index.ts # orchestrator: runs searches, writes CSV
|
index.ts # orchestrator: runs searches, writes CSV
|
||||||
recheck.ts # raw-mode fetch, lists filtered-out jobs
|
recheck.ts # raw-mode fetch, lists filtered-out jobs
|
||||||
daily.ts # digest: fetch → match against profile.ts → send to Telegram
|
daily.ts # digest: fetch → match against profile.ts → send to Telegram
|
||||||
@@ -43,7 +43,10 @@ plans/ # LinkedIn post drafts
|
|||||||
- Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform) and `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform and Field add a Switzerland-wide remote pass.
|
- Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform) and `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform and Field add a Switzerland-wide remote pass.
|
||||||
- `RESEARCH_KEYWORDS` deliberately contains no bioinformatics/computational-biology terms — see `BIO_DOMAIN_RE`.
|
- `RESEARCH_KEYWORDS` deliberately contains no bioinformatics/computational-biology terms — see `BIO_DOMAIN_RE`.
|
||||||
- **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`.
|
- **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`.
|
||||||
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com). Imported by `index.ts`.
|
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch). Imported by `index.ts`.
|
||||||
|
- **novartis.com** (`fetchNovartisBasel`, added 2026-09-21) — Rafael has an internal contact at Novartis Basel who can refer him, so this source exists purely to catch a genuine software/engineering opening there, not to widen the CSV generally. Its career-search page (`novartis.com/careers/career-search?country[]=LOC_CH`) is a server-rendered Drupal view — no headless browser needed — but only at the **locale-less English path** (`/careers/career-search`); the `ch-de` German path's job-detail links 302 straight back to the search page for both locales when hit without a session, so detail pages only resolve through the English path. Pagination is 0-indexed (`page=0` implicit, `page=1` is the second page of 10) and the page's own "Showing N results" line gives the exact total, so the page-loop bound is computed rather than probed — the listing has also been observed to **repeat rows across pages within one sweep** (measured 2026-09-21: several titles appeared twice across 5 pages), so results are deduped by `req-<id>` before filtering, not trusted from the pager.
|
||||||
|
|
||||||
|
Novartis Basel's titling is almost entirely non-technical "Director, `<business function>`" roles (Portfolio Analytics, Therapeutic Area Strategy, Policy Communications, ...) plus device/hardware "Engineering" roles (autoinjector/drug-delivery hardware, not software) — the generic `SOFTWARE_TITLE_RE` used by every other source in this file matched **13 of 47** CH listings on title alone in an initial run, and every single one was a false positive (no engineering-team role among them). `NOVARTIS_TITLE_RE` replaces it here: it requires an explicit software/IT engineering job-family term (`software engineer`, `platform engineer`, `data engineer`, `devops`, `sre`, `solutions architect`, ...), not just a bare seniority or business-function word. `BIO_DOMAIN_RE` and job-room.ch's `PHARMA_ROLE_RE` still run on top to catch domain-science titles the stricter regex wouldn't otherwise need to worry about. Narrowed to Basel/Muttenz specifically (`NOVARTIS_BASEL_RE`) — the CH listing also includes Stein (Aargau), a different commute the contact couldn't refer into. As of 2026-09-21 this source keeps **zero** listings — expected; it's watching for the rare opening, not sweeping volume. Each detail page carries the full description inline (`job_details_content_center`…`job_details_content_bottom`), so the DE/FR + `requiresGerman` check runs in-process per candidate rather than costing `index.ts` a page fetch.
|
||||||
- **datacareer.ch** inverts the usual bot rule: its nginx **403s browser-like User-Agents** and serves honest ones (no UA, `curl/8.0`, Googlebot) fine. The spoofed Chrome UA in the shared `HEADERS` was the entire cause of the long-standing 403 — it uses `PLAIN_HEADERS` instead. Small board: Basel returns **8 jobs on one page with no pagination**, so `?page=2` legitimately yields 0 items.
|
- **datacareer.ch** inverts the usual bot rule: its nginx **403s browser-like User-Agents** and serves honest ones (no UA, `curl/8.0`, Googlebot) fine. The spoofed Chrome UA in the shared `HEADERS` was the entire cause of the long-standing 403 — it uses `PLAIN_HEADERS` instead. Small board: Basel returns **8 jobs on one page with no pagination**, so `?page=2` legitimately yields 0 items.
|
||||||
- **job-room.ch** carries its language requirement as a **structured field** (`languageSkills[]`, e.g. `{languageIsoCode: "de", spokenLevel: "PROFICIENT"}`), entirely separate from the free-text `jobDescriptions[].description` — the UI renders it as "Deutsch (Fliessend)". `requiresGerman()`/`isViableDescription()` only ever scan the prose, so this field slipped through unchecked until 2026-09-04: measured against 1843 ads, 69 required PROFICIENT-level German and the prose check caught only 3 of them incidentally. `jobRoomRequiresGerman()` now checks the structured field directly (`spokenLevel`/`writtenLevel === "PROFICIENT"`) before the description-based check runs. Levels seen: `NONE`, `BASIC`, `INTERMEDIATE`, `PROFICIENT` — only `PROFICIENT` (fluent/C1+) is dropped, matching the B1 cutoff used everywhere else.
|
- **job-room.ch** carries its language requirement as a **structured field** (`languageSkills[]`, e.g. `{languageIsoCode: "de", spokenLevel: "PROFICIENT"}`), entirely separate from the free-text `jobDescriptions[].description` — the UI renders it as "Deutsch (Fliessend)". `requiresGerman()`/`isViableDescription()` only ever scan the prose, so this field slipped through unchecked until 2026-09-04: measured against 1843 ads, 69 required PROFICIENT-level German and the prose check caught only 3 of them incidentally. `jobRoomRequiresGerman()` now checks the structured field directly (`spokenLevel`/`writtenLevel === "PROFICIENT"`) before the description-based check runs. Levels seen: `NONE`, `BASIC`, `INTERMEDIATE`, `PROFICIENT` — only `PROFICIENT` (fluent/C1+) is dropped, matching the B1 cutoff used everywhere else.
|
||||||
- **jobbasel.ch** looks like a JS-only React/Vite SPA (no jobs in the homepage's raw HTML), but its search route is server-rendered: `GET /job/suche-{slug}` returns full HTML with a `window.__PRELOADED_STATE__` blob containing the matched vacancies as JSON — no headless browser needed. Pagination is a URL suffix (`-seite-{n}`); a bare `?page=` query param is silently ignored. Swept with the same fixed keyword list as job-room.ch (no keyword-union endpoint exists) and deduped by vacancy id. Each item carries its full German description, so the DE/FR check runs in-process.
|
- **jobbasel.ch** looks like a JS-only React/Vite SPA (no jobs in the homepage's raw HTML), but its search route is server-rendered: `GET /job/suche-{slug}` returns full HTML with a `window.__PRELOADED_STATE__` blob containing the matched vacancies as JSON — no headless browser needed. Pagination is a URL suffix (`-seite-{n}`); a bare `?page=` query param is silently ignored. Swept with the same fixed keyword list as job-room.ch (no keyword-union endpoint exists) and deduped by vacancy id. Each item carries its full German description, so the DE/FR check runs in-process.
|
||||||
@@ -86,6 +89,7 @@ plans/ # LinkedIn post drafts
|
|||||||
- **swisscom.com** (`fetchSwisscom`, added 2026-09-18) — Swisscom's own Workday career site (`swisscom.wd103.myworkdayjobs.com/SwisscomExternalCareers`). Small board (~47 openings, Switzerland-wide), so no keyword sweep — the full listing is pulled and filtered locally, like itjobs.ch. The listing page itself is a bare JS shell with no jobs in the raw HTML; postings live behind the standard Workday CXS JSON API (`POST /wday/cxs/swisscom/SwisscomExternalCareers/jobs`). `total` on the first page (offset 0) is authoritative — later pages report `total: 0` (a tenant quirk), and an offset past the true total silently wraps around and re-serves page 1 instead of an empty list, so the loop's stop condition is the first page's total, never a later one's. Listing rows carry no description, so each title-filtered candidate costs one extra fetch against the job detail endpoint (`/job{externalPath}`), which returns the full HTML description — the DE/FR + `requiresGerman` check runs on that in-process (`contentChecked: true`).
|
- **swisscom.com** (`fetchSwisscom`, added 2026-09-18) — Swisscom's own Workday career site (`swisscom.wd103.myworkdayjobs.com/SwisscomExternalCareers`). Small board (~47 openings, Switzerland-wide), so no keyword sweep — the full listing is pulled and filtered locally, like itjobs.ch. The listing page itself is a bare JS shell with no jobs in the raw HTML; postings live behind the standard Workday CXS JSON API (`POST /wday/cxs/swisscom/SwisscomExternalCareers/jobs`). `total` on the first page (offset 0) is authoritative — later pages report `total: 0` (a tenant quirk), and an offset past the true total silently wraps around and re-serves page 1 instead of an empty list, so the loop's stop condition is the first page's total, never a later one's. Listing rows carry no description, so each title-filtered candidate costs one extra fetch against the job detail endpoint (`/job{externalPath}`), which returns the full HTML description — the DE/FR + `requiresGerman` check runs on that in-process (`contentChecked: true`).
|
||||||
|
|
||||||
Swisscom mostly hires in German — of 47 postings, only ~16 clear the English/software title filters, and nearly all of those turned out to be written in German too; only genuinely-English, German-optional roles survive. Adding this source surfaced a real bug in `requiresGerman()`: the stock line *"Fluency in English is required; proficiency in German is considered an advantage"* had the noun "proficiency" (a required-marker word) sitting right next to "German", closer than the real optional-marker "advantage" — nearest() picked the wrong one and wrongly flagged the posting as requiring German. Fixed by adding a local-predicate override (`GERMAN_LOCAL_PREDICATE_RE`): when a copula ("is"/"are", optionally "considered"/"deemed") directly follows the German mention, that immediate predicate decides the verdict outright instead of competing on raw character distance with markers elsewhere in the window — "German is considered an advantage" resolves as optional regardless of what else is nearby. It only fires when a copula immediately follows the mention, so *"Work proficiency in German, English, and French"* (no copula after "German", a genuine hard requirement — Senior Operation Architect (FR/GER/ENG)) still falls through to the general nearest-marker logic and is correctly caught as required. Re-check both cases (and the pre-existing Helvetia/albedis/Karakun/J. Safra Sarasin/BBF/spec-sheet cases in the paragraph above) before touching `requiresGerman` again.
|
Swisscom mostly hires in German — of 47 postings, only ~16 clear the English/software title filters, and nearly all of those turned out to be written in German too; only genuinely-English, German-optional roles survive. Adding this source surfaced a real bug in `requiresGerman()`: the stock line *"Fluency in English is required; proficiency in German is considered an advantage"* had the noun "proficiency" (a required-marker word) sitting right next to "German", closer than the real optional-marker "advantage" — nearest() picked the wrong one and wrongly flagged the posting as requiring German. Fixed by adding a local-predicate override (`GERMAN_LOCAL_PREDICATE_RE`): when a copula ("is"/"are", optionally "considered"/"deemed") directly follows the German mention, that immediate predicate decides the verdict outright instead of competing on raw character distance with markers elsewhere in the window — "German is considered an advantage" resolves as optional regardless of what else is nearby. It only fires when a copula immediately follows the mention, so *"Work proficiency in German, English, and French"* (no copula after "German", a genuine hard requirement — Senior Operation Architect (FR/GER/ENG)) still falls through to the general nearest-marker logic and is correctly caught as required. Re-check both cases (and the pre-existing Helvetia/albedis/Karakun/J. Safra Sarasin/BBF/spec-sheet cases in the paragraph above) before touching `requiresGerman` again.
|
||||||
|
- **more-jobs.ch** (`fetchMoreJobs`, added 2026-10-02) — server-rendered PHP aggregator. `/recherche?q=<term>&page=n` is fast (~0.5 s) with ~20 cards/page, but `/emplois` and out-of-range pages are slow (20 s+) and **clamp to the last page instead of going empty**, so a term's sweep ends on "no new ids", not an empty list. Swept with `MOREJOBS_TERMS` (≤5 pages each), then each title-filtered candidate gets one detail fetch (4 in parallel, 20 s timeout) for the real `Publication date` (dd.mm.yyyy) and the in-process DE/FR + `requiresGerman` check (`contentChecked: true`). The board **reposts one opening under several IDs** (measured: Accelleron, Swissquote, Bayer ×3), and it also mirrors jobs that LinkedIn/jobs.ch carry, so its `extraSources` entry sets `dedupeByTitle`: `index.ts` skips any job whose `langKey(title, company)` is already in the CSV (or claimed earlier in the same run). Other sources are deliberately not deduped this way. The board also lists **foreign postings** ("Eastern Europe, Poland", "95050 Santa Clara", "London") and its JSON-LD stamps `addressCountry: "CH"` on every one, so the card's location text is the only signal: `MOREJOBS_FOREIGN_RE` drops locations naming another country/foreign city or carrying a 5-digit postcode before the detail fetch (19 such rows had leaked into the CSV by 2026-10-03, several reaching the Telegram digest).
|
||||||
- **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV. Writes both `search` (the query/source that found the job) and `location` (the posting's own location, as parsed off its card by `fetch.ts`/`fetch-other.ts` — no extra page fetch) to the CSV. Migrates old CSVs on startup: adds `source` if missing, then `location` (backfilled `""` for rows written before the column existed).
|
- **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV. Writes both `search` (the query/source that found the job) and `location` (the posting's own location, as parsed off its card by `fetch.ts`/`fetch-other.ts` — no extra page fetch) to the CSV. Migrates old CSVs on startup: adds `source` if missing, then `location` (backfilled `""` for rows written before the column existed).
|
||||||
- Every LinkedIn search and every `fetch-other.ts` source hits a different host, so they run **concurrently** (bounded: 3 at a time within each group, both groups started together) instead of one after another — cut a full `bun fetch` from ~16-20 min to well under 2 min. `existingLinks.add()` happens synchronously right after the `.has()` check and before any `await`, which is what makes concurrent workers safe against double-claiming the same link. Concurrency surfaced a real gap: LinkedIn started returning occasional 429s under parallel load, which `fetchJobs` used to treat as end-of-results (silently truncating that search) — `fetchJobs` now retries a 429 up to 3× with backoff (1s/2s/4s) before giving up.
|
- Every LinkedIn search and every `fetch-other.ts` source hits a different host, so they run **concurrently** (bounded: 3 at a time within each group, both groups started together) instead of one after another — cut a full `bun fetch` from ~16-20 min to well under 2 min. `existingLinks.add()` happens synchronously right after the `.has()` check and before any `await`, which is what makes concurrent workers safe against double-claiming the same link. Concurrency surfaced a real gap: LinkedIn started returning occasional 429s under parallel load, which `fetchJobs` used to treat as end-of-results (silently truncating that search) — `fetchJobs` now retries a 429 up to 3× with backoff (1s/2s/4s) before giving up.
|
||||||
- **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions.
|
- **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions.
|
||||||
|
|||||||
@@ -70,6 +70,7 @@ LinkedIn is swept by `config.ts`'s search families. Everything else lives in `fe
|
|||||||
| **Basel intl orgs** | Basel | BIS (via the UNjobs mirror), Swiss TPH, Basel Institute on Governance |
|
| **Basel intl orgs** | Basel | BIS (via the UNjobs mirror), Swiss TPH, Basel Institute on Governance |
|
||||||
| **scicore.unibas.ch** | Basel | sciCORE's open-positions announcement list; used to be its own separate 08:47 watcher, now just another source |
|
| **scicore.unibas.ch** | Basel | sciCORE's open-positions announcement list; used to be its own separate 08:47 watcher, now just another source |
|
||||||
| **swisscom.com** | Switzerland | Swisscom's own Workday career site, pulled via its public CXS JSON API; small board and mostly German-language, so few survive the filters |
|
| **swisscom.com** | Switzerland | Swisscom's own Workday career site, pulled via its public CXS JSON API; small board and mostly German-language, so few survive the filters |
|
||||||
|
| **more-jobs.ch** | Switzerland | Server-rendered aggregator; keyword sweep of `/recherche?q=`, then one detail fetch per title-filtered candidate (real date + DE/FR check). Reposts of jobs other sources already carry are dropped by normalized title+company |
|
||||||
|
|
||||||
Two of these were dead until recently and are worth knowing about:
|
Two of these were dead until recently and are worth knowing about:
|
||||||
|
|
||||||
|
|||||||
+261
-1
@@ -1,4 +1,4 @@
|
|||||||
import { NON_ENGLISH_RE, JUNIOR_TITLE_RE, SOFTWARE_TITLE_RE, requiresGerman, hasNonEnglishUmlaut } from "./fetch.ts";
|
import { NON_ENGLISH_RE, JUNIOR_TITLE_RE, SOFTWARE_TITLE_RE, BIO_DOMAIN_RE, requiresGerman, hasNonEnglishUmlaut } from "./fetch.ts";
|
||||||
import type { Job } from "./fetch.ts";
|
import type { Job } from "./fetch.ts";
|
||||||
|
|
||||||
const HEADERS = {
|
const HEADERS = {
|
||||||
@@ -864,6 +864,111 @@ export async function fetchEnglishJobSearch(): Promise<Job[]> {
|
|||||||
return jobs;
|
return jobs;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// more-jobs.ch — server-rendered PHP board, an aggregator (the same posting
|
||||||
|
// often also sits on LinkedIn/jobs.ch, deduped by title+company in index.ts).
|
||||||
|
// The site is Swiss-wide with ~20 cards per page; /recherche?q= is fast on the
|
||||||
|
// first pages but deep pages take 20s+ and out-of-range pages clamp to the last
|
||||||
|
// one instead of going empty, so "no new ids" ends a keyword's sweep, not an
|
||||||
|
// empty list. Cards link to /emploi/<slug>-<id>.html; the listing carries no
|
||||||
|
// description, so each title-filtered candidate costs one detail fetch, which
|
||||||
|
// also gives the real publication date (dd.mm.yyyy) and runs the DE/FR check.
|
||||||
|
const MOREJOBS_TERMS = ["software", "engineer", "developer", "devops", "platform", "architect", "data", "python"];
|
||||||
|
const MOREJOBS_MAX_PAGES = 5;
|
||||||
|
// The board is Swiss but lists foreign postings too ("Eastern Europe, Poland",
|
||||||
|
// "95050 Santa Clara", "London"), and its JSON-LD stamps addressCountry "CH" on
|
||||||
|
// all of them — so the card's location text is the only signal. Drop anything
|
||||||
|
// naming another country/region/foreign city, or carrying a 5-digit postcode
|
||||||
|
// (Swiss ones are 4 digits).
|
||||||
|
const MOREJOBS_FOREIGN_RE = new RegExp(
|
||||||
|
"\\b(?:usa|united states|canada|mexico|brazil|india|china|japan|singapore|thailand|australia|" +
|
||||||
|
"united kingdom|uk|england|scotland|ireland|london|cambridge|cambridgeshire|manchester|" +
|
||||||
|
"germany|deutschland|berlin|munich|m(?:ü|u)nchen|frankfurt|hamburg|stuttgart|cologne|k(?:ö|o)ln|" +
|
||||||
|
"france|paris|lyon|annecy|italy|italia|milan|milano|rome|spain|madrid|barcelona|portugal|porto|lisbon|lisboa|" +
|
||||||
|
"austria|(?:ö|o)sterreich|vienna|wien|netherlands|amsterdam|belgium|brussels|luxembourg|denmark|sweden|norway|finland|" +
|
||||||
|
"poland|warsaw|krakow|romania|bucharest|czech|prague|hungary|budapest|bulgaria|sofia|serbia|belgrade|croatia|greece|" +
|
||||||
|
"slovakia|slovenia|ukraine|turkey|israel|dubai|uae)\\b|\\b\\d{5}\\b",
|
||||||
|
"i"
|
||||||
|
);
|
||||||
|
|
||||||
|
function moreJobsLocation(raw: string): string {
|
||||||
|
// Some cards repeat the location twice ("Zurich, Switzerland Zurich, Switzerland").
|
||||||
|
const half = raw.slice(0, raw.length / 2).trim();
|
||||||
|
const loc = raw.length % 2 === 1 && raw === `${half} ${half}` ? half : raw;
|
||||||
|
return loc.replace(/(?:,\s*Switzerland)+$/i, "").replace(/^[,\s]+|[,\s]+$/g, "");
|
||||||
|
}
|
||||||
|
|
||||||
|
async function fetchMoreJobsPage(url: string): Promise<string | null> {
|
||||||
|
for (let attempt = 0; attempt < 2; attempt++) {
|
||||||
|
try {
|
||||||
|
const res = await fetch(url, { headers: HEADERS, signal: AbortSignal.timeout(20_000) });
|
||||||
|
if (res.ok) return await res.text();
|
||||||
|
} catch {}
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function fetchMoreJobs(): Promise<Job[]> {
|
||||||
|
const candidates = new Map<string, { title: string; company: string; location: string; path: string }>();
|
||||||
|
|
||||||
|
for (const term of MOREJOBS_TERMS) {
|
||||||
|
process.stdout.write(` ${term} … `);
|
||||||
|
let found = 0;
|
||||||
|
for (let page = 1; page <= MOREJOBS_MAX_PAGES; page++) {
|
||||||
|
const html = await fetchMoreJobsPage(
|
||||||
|
`https://www.more-jobs.ch/recherche?q=${encodeURIComponent(term)}&page=${page}`
|
||||||
|
);
|
||||||
|
if (!html) break;
|
||||||
|
const cards = [...html.matchAll(
|
||||||
|
/<a class="positions-list-item[^"]*" href="(\/emploi\/[^"]*-(\d+)\.html)"[^>]*>\s*<h2><span>([^<]*)<\/span><\/h2>[\s\S]*?<h3><span[^>]*>([^<]*)<\/span> - <span>([^<]*)<\/span>/g
|
||||||
|
)];
|
||||||
|
let fresh = 0;
|
||||||
|
for (const [, path, id, title, company, location] of cards) {
|
||||||
|
if (candidates.has(id)) continue;
|
||||||
|
fresh++;
|
||||||
|
candidates.set(id, {
|
||||||
|
title: decodeHtml(title.trim()),
|
||||||
|
company: decodeHtml(company.trim()),
|
||||||
|
location: moreJobsLocation(decodeHtml(location.trim())),
|
||||||
|
path,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
found += fresh;
|
||||||
|
if (fresh === 0 || cards.length < 20) break;
|
||||||
|
await Bun.sleep(300);
|
||||||
|
}
|
||||||
|
console.log(`${found} new`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const jobs: Job[] = [];
|
||||||
|
const queue = [...candidates].filter(([, c]) =>
|
||||||
|
applyFilters(c.title) && !MS_STACK_RE.test(c.title) && !MOREJOBS_FOREIGN_RE.test(c.location)
|
||||||
|
);
|
||||||
|
console.log(` ${candidates.size} unique, ${queue.length} pass the title filters — fetching details`);
|
||||||
|
async function worker() {
|
||||||
|
while (queue.length) {
|
||||||
|
const [id, c] = queue.shift()!;
|
||||||
|
const html = await fetchMoreJobsPage(`https://www.more-jobs.ch${c.path}`);
|
||||||
|
if (!html) continue; // next run retries — nothing claimed yet
|
||||||
|
const body = html.match(/<div class="job-content"[^>]*>([\s\S]*?)<\/div>\s*(?:<!--|<div class="clearfix)/)?.[1] ?? "";
|
||||||
|
if (body && !isViableDescription(body).ok) continue;
|
||||||
|
const d = html.match(/Publication date<\/dt>\s*<dd>(\d{2})\.(\d{2})\.(\d{4})/);
|
||||||
|
jobs.push({
|
||||||
|
id: `morejobs-${id}`,
|
||||||
|
title: c.title,
|
||||||
|
company: c.company,
|
||||||
|
location: c.location ? `${c.location}, Switzerland` : "Switzerland",
|
||||||
|
postedAt: d ? `${d[3]}-${d[2]}-${d[1]}` : "",
|
||||||
|
link: `https://www.more-jobs.ch${c.path}`,
|
||||||
|
search: "more-jobs.ch",
|
||||||
|
source: "more-jobs.ch",
|
||||||
|
contentChecked: body.length > 40,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
await Promise.all(Array.from({ length: 4 }, worker));
|
||||||
|
return jobs;
|
||||||
|
}
|
||||||
|
|
||||||
// stellen-basel.ch — WordPress REST API
|
// stellen-basel.ch — WordPress REST API
|
||||||
export async function fetchStellenBasel(): Promise<Job[]> {
|
export async function fetchStellenBasel(): Promise<Job[]> {
|
||||||
const allJobs: Job[] = [];
|
const allJobs: Job[] = [];
|
||||||
@@ -1201,6 +1306,161 @@ function parseBaselJobsCards(
|
|||||||
return { items: items.length, added };
|
return { items: items.length, added };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// novartis.com — Rafael has an internal contact at Novartis Basel who can refer
|
||||||
|
// him, so this source watches specifically for roles that would be worth
|
||||||
|
// flagging to that contact rather than applying cold.
|
||||||
|
//
|
||||||
|
// The career-search page (Drupal Views) is server-rendered HTML, no headless
|
||||||
|
// browser needed — but only at the locale-less path (`/careers/career-search`,
|
||||||
|
// English). The `ch-de` path serves the same listing in German and its
|
||||||
|
// "job/details" links 302 back to the search page for both locales when hit
|
||||||
|
// without a referrer/session, so job detail pages are only readable through
|
||||||
|
// the English path (`/careers/career-search/job/details/req-...`), which
|
||||||
|
// resolves fine and returns the full posting.
|
||||||
|
//
|
||||||
|
// Pagination is 0-indexed (`page=0` is implicit/omitted, `page=1` is the
|
||||||
|
// second page of 10) and the page's own "Showing N results" header gives the
|
||||||
|
// exact total, so the loop bound is computed rather than probed.
|
||||||
|
//
|
||||||
|
// Novartis is 100% pharma, and its Basel HQ titling skews heavily to
|
||||||
|
// non-technical "Director, <business function>" roles (Portfolio Analytics,
|
||||||
|
// Therapeutic Area Strategy, Policy Communications, ...) plus device/hardware
|
||||||
|
// "Engineering" roles (autoinjector/drug-delivery hardware, not software).
|
||||||
|
// SOFTWARE_TITLE_RE's bare "director"/"analytics"/"ai\b" branches — tuned for
|
||||||
|
// smaller, less title-inflated boards — matched every one of those in an
|
||||||
|
// initial run (measured 2026-09-21: 13 of 47 CH listings "kept", none of them
|
||||||
|
// an actual engineering-team role). A dedicated, stricter title match is used
|
||||||
|
// here instead: it requires an explicit software/IT engineering job-family
|
||||||
|
// term, not just a seniority or business-function word BIO_DOMAIN_RE and
|
||||||
|
// PHARMA_ROLE_RE happen to miss.
|
||||||
|
const NOVARTIS_TITLE_RE =
|
||||||
|
/\b(software\s+(engineer|developer|architect)|solutions?\s+architect|platform\s+engineer|cloud\s+engineer|devops(?:\s+engineer)?|data\s+engineer|site\s+reliability\s+engineer|\bsre\b|backend\s+(?:engineer|developer)|frontend\s+(?:engineer|developer)|full.?stack\s+(?:developer|engineer)|machine\s+learning\s+engineer|\bmlops\b|infrastructure\s+engineer|integration\s+engineer|automation\s+engineer|it\s+engineer|systems?\s+engineer|application[s]?\s+engineer|security\s+engineer|ai\s+engineer|engineering\s+manager|technical\s+lead|tech\s+lead|principal\s+(?:software\s+)?engineer|staff\s+engineer|product\s+engineer|programmer|it\s+architect|enterprise\s+architect|data\s+scientist|data\s+platform|digital\s+product\s+(?:engineer|manager)|it\s+project\s+manager|it\s+manager)\b/i;
|
||||||
|
|
||||||
|
// Narrowed to the Basel region specifically (per the source's whole purpose —
|
||||||
|
// the inside contact is there), not all of Switzerland: Novartis CH postings
|
||||||
|
// also list Stein (Aargau) and a handful of remote/blank locations that are a
|
||||||
|
// different commute and not what the contact could refer into.
|
||||||
|
const NOVARTIS_BASE = "https://www.novartis.com";
|
||||||
|
const NOVARTIS_BASEL_RE = /\bbasel\b|\bmuttenz\b/i;
|
||||||
|
|
||||||
|
interface NovartisListing {
|
||||||
|
title: string;
|
||||||
|
href: string;
|
||||||
|
businessUnit: string;
|
||||||
|
location: string;
|
||||||
|
postedAt: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseNovartisDate(text: string): string {
|
||||||
|
const d = new Date(text.trim());
|
||||||
|
return isNaN(d.getTime()) ? "" : d.toISOString().slice(0, 10);
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseNovartisPage(html: string): { total: number; listings: NovartisListing[] } {
|
||||||
|
const totalM = /Showing\s+(\d+)\s+results/i.exec(html);
|
||||||
|
const total = totalM ? Number(totalM[1]) : 0;
|
||||||
|
|
||||||
|
const listings: NovartisListing[] = [];
|
||||||
|
const rows = [...html.matchAll(/<tr class="(?:odd|even)"[^>]*>([\s\S]*?)<\/tr>/g)];
|
||||||
|
for (const [, row] of rows) {
|
||||||
|
const titleM = /views-field-field-job-title"><a href="([^"]+)">([^<]*)<\/a>/.exec(row);
|
||||||
|
const buM = /views-field-field-job-business-unit">([^<]*)<\/td>/.exec(row);
|
||||||
|
const locM = /views-field-field-job-work-location">([^<]*)<\/td>/.exec(row);
|
||||||
|
const dateM = /views-field-field-job-posted-date">([^<]*)<\/td>/.exec(row);
|
||||||
|
if (!titleM) continue;
|
||||||
|
|
||||||
|
listings.push({
|
||||||
|
href: titleM[1],
|
||||||
|
title: decodeHtml(titleM[2]),
|
||||||
|
businessUnit: buM ? decodeHtml(buM[1]) : "",
|
||||||
|
location: locM ? decodeHtml(locM[1]) : "",
|
||||||
|
postedAt: dateM ? parseNovartisDate(dateM[1]) : "",
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return { total, listings };
|
||||||
|
}
|
||||||
|
|
||||||
|
function applyNovartisFilters(title: string): boolean {
|
||||||
|
if (NON_ENGLISH_RE.test(title)) return false;
|
||||||
|
if (JUNIOR_TITLE_RE.test(title)) return false;
|
||||||
|
if (MS_STACK_RE.test(title)) return false;
|
||||||
|
if (BIO_DOMAIN_RE.test(title)) return false;
|
||||||
|
if (PHARMA_ROLE_RE.test(title)) return false;
|
||||||
|
return NOVARTIS_TITLE_RE.test(title);
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function fetchNovartisBasel(): Promise<Job[]> {
|
||||||
|
const jobs: Job[] = [];
|
||||||
|
// The listing has been observed to repeat rows across pages within one
|
||||||
|
// sweep (measured 2026-09-21: several titles appeared twice across the 5
|
||||||
|
// pages of one run) — dedupe by req id rather than trusting the pager.
|
||||||
|
const byReqId = new Map<string, NovartisListing>();
|
||||||
|
|
||||||
|
const firstHtml = await (
|
||||||
|
await fetch(`${NOVARTIS_BASE}/careers/career-search?search_api_fulltext=&country%5B0%5D=LOC_CH&field_job_posted_date=All&op=Submit`, { headers: HEADERS })
|
||||||
|
).text();
|
||||||
|
const { total, listings: firstListings } = parseNovartisPage(firstHtml);
|
||||||
|
const totalPages = Math.max(1, Math.ceil(total / 10));
|
||||||
|
console.log(` Novartis … ${total} CH results, ${totalPages} pages`);
|
||||||
|
for (const item of firstListings) {
|
||||||
|
const reqId = /req-(\d+)/.exec(item.href)?.[1];
|
||||||
|
if (reqId) byReqId.set(reqId, item);
|
||||||
|
}
|
||||||
|
|
||||||
|
for (let page = 1; page < totalPages; page++) {
|
||||||
|
await Bun.sleep(400);
|
||||||
|
const res = await fetch(
|
||||||
|
`${NOVARTIS_BASE}/careers/career-search?search_api_fulltext=&country%5B0%5D=LOC_CH&field_job_posted_date=All&op=Submit&page=${page}`,
|
||||||
|
{ headers: HEADERS }
|
||||||
|
);
|
||||||
|
if (!res.ok) { console.log(` Novartis … page ${page} failed (${res.status})`); break; }
|
||||||
|
const { listings } = parseNovartisPage(await res.text());
|
||||||
|
for (const item of listings) {
|
||||||
|
const reqId = /req-(\d+)/.exec(item.href)?.[1];
|
||||||
|
if (reqId) byReqId.set(reqId, item);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let kept = 0;
|
||||||
|
let langDropped = 0;
|
||||||
|
for (const [reqId, item] of byReqId) {
|
||||||
|
if (!NOVARTIS_BASEL_RE.test(item.location)) continue;
|
||||||
|
if (!applyNovartisFilters(item.title)) continue;
|
||||||
|
|
||||||
|
const link = `${NOVARTIS_BASE}${item.href}`;
|
||||||
|
await Bun.sleep(300);
|
||||||
|
const detailRes = await fetch(link, { headers: HEADERS });
|
||||||
|
let contentChecked = false;
|
||||||
|
if (detailRes.ok) {
|
||||||
|
const detailHtml = await detailRes.text();
|
||||||
|
const start = detailHtml.indexOf("job_details_content_center");
|
||||||
|
const end = detailHtml.indexOf("job_details_content_bottom");
|
||||||
|
if (start !== -1 && end !== -1) {
|
||||||
|
const description = detailHtml.slice(start, end).replace(/<[^>]+>/g, " ");
|
||||||
|
const check = isViableDescription(description);
|
||||||
|
if (!check.ok) { langDropped++; continue; }
|
||||||
|
contentChecked = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
jobs.push({
|
||||||
|
id: `nov-${reqId}`,
|
||||||
|
title: item.title,
|
||||||
|
company: "Novartis",
|
||||||
|
location: `${item.location}, Switzerland`,
|
||||||
|
postedAt: item.postedAt,
|
||||||
|
link,
|
||||||
|
search: "novartis.com",
|
||||||
|
source: "novartis.com",
|
||||||
|
contentChecked,
|
||||||
|
});
|
||||||
|
kept++;
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(` Novartis … ${byReqId.size} unique CH listings, ${kept} kept (Basel-area + profile match), ${langDropped} non-English/German-required dropped`);
|
||||||
|
return jobs;
|
||||||
|
}
|
||||||
|
|
||||||
export async function fetchBaselJobs(): Promise<Job[]> {
|
export async function fetchBaselJobs(): Promise<Job[]> {
|
||||||
const allJobs: Job[] = [];
|
const allJobs: Job[] = [];
|
||||||
const seen = new Set<string>();
|
const seen = new Set<string>();
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
#!/usr/bin/env bun
|
#!/usr/bin/env bun
|
||||||
import { searches } from "./config.ts";
|
import { searches } from "./config.ts";
|
||||||
import { fetchJobs, isViableContent, isViableUrl } from "./fetch.ts";
|
import { fetchJobs, isViableContent, isViableUrl } from "./fetch.ts";
|
||||||
import { fetchDataCareer, fetchStellenBasel, fetchItJobsCh, fetchJobRoom, fetchBaselIntlOrgs, fetchJobBasel, fetchEnglishJobSearch, fetchBaselJobs, fetchSciCore, fetchSwisscom } from "./fetch-other.ts";
|
import { fetchDataCareer, fetchStellenBasel, fetchItJobsCh, fetchJobRoom, fetchBaselIntlOrgs, fetchJobBasel, fetchEnglishJobSearch, fetchBaselJobs, fetchSciCore, fetchSwisscom, fetchNovartisBasel, fetchMoreJobs } from "./fetch-other.ts";
|
||||||
|
|
||||||
const CSV = "./results/jobs.csv";
|
const CSV = "./results/jobs.csv";
|
||||||
const OLD_HEADER = "date_found,posted_at,search,title,company,link";
|
const OLD_HEADER = "date_found,posted_at,search,title,company,link";
|
||||||
@@ -47,12 +47,17 @@ if (csvExists) {
|
|||||||
// posting on every run (found 2026-09-08: 4 duplicate rows for one link,
|
// posting on every run (found 2026-09-08: 4 duplicate rows for one link,
|
||||||
// added on both 09-07 and 09-08).
|
// added on both 09-07 and 09-08).
|
||||||
const existingLinks = new Set<string>();
|
const existingLinks = new Set<string>();
|
||||||
|
// Normalized title+company of every row already in the CSV, for aggregator
|
||||||
|
// sources that repost jobs other sources carry (see `dedupeByTitle`).
|
||||||
|
const knownJobKeys = new Set<string>();
|
||||||
try {
|
try {
|
||||||
const text = await Bun.file(CSV).text();
|
const text = await Bun.file(CSV).text();
|
||||||
for (const line of text.split("\n").slice(1)) {
|
for (const line of text.split("\n").slice(1)) {
|
||||||
if (!line.trim()) continue;
|
if (!line.trim()) continue;
|
||||||
const link = line.replace(/^"|"$/g, "").split('","')[5];
|
const cells = line.replace(/^"|"$/g, "").split('","');
|
||||||
|
const link = cells[5];
|
||||||
if (link) existingLinks.add(link);
|
if (link) existingLinks.add(link);
|
||||||
|
if (cells[3]) knownJobKeys.add(langKey(cells[3], cells[4] ?? ""));
|
||||||
}
|
}
|
||||||
} catch {
|
} catch {
|
||||||
// first run
|
// first run
|
||||||
@@ -159,6 +164,7 @@ async function processLinkedInSearch(config: (typeof searches)[number]): Promise
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
console.log("ok");
|
console.log("ok");
|
||||||
|
knownJobKeys.add(langKey(job.title, job.company));
|
||||||
newRows.push(
|
newRows.push(
|
||||||
[date, job.postedAt, config.label, job.title, job.company, job.link, "linkedin.com", job.location]
|
[date, job.postedAt, config.label, job.title, job.company, job.link, "linkedin.com", job.location]
|
||||||
.map(csvCell)
|
.map(csvCell)
|
||||||
@@ -169,7 +175,7 @@ async function processLinkedInSearch(config: (typeof searches)[number]): Promise
|
|||||||
console.log(` [${config.label}] ${jobs.length} found, ${added} new, ${langDropped} non-English dropped`);
|
console.log(` [${config.label}] ${jobs.length} found, ${added} new, ${langDropped} non-English dropped`);
|
||||||
}
|
}
|
||||||
|
|
||||||
const extraSources: Array<{ label: string; fetch: () => ReturnType<typeof fetchDataCareer> }> = [
|
const extraSources: Array<{ label: string; fetch: () => ReturnType<typeof fetchDataCareer>; dedupeByTitle?: boolean }> = [
|
||||||
{ label: "datacareer.ch", fetch: fetchDataCareer },
|
{ label: "datacareer.ch", fetch: fetchDataCareer },
|
||||||
{ label: "stellen-basel.ch", fetch: fetchStellenBasel },
|
{ label: "stellen-basel.ch", fetch: fetchStellenBasel },
|
||||||
{ label: "itjobs.ch", fetch: fetchItJobsCh },
|
{ label: "itjobs.ch", fetch: fetchItJobsCh },
|
||||||
@@ -180,9 +186,13 @@ const extraSources: Array<{ label: string; fetch: () => ReturnType<typeof fetchD
|
|||||||
{ label: "basel-jobs.ch", fetch: fetchBaselJobs },
|
{ label: "basel-jobs.ch", fetch: fetchBaselJobs },
|
||||||
{ label: "scicore.unibas.ch", fetch: fetchSciCore },
|
{ label: "scicore.unibas.ch", fetch: fetchSciCore },
|
||||||
{ label: "swisscom.com", fetch: fetchSwisscom },
|
{ label: "swisscom.com", fetch: fetchSwisscom },
|
||||||
|
{ label: "novartis.com", fetch: fetchNovartisBasel },
|
||||||
|
// Aggregator: reposts what LinkedIn/jobs.ch/etc. already carry, so also skip
|
||||||
|
// anything whose normalized title+company is already in the CSV.
|
||||||
|
{ label: "more-jobs.ch", fetch: fetchMoreJobs, dedupeByTitle: true },
|
||||||
];
|
];
|
||||||
|
|
||||||
async function processExtraSource({ label, fetch: fetchFn }: (typeof extraSources)[number]): Promise<void> {
|
async function processExtraSource({ label, fetch: fetchFn, dedupeByTitle }: (typeof extraSources)[number]): Promise<void> {
|
||||||
console.log(`\n[${label}]`);
|
console.log(`\n[${label}]`);
|
||||||
let jobs;
|
let jobs;
|
||||||
try {
|
try {
|
||||||
@@ -202,6 +212,11 @@ async function processExtraSource({ label, fetch: fetchFn }: (typeof extraSource
|
|||||||
for (const job of jobs) {
|
for (const job of jobs) {
|
||||||
if (existingLinks.has(job.link)) continue;
|
if (existingLinks.has(job.link)) continue;
|
||||||
existingLinks.add(job.link); // claim before the await — see note above
|
existingLinks.add(job.link); // claim before the await — see note above
|
||||||
|
if (dedupeByTitle) {
|
||||||
|
const key = langKey(job.title, job.company);
|
||||||
|
if (knownJobKeys.has(key)) continue;
|
||||||
|
knownJobKeys.add(key); // same synchronous-claim rule as the link
|
||||||
|
}
|
||||||
|
|
||||||
if (!job.contentChecked) {
|
if (!job.contentChecked) {
|
||||||
process.stdout.write(` [${label}] checking content: ${job.title} … `);
|
process.stdout.write(` [${label}] checking content: ${job.title} … `);
|
||||||
@@ -218,6 +233,7 @@ async function processExtraSource({ label, fetch: fetchFn }: (typeof extraSource
|
|||||||
}
|
}
|
||||||
console.log("ok");
|
console.log("ok");
|
||||||
}
|
}
|
||||||
|
knownJobKeys.add(langKey(job.title, job.company));
|
||||||
newRows.push(
|
newRows.push(
|
||||||
[date, job.postedAt, job.search, job.title, job.company, job.link, job.source, job.location]
|
[date, job.postedAt, job.search, job.title, job.company, job.link, job.source, job.location]
|
||||||
.map(csvCell)
|
.map(csvCell)
|
||||||
|
|||||||
+1
-1
@@ -9,6 +9,6 @@
|
|||||||
},
|
},
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"playwright-core": "^1.60.0"
|
"playwright-core": "^1.63.0"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user