Compare commits

...
4 Commits
7 changed files with 177 additions and 14 deletions
+7 -4
View File
@@ -21,7 +21,7 @@ Temp scripts: `bun run scripts/<name>-temp.ts` — never edit `config.ts` for on
``` ```
config.ts # search keywords and areas config.ts # search keywords and areas
fetch.ts # scraper, HTML parser, all filters fetch.ts # scraper, HTML parser, all filters
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch, startup.jobs) fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch + data-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch, startup.jobs, arbeitnow.ch)
index.ts # orchestrator: runs searches, writes CSV index.ts # orchestrator: runs searches, writes CSV
recheck.ts # raw-mode fetch, lists filtered-out jobs recheck.ts # raw-mode fetch, lists filtered-out jobs
daily.ts # digest: fetch → match against profile.ts → send to Telegram daily.ts # digest: fetch → match against profile.ts → send to Telegram
@@ -40,10 +40,10 @@ plans/ # LinkedIn post drafts
## Architecture ## Architecture
- **`config.ts`** — `KEYWORDS` string (joined with ` OR `) and `searches: SearchConfig[]`. Edit here to add/remove search areas or keywords. - **`config.ts`** — `KEYWORDS` string (joined with ` OR `) and `searches: SearchConfig[]`. Edit here to add/remove search areas or keywords.
- Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform) and `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform and Field add a Switzerland-wide remote pass. - Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform), `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product) and `FRONTEND_KEYWORDS` (frontend/React/web, data visualisation, geospatial/GIS, WebGL — added 2026-10-20 after a one-off search found 14 of 18 relevant frontend/viz roles missing from the CSV). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform, Field and Frontend add a Switzerland-wide remote pass.
- `RESEARCH_KEYWORDS` deliberately contains no bioinformatics/computational-biology terms — see `BIO_DOMAIN_RE`. - `RESEARCH_KEYWORDS` deliberately contains no bioinformatics/computational-biology terms — see `BIO_DOMAIN_RE`.
- **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`. - **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`.
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch, startup.jobs). Imported by `index.ts`. - **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch + data-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch, startup.jobs, arbeitnow.ch). Imported by `index.ts`.
- **novartis.com** (`fetchNovartisBasel`, added 2026-09-21) — Rafael has an internal contact at Novartis Basel who can refer him, so this source exists purely to catch a genuine software/engineering opening there, not to widen the CSV generally. Its career-search page (`novartis.com/careers/career-search?country[]=LOC_CH`) is a server-rendered Drupal view — no headless browser needed — but only at the **locale-less English path** (`/careers/career-search`); the `ch-de` German path's job-detail links 302 straight back to the search page for both locales when hit without a session, so detail pages only resolve through the English path. Pagination is 0-indexed (`page=0` implicit, `page=1` is the second page of 10) and the page's own "Showing N results" line gives the exact total, so the page-loop bound is computed rather than probed — the listing has also been observed to **repeat rows across pages within one sweep** (measured 2026-09-21: several titles appeared twice across 5 pages), so results are deduped by `req-<id>` before filtering, not trusted from the pager. - **novartis.com** (`fetchNovartisBasel`, added 2026-09-21) — Rafael has an internal contact at Novartis Basel who can refer him, so this source exists purely to catch a genuine software/engineering opening there, not to widen the CSV generally. Its career-search page (`novartis.com/careers/career-search?country[]=LOC_CH`) is a server-rendered Drupal view — no headless browser needed — but only at the **locale-less English path** (`/careers/career-search`); the `ch-de` German path's job-detail links 302 straight back to the search page for both locales when hit without a session, so detail pages only resolve through the English path. Pagination is 0-indexed (`page=0` implicit, `page=1` is the second page of 10) and the page's own "Showing N results" line gives the exact total, so the page-loop bound is computed rather than probed — the listing has also been observed to **repeat rows across pages within one sweep** (measured 2026-09-21: several titles appeared twice across 5 pages), so results are deduped by `req-<id>` before filtering, not trusted from the pager.
Novartis Basel's titling is almost entirely non-technical "Director, `<business function>`" roles (Portfolio Analytics, Therapeutic Area Strategy, Policy Communications, ...) plus device/hardware "Engineering" roles (autoinjector/drug-delivery hardware, not software) — the generic `SOFTWARE_TITLE_RE` used by every other source in this file matched **13 of 47** CH listings on title alone in an initial run, and every single one was a false positive (no engineering-team role among them). `NOVARTIS_TITLE_RE` replaces it here: it requires an explicit software/IT engineering job-family term (`software engineer`, `platform engineer`, `data engineer`, `devops`, `sre`, `solutions architect`, ...), not just a bare seniority or business-function word. `BIO_DOMAIN_RE` and job-room.ch's `PHARMA_ROLE_RE` still run on top to catch domain-science titles the stricter regex wouldn't otherwise need to worry about. Narrowed to Basel/Muttenz specifically (`NOVARTIS_BASEL_RE`) — the CH listing also includes Stein (Aargau), a different commute the contact couldn't refer into. As of 2026-09-21 this source keeps **zero** listings — expected; it's watching for the rare opening, not sweeping volume. Each detail page carries the full description inline (`job_details_content_center`…`job_details_content_bottom`), so the DE/FR + `requiresGerman` check runs in-process per candidate rather than costing `index.ts` a page fetch. Novartis Basel's titling is almost entirely non-technical "Director, `<business function>`" roles (Portfolio Analytics, Therapeutic Area Strategy, Policy Communications, ...) plus device/hardware "Engineering" roles (autoinjector/drug-delivery hardware, not software) — the generic `SOFTWARE_TITLE_RE` used by every other source in this file matched **13 of 47** CH listings on title alone in an initial run, and every single one was a false positive (no engineering-team role among them). `NOVARTIS_TITLE_RE` replaces it here: it requires an explicit software/IT engineering job-family term (`software engineer`, `platform engineer`, `data engineer`, `devops`, `sre`, `solutions architect`, ...), not just a bare seniority or business-function word. `BIO_DOMAIN_RE` and job-room.ch's `PHARMA_ROLE_RE` still run on top to catch domain-science titles the stricter regex wouldn't otherwise need to worry about. Narrowed to Basel/Muttenz specifically (`NOVARTIS_BASEL_RE`) — the CH listing also includes Stein (Aargau), a different commute the contact couldn't refer into. As of 2026-09-21 this source keeps **zero** listings — expected; it's watching for the rare opening, not sweeping volume. Each detail page carries the full description inline (`job_details_content_center`…`job_details_content_bottom`), so the DE/FR + `requiresGerman` check runs in-process per candidate rather than costing `index.ts` a page fetch.
@@ -78,6 +78,8 @@ plans/ # LinkedIn post drafts
Pagination/dedup: page 4+ of the category still serves 2 sponsored slots, so **"no new ids" is the end condition**, not an empty list. Sponsored cards repeat within a page, across pages, and across both passes — dedupe on `data-gtm-jid`. Cards carry no date, so `postedAt` is the discovery date. Canton comes from `data-gtm-jp` (sometimes `"BL,BS"`). Pagination/dedup: page 4+ of the category still serves 2 sponsored slots, so **"no new ids" is the end condition**, not an empty list. Sponsored cards repeat within a page, across pages, and across both passes — dedupe on `data-gtm-jid`. Cards carry no date, so `postedAt` is the discovery date. Canton comes from `data-gtm-jp` (sometimes `"BL,BS"`).
**Pass 3: data-jobs.ch** (added 2026-10-22) — the same jobchannel network's data/analytics board, swept for BS/BL/ZH/BE/LU (`DATAJOBS_PROVINCES`) inside `fetchBaselJobs`: same card markup and parser, and the same DataDome cookie clears both hosts (a cookie rotated by one host works on the other). `/search?provinces=…&lang=en-ch&page=n` walks the board's own listings (100 per page; past the end it still serves a few sponsored cards, so "no new ids" ends the walk), then one `&page=1&websearch=1` request returns all the "further jobs from the web search" (it ignores `page`). Measured 2026-10-22: 166 own + 98 web search = 264 unique for the five cantons. Vacancy ids are network-wide, so it runs last and shares `seen` — a posting on both boards keeps its basel-jobs.ch link. Its inventory is **not** a subset of basel-jobs.ch (data-only postings 404 there): measured 2026-10-22, 26 ids the basel-jobs passes missed. A posting on both boards keeps the same `/job/<slug>/<id>` path, so `knownJobchannelTails()` skips any card whose path is already in the CSV under either host — before its detail fetch, not after. Re-checking every known posting on every run is what got the cookie blocked on the first five-canton test (2026-10-22). For the same reason a detail page that fails to load is now **skipped** (retried next run) rather than kept unchecked: a blocked cookie fails every remaining page at once. Rows get `source`/`search` = `data-jobs.ch`.
**Language-check the `div.plain-content` block only, never the whole page.** The site chrome is German and on its own clears the ≥ 6 DE/FR word threshold — checking the full page dropped *every* job, English ones included. That block is the mobile rendering of the description; the desktop view iframes the same text, so it's the only copy in the server-rendered HTML. **Language-check the `div.plain-content` block only, never the whole page.** The site chrome is German and on its own clears the ≥ 6 DE/FR word threshold — checking the full page dropped *every* job, English ones included. That block is the mobile rendering of the description; the desktop view iframes the same text, so it's the only copy in the server-rendered HTML.
- **Basel international orgs** (`fetchBaselIntlOrgs`) sweeps three boards under one label, each only a handful of postings: - **Basel international orgs** (`fetchBaselIntlOrgs`) sweeps three boards under one label, each only a handful of postings:
- **BIS** — the career site is JS-rendered with no usable API, so we read the UNjobs mirror at `unjobs.org/offices/bis_182320855689338880`. Titles arrive as `Role, City, Country` and get split back apart. BIS also cycles Senior Software / Data / Data & Analytics Engineer roles through this board. - **BIS** — the career site is JS-rendered with no usable API, so we read the UNjobs mirror at `unjobs.org/offices/bis_182320855689338880`. Titles arrive as `Role, City, Country` and get split back apart. BIS also cycles Senior Software / Data / Data & Analytics Engineer roles through this board.
@@ -93,10 +95,11 @@ plans/ # LinkedIn post drafts
- **startup.jobs** (`fetchStartupJobs`, added 2026-10-06) — global startup board, swept Switzerland-wide. **Every path is behind a Cloudflare managed challenge** (403 on plain HTTP for any UA, feeds and `/api` included), and unlike basel-jobs.ch's DataDome cookie, `cf_clearance` **does not replay** over Bun's fetch — measured 403 with the exact cookie + UA; it's bound to Chrome's TLS fingerprint. What works: Chrome `--headless=new` with a throwaway profile clears the challenge **only if the UA has no "HeadlessChrome" token** (the version is read from `Chrome --version` so it doesn't drift). So this source launches a headless browser on every run — unlike basel-jobs.ch's minting, that works fine under launchd. `LINKME_NO_BROWSER=1` disables it; any failure logs `unavailable` and returns `[]`. - **startup.jobs** (`fetchStartupJobs`, added 2026-10-06) — global startup board, swept Switzerland-wide. **Every path is behind a Cloudflare managed challenge** (403 on plain HTTP for any UA, feeds and `/api` included), and unlike basel-jobs.ch's DataDome cookie, `cf_clearance` **does not replay** over Bun's fetch — measured 403 with the exact cookie + UA; it's bound to Chrome's TLS fingerprint. What works: Chrome `--headless=new` with a throwaway profile clears the challenge **only if the UA has no "HeadlessChrome" token** (the version is read from `Chrome --version` so it doesn't drift). So this source launches a headless browser on every run — unlike basel-jobs.ch's minting, that works fine under launchd. `LINKME_NO_BROWSER=1` disables it; any failure logs `unavailable` and returns `[]`.
Listings come from **Algolia** (app `4CQMTMMK73`, index `Post_production`), not the HTML: the page embeds a *secured* search key in `meta[name="current-algolia-api-key-search"]`, and algolia.net is not behind Cloudflare, so the queries run over plain fetch. The key's `validUntil` is **~3h after page load** — read fresh every run, never cached. The key rejects every country filter (`country_code`/`country_id` → 0 hits), so the sweep is a geo bounding box around Switzerland with `country_code === "CH"` checked locally (the box also catches DE/FR/AT/LI/IT), limited to the last 60 days. Algolia caps a query at **500 retrievable hits** (100 × 5 pages) — a box over 500 is split into quadrants recursively (all ~1400 CH-box postings vs 556 in the 60-day window, measured 2026-10-06). `tagFilters` on `_tags` (Developer/Engineer) was tried and dropped: it covered only 365 of them. Records carry no description: each title-filtered candidate's detail page is fetched *inside* the browser (`page.evaluate(fetch)`, same TLS) and its JSON-LD `JobPosting.description` goes through `isViableDescription` (`contentChecked: true`). Links already in `results/jobs.csv` are skipped before that fetch. It aggregates company ATS feeds (Wise, Stripe, Avaloq, Nebius…) that LinkedIn also carries, so its `extraSources` entry sets `dedupeByTitle` like more-jobs.ch. The user's original URL was `?q=Developer&w=hybrid` around Basel; the source deliberately keeps on-site/remote too and uses `SOFTWARE_TITLE_RE` instead of the `Developer` query, matching every other source. Listings come from **Algolia** (app `4CQMTMMK73`, index `Post_production`), not the HTML: the page embeds a *secured* search key in `meta[name="current-algolia-api-key-search"]`, and algolia.net is not behind Cloudflare, so the queries run over plain fetch. The key's `validUntil` is **~3h after page load** — read fresh every run, never cached. The key rejects every country filter (`country_code`/`country_id` → 0 hits), so the sweep is a geo bounding box around Switzerland with `country_code === "CH"` checked locally (the box also catches DE/FR/AT/LI/IT), limited to the last 60 days. Algolia caps a query at **500 retrievable hits** (100 × 5 pages) — a box over 500 is split into quadrants recursively (all ~1400 CH-box postings vs 556 in the 60-day window, measured 2026-10-06). `tagFilters` on `_tags` (Developer/Engineer) was tried and dropped: it covered only 365 of them. Records carry no description: each title-filtered candidate's detail page is fetched *inside* the browser (`page.evaluate(fetch)`, same TLS) and its JSON-LD `JobPosting.description` goes through `isViableDescription` (`contentChecked: true`). Links already in `results/jobs.csv` are skipped before that fetch. It aggregates company ATS feeds (Wise, Stripe, Avaloq, Nebius…) that LinkedIn also carries, so its `extraSources` entry sets `dedupeByTitle` like more-jobs.ch. The user's original URL was `?q=Developer&w=hybrid` around Basel; the source deliberately keeps on-site/remote too and uses `SOFTWARE_TITLE_RE` instead of the `Developer` query, matching every other source.
- **arbeitnow.ch** (`fetchArbeitnow`, added 2026-11-03) — free public JSON API (`/api/job-board-api?page=n`, 100/page, no auth), fed straight from company ATS feeds. Rolling ~7-day window (~195 jobs), so 2 requests cover it. Full description inline → in-process DE/FR + `requiresGerman` check. `dedupeByTitle` (overlaps ~50% with LinkedIn). **pilotjob.ch was evaluated and rejected**: its listing only offers `src=arbeitnow|jobroom`, i.e. it re-serves this feed plus job-room.ch. englishjobsearch.ch/jobs/remote shares only 1 of arbeitnow's 19 remote postings.
- **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV. Writes both `search` (the query/source that found the job) and `location` (the posting's own location, as parsed off its card by `fetch.ts`/`fetch-other.ts` — no extra page fetch) to the CSV. Migrates old CSVs on startup: adds `source` if missing, then `location` (backfilled `""` for rows written before the column existed). - **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV. Writes both `search` (the query/source that found the job) and `location` (the posting's own location, as parsed off its card by `fetch.ts`/`fetch-other.ts` — no extra page fetch) to the CSV. Migrates old CSVs on startup: adds `source` if missing, then `location` (backfilled `""` for rows written before the column existed).
- Every LinkedIn search and every `fetch-other.ts` source hits a different host, so they run **concurrently** (bounded: 3 at a time within each group, both groups started together) instead of one after another — cut a full `bun fetch` from ~16-20 min to well under 2 min. `existingLinks.add()` happens synchronously right after the `.has()` check and before any `await`, which is what makes concurrent workers safe against double-claiming the same link. Concurrency surfaced a real gap: LinkedIn started returning occasional 429s under parallel load, which `fetchJobs` used to treat as end-of-results (silently truncating that search) — `fetchJobs` now retries a 429 up to 3× with backoff (1s/2s/4s) before giving up. - Every LinkedIn search and every `fetch-other.ts` source hits a different host, so they run **concurrently** (bounded: 3 at a time within each group, both groups started together) instead of one after another — cut a full `bun fetch` from ~16-20 min to well under 2 min. `existingLinks.add()` happens synchronously right after the `.has()` check and before any `await`, which is what makes concurrent workers safe against double-claiming the same link. Concurrency surfaced a real gap: LinkedIn started returning occasional 429s under parallel load, which `fetchJobs` used to treat as end-of-results (silently truncating that search) — `fetchJobs` now retries a 429 up to 3× with backoff (1s/2s/4s) before giving up.
- **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions. - **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions.
- **`daily.ts`** — the notification layer on top of `index.ts`. Snapshots the CSV's links, shells out to `bun fetch`, then diffs to find what's new. Survivors are filtered through `matchesProfile` and grouped by area (Basel / Remote / Zürich / Bern) so the message scans on a phone. `results/sent.txt` is written **only after a successful send**, so a Telegram failure retries next run instead of silently losing jobs. A non-zero exit from `bun fetch` sends a failure notice rather than failing quietly. - **`daily.ts`** — the notification layer on top of `index.ts`. Snapshots the CSV's links, shells out to `bun fetch`, then diffs to find what's new. Survivors are filtered through `matchesProfile` and grouped by area (Basel / Remote / Zürich / Bern / Luzern) so the message scans on a phone. `results/sent.txt` is written **only after a successful send**, so a Telegram failure retries next run instead of silently losing jobs. A non-zero exit from `bun fetch` sends a failure notice rather than failing quietly.
- The area grouping matches against the CSV's `location` column (the posting's real location) first, falling back to `search` (the query/source label) only when `location` is empty — i.e. for rows written before the `location` column existed. This fixed two real mislabels: a LinkedIn "Bern" sweep can surface a job actually based in Lausanne (search label ≠ real location), and `fetch-other.ts` sources (job-room.ch, itjobs.ch, stellen-basel.ch, datacareer.ch) write the source name into `search` instead of a city, which used to dump their jobs into "⚪️ Other" regardless of where they actually were. - The area grouping matches against the CSV's `location` column (the posting's real location) first, falling back to `search` (the query/source label) only when `location` is empty — i.e. for rows written before the `location` column existed. This fixed two real mislabels: a LinkedIn "Bern" sweep can surface a job actually based in Lausanne (search label ≠ real location), and `fetch-other.ts` sources (job-room.ch, itjobs.ch, stellen-basel.ch, datacareer.ch) write the source name into `search` instead of a city, which used to dump their jobs into "⚪️ Other" regardless of where they actually were.
- **`profile.ts`** — `matchesProfile(title, company, minScore = 4)`. Hard rejects run first (Microsoft stack, Java/C++/SAP, life-science, non-software functions, blocked companies Jobgether and ALLPS), then `SIGNALS` scores stack and seniority terms from `cv/polo.md`. This is a **second, stricter gate than the fetch filters**: `fetch.ts` decides what lands in the CSV, `profile.ts` decides what's worth a phone buzz. Widen `SOFTWARE_TITLE_RE` to catch more in the CSV; tune `SIGNALS`/`minScore` to change what gets sent. - **`profile.ts`** — `matchesProfile(title, company, minScore = 4)`. Hard rejects run first (Microsoft stack, Java/C++/SAP, life-science, non-software functions, blocked companies Jobgether and ALLPS), then `SIGNALS` scores stack and seniority terms from `cv/polo.md`. This is a **second, stricter gate than the fetch filters**: `fetch.ts` decides what lands in the CSV, `profile.ts` decides what's worth a phone buzz. Widen `SOFTWARE_TITLE_RE` to catch more in the CSV; tune `SIGNALS`/`minScore` to change what gets sent.
- **`telegram.ts`** — reads `.env` directly (no dotenv dep). Accepts `TELEGRAM_BOT` or `TELEGRAM_TOKEN` for the token. Splits at 3900 chars on line boundaries (Telegram hard-caps at 4096) and sleeps 400 ms between parts to stay under the rate limit. Messages are HTML — run titles/companies through `escapeHtml`. - **`telegram.ts`** — reads `.env` directly (no dotenv dep). Accepts `TELEGRAM_BOT` or `TELEGRAM_TOKEN` for the token. Splits at 3900 chars on line boundaries (Telegram hard-caps at 4096) and sleeps 400 ms between parts to stay under the rate limit. Messages are HTML — run titles/companies through `escapeHtml`.
+17 -1
View File
@@ -184,6 +184,21 @@ export const searches: SearchConfig[] = [
}, },
]; ];
/** Frontend, web and data-visualisation / geospatial — the Cantography lane. */
export const FRONTEND_KEYWORDS = [
"Frontend Engineer",
"Front-end Developer",
"React",
"React Developer",
"Full-Stack Developer",
"Web Developer",
"Data Visualization",
"Geospatial",
"GIS Developer",
"WebGL",
"Creative Technologist",
].join(" OR ");
// --- Role-family sweeps --------------------------------------------------- // --- Role-family sweeps ---------------------------------------------------
// Same filters as the core searches, run across a narrower set of areas. // Same filters as the core searches, run across a narrower set of areas.
// Research is Basel/Zürich only (that's where the institutes are); platform // Research is Basel/Zürich only (that's where the institutes are); platform
@@ -226,5 +241,6 @@ function sweep(
searches.push( searches.push(
...sweep("Platform", PLATFORM_KEYWORDS, CORE_AREAS, { remote: true }), ...sweep("Platform", PLATFORM_KEYWORDS, CORE_AREAS, { remote: true }),
...sweep("Research", RESEARCH_KEYWORDS, RESEARCH_AREAS), ...sweep("Research", RESEARCH_KEYWORDS, RESEARCH_AREAS),
...sweep("Field", FIELD_KEYWORDS, CORE_AREAS, { remote: true }) ...sweep("Field", FIELD_KEYWORDS, CORE_AREAS, { remote: true }),
...sweep("Frontend", FRONTEND_KEYWORDS, CORE_AREAS, { remote: true })
); );
+2 -1
View File
@@ -133,9 +133,10 @@ const area = (r: Row) => {
if (text.includes("basel")) return "🟢 Basel"; if (text.includes("basel")) return "🟢 Basel";
if (text.includes("zürich") || text.includes("zurich")) return "🔵 Zürich"; if (text.includes("zürich") || text.includes("zurich")) return "🔵 Zürich";
if (text.includes("bern")) return "🟣 Bern"; if (text.includes("bern")) return "🟣 Bern";
if (text.includes("luzern") || text.includes("lucerne")) return "🟠 Luzern";
return "⚪️ Other"; return "⚪️ Other";
}; };
const order = ["🟢 Basel", "🌍 Remote", "🔵 Zürich", "🟣 Bern", "⚪️ Other"]; const order = ["🟢 Basel", "🌍 Remote", "🔵 Zürich", "🟣 Bern", "🟠 Luzern", "⚪️ Other"];
const groups = new Map<string, Row[]>(); const groups = new Map<string, Row[]>();
for (const r of matched) { for (const r of matched) {
const k = area(r); const k = area(r);
+121 -6
View File
@@ -221,6 +221,60 @@ export async function fetchItJobsCh(): Promise<Job[]> {
return allJobs; return allJobs;
} }
// arbeitnow.ch — Swiss instance of arbeitnow, which pulls postings straight from
// company ATS feeds (Personio, Greenhouse, …). Free public JSON API, no auth:
// /api/job-board-api?page=n, 100 per page, newest first, `links.next` null on
// the last page. It's a rolling ~7-day window (measured 2026-11-03: 195 jobs,
// oldest 7 days old), so 2 requests cover the whole board. Each item carries
// the full HTML description, so the DE/FR + requiresGerman check runs here.
// pilotjob.ch was evaluated first and is NOT a source in its own right: its
// listing only offers `src=arbeitnow|jobroom`, i.e. it re-serves this feed plus
// job-room.ch, which we already sweep.
export async function fetchArbeitnow(): Promise<Job[]> {
const allJobs: Job[] = [];
const seen = new Set<string>();
for (let page = 1; page <= 20; page++) {
if (page > 1) await Bun.sleep(500);
process.stdout.write(` page ${page} … `);
const res = await fetch(`https://www.arbeitnow.ch/api/job-board-api?page=${page}`, {
headers: { ...PLAIN_HEADERS, Accept: "application/json" },
});
if (!res.ok) { console.log(`end (${res.status})`); break; }
const data = (await res.json()) as {
data: Array<{ slug: string; company_name: string; title: string; description?: string; remote: boolean; url: string; location: string; created_at: number }>;
links: { next: string | null };
};
let kept = 0;
let langDropped = 0;
for (const item of data.data) {
const title = decodeHtml(item.title);
if (!applyFilters(title) || MS_STACK_RE.test(title) || BIO_DOMAIN_RE.test(title)) continue;
const id = `an-${item.slug}`;
if (seen.has(id)) continue;
seen.add(id);
if (!isViableDescription(item.description).ok) { langDropped++; continue; }
const loc = decodeHtml(item.location ?? "");
allJobs.push({
id,
title,
company: decodeHtml(item.company_name ?? ""),
location: /^remote/i.test(loc) || (item.remote && !loc) ? "Remote" : loc || "Switzerland",
postedAt: item.created_at ? new Date(item.created_at * 1000).toISOString().slice(0, 10) : "",
link: item.url,
search: "arbeitnow.ch",
source: "arbeitnow.ch",
contentChecked: true,
});
kept++;
}
console.log(`${data.data.length} items, ${kept} kept, ${langDropped} non-English`);
if (!data.links?.next || data.data.length === 0) break;
}
return allJobs;
}
// swisscom.com — Swisscom's own Workday career site. Small board (~47 openings, // swisscom.com — Swisscom's own Workday career site. Small board (~47 openings,
// Switzerland-wide), so no keyword sweep — pull the full listing and filter // Switzerland-wide), so no keyword sweep — pull the full listing and filter
// locally, like itjobs.ch. The listing page itself is a bare JS shell (no jobs // locally, like itjobs.ch. The listing page itself is a bare JS shell (no jobs
@@ -1054,6 +1108,12 @@ export async function fetchStellenBasel(): Promise<Job[]> {
const DATADOME_CACHE = "results/.datadome.json"; const DATADOME_CACHE = "results/.datadome.json";
const CHROME_BIN = "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"; const CHROME_BIN = "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome";
const BASELJOBS_ORIGIN = "https://basel-jobs.ch"; const BASELJOBS_ORIGIN = "https://basel-jobs.ch";
// Sister board on the same jobchannel network: same markup, same DataDome
// customer (one cookie, rotated or not, clears both hosts), same `/job/<slug>/<id>`
// path for a posting listed on both. Its inventory is not a subset of
// basel-jobs.ch, though — data-only postings 404 on the basel-jobs.ch host.
const DATAJOBS_ORIGIN = "https://data-jobs.ch";
const DATAJOBS_PROVINCES = "BS,BL,ZH,BE,LU";
interface DataDomeSession { interface DataDomeSession {
cookie: string; cookie: string;
@@ -1258,14 +1318,16 @@ function parseBaselJobsCards(
html: string, html: string,
seen: Set<string>, seen: Set<string>,
out: Job[], out: Job[],
today: string today: string,
host = "basel-jobs.ch",
knownTails = new Map<string, string>()
): { items: number; added: number } { ): { items: number; added: number } {
const items = [...html.matchAll(/<li class="item"(.*?)<\/li>/gs)].map((m) => m[1]); const items = [...html.matchAll(/<li class="item"(.*?)<\/li>/gs)].map((m) => m[1]);
let added = 0; let added = 0;
for (const card of items) { for (const card of items) {
const jid = /data-gtm-jid="(\d+)"/.exec(card)?.[1]; const jid = /data-gtm-jid="(\d+)"/.exec(card)?.[1];
const linkM = /href="(https:\/\/basel-jobs\.ch\/job\/[^"]+)"/.exec(card); const linkM = /href="(https:\/\/(?:basel|data)-jobs\.ch\/job\/[^"]+)"/.exec(card);
const titleM = /<span class="jobtitle">(.*?)<\/span>/s.exec(card); const titleM = /<span class="jobtitle">(.*?)<\/span>/s.exec(card);
if (!jid || !linkM || !titleM) continue; if (!jid || !linkM || !titleM) continue;
@@ -1276,6 +1338,14 @@ function parseBaselJobsCards(
seen.add(id); seen.add(id);
added++; added++;
// Already in the CSV, under either host (a posting carried by both boards
// shares its `/job/<slug>/<id>` tail, which index.ts's link dedupe can't
// see). Skipping here also spares the detail fetch below: re-checking every
// known posting on every run is what got the cookie blocked once
// data-jobs.ch's five cantons were added (measured 2026-10-22).
const tail = linkM[1].replace(/^https:\/\/[^/]+/, "");
if (knownTails.has(tail)) continue;
const title = decodeHtml(titleM[1]); const title = decodeHtml(titleM[1]);
if (!applyFilters(title)) continue; if (!applyFilters(title)) continue;
@@ -1300,8 +1370,8 @@ function parseBaselJobsCards(
// best available, same as any board that hides posting age. // best available, same as any board that hides posting age.
postedAt: today, postedAt: today,
link: linkM[1], link: linkM[1],
search: "basel-jobs.ch", search: host,
source: "basel-jobs.ch", source: host,
}); });
} }
@@ -1463,11 +1533,24 @@ export async function fetchNovartisBasel(): Promise<Job[]> {
return jobs; return jobs;
} }
/** `/job/<slug>/<id>` → stored link, for every basel-jobs.ch/data-jobs.ch row in the CSV. */
async function knownJobchannelTails(): Promise<Map<string, string>> {
const tails = new Map<string, string>();
try {
const text = await Bun.file("results/jobs.csv").text();
for (const link of text.match(/https:\/\/(?:basel|data)-jobs\.ch\/job\/[^\s",]+/g) ?? []) {
tails.set(link.replace(/^https:\/\/[^/]+/, ""), link);
}
} catch { /* no CSV yet */ }
return tails;
}
export async function fetchBaselJobs(): Promise<Job[]> { export async function fetchBaselJobs(): Promise<Job[]> {
const allJobs: Job[] = []; const allJobs: Job[] = [];
const seen = new Set<string>(); const seen = new Set<string>();
const state = { session: await loadDataDomeSession(), minted: false }; const state = { session: await loadDataDomeSession(), minted: false };
const today = new Date().toISOString().slice(0, 10); const today = new Date().toISOString().slice(0, 10);
const knownTails = await knownJobchannelTails();
// Pass 1 — the board's IT category, ~300 postings over 3 pages. // Pass 1 — the board's IT category, ~300 postings over 3 pages.
for (let page = 1; page <= 10; page++) { for (let page = 1; page <= 10; page++) {
@@ -1523,11 +1606,40 @@ export async function fetchBaselJobs(): Promise<Job[]> {
const html = await baselJobsFetch(`${BASELJOBS_ORIGIN}/search?${qs}`, state); const html = await baselJobsFetch(`${BASELJOBS_ORIGIN}/search?${qs}`, state);
if (html === null) { console.log("unavailable"); break; } if (html === null) { console.log("unavailable"); break; }
const { items, added } = parseBaselJobsCards(html, seen, allJobs, today); const { items, added } = parseBaselJobsCards(html, seen, allJobs, today, "basel-jobs.ch", knownTails);
console.log(`${items} items, ${added} new`); console.log(`${items} items, ${added} new`);
} }
} }
// Pass 3 — data-jobs.ch, the network's data/analytics board, filtered to
// the cantons Rafael would work in. Runs after the basel-jobs.ch passes and
// shares `seen` (vacancy ids are network-wide), so a posting on both boards
// keeps its basel-jobs.ch link. "N jobs found" is the board's own listings
// plus the "further jobs from the web search" behind `websearch=1`. The own
// listings paginate (`page=n`, 100 per page) and past the end still serve a
// few sponsored cards, so "no new ids" ends the walk; the web search ignores
// `page` and returns everything at once. Measured 2026-10-22 for
// BS,BL,ZH,BE,LU: 166 own + 98 web search = 264 unique; for BS,BL alone only
// 7 of 24 own listings were already in the CSV via basel-jobs.ch.
const dataJobsSearch = `${DATAJOBS_ORIGIN}/search?provinces=${DATAJOBS_PROVINCES}&lang=en-ch`;
for (let page = 1; page <= 10; page++) {
await Bun.sleep(400);
process.stdout.write(` data-jobs.ch page ${page} … `);
const html = await baselJobsFetch(`${dataJobsSearch}&page=${page}`, state);
if (html === null) { console.log("unavailable"); break; }
const { items, added } = parseBaselJobsCards(html, seen, allJobs, today, "data-jobs.ch", knownTails);
console.log(`${items} items, ${added} new`);
if (items === 0 || added === 0) break;
}
await Bun.sleep(400);
process.stdout.write(` data-jobs.ch websearch … `);
const webHtml = await baselJobsFetch(`${dataJobsSearch}&page=1&websearch=1`, state);
if (webHtml === null) console.log("unavailable");
else {
const { items, added } = parseBaselJobsCards(webHtml, seen, allJobs, today, "data-jobs.ch", knownTails);
console.log(`${items} items, ${added} new`);
}
// Detail pages carry the full description inline, so the DE/FR and // Detail pages carry the full description inline, so the DE/FR and
// German-requirement checks run here instead of costing index.ts a fetch that // German-requirement checks run here instead of costing index.ts a fetch that
// would hit DataDome without the cookie and be dropped as unreachable. // would hit DataDome without the cookie and be dropped as unreachable.
@@ -1536,7 +1648,10 @@ export async function fetchBaselJobs(): Promise<Job[]> {
await Bun.sleep(300); await Bun.sleep(300);
process.stdout.write(` checking ${job.title} … `); process.stdout.write(` checking ${job.title} … `);
const page = await baselJobsFetch(job.link, state); const page = await baselJobsFetch(job.link, state);
if (page === null) { console.log("unreachable, keeping"); viable.push(job); continue; } // Not kept: a blocked cookie fails every remaining detail page at once, and
// keeping those would land them in the CSV unchecked for good. Left out,
// they're simply retried on the next run.
if (page === null) { console.log("unreachable, skipping"); continue; }
// Only the posting's own text, never the page. basel-jobs.ch is a German // Only the posting's own text, never the page. basel-jobs.ch is a German
// site, so its nav/footer/category chrome alone clears the ≥ 6 DE/FR word // site, so its nav/footer/category chrome alone clears the ≥ 6 DE/FR word
+1 -1
View File
@@ -161,7 +161,7 @@ export const BIO_DOMAIN_RE = new RegExp(
"i" "i"
); );
export const SOFTWARE_TITLE_RE = /\b(software|developer|architect|devops|devsecops|platform|data|backend|frontend|fullstack|full.stack|tech\s*lead|technical|typescript|python|rails|node\.?js|ai\b|agentic|llm|machine\s*learning|cloud|security|analytics|sre|reliability|infrastructure|mlops|solutions?|consultant|applications?\s*engineer|system[\s-]*engineer|informatics|scientific\s*(?:software|comput\w*|programmer)|research\s*(?:software|engineer)|hpc|forward\s*deployed|fde|palantir|field\s*engineer|customer\s*engineer|sales\s*engineer|pre.?sales|integration\s*engineer|api\s*engineer|founding\s*engineer|product\s*engineer|automation\s*engineer|director|product\s*manager|engineering\s*manager|it\s*manager|it\s*project|staff\s*engineer|principal\s*engineer|cto|vp\s*eng)\b/i; export const SOFTWARE_TITLE_RE = /\b(software|developer|architect|devops|devsecops|platform|data|backend|frontend|front.end|fullstack|full.stack|react|web\s*developer|gis|geospatial|visuali[sz]ation|webgl|graphics|creative\s*technologist|cartograph\w*|tech\s*lead|technical|typescript|python|rails|node\.?js|ai\b|agentic|llm|machine\s*learning|cloud|security|analytics|sre|reliability|infrastructure|mlops|solutions?|consultant|applications?\s*engineer|system[\s-]*engineer|informatics|scientific\s*(?:software|comput\w*|programmer)|research\s*(?:software|engineer)|hpc|forward\s*deployed|fde|palantir|field\s*engineer|customer\s*engineer|sales\s*engineer|pre.?sales|integration\s*engineer|api\s*engineer|founding\s*engineer|product\s*engineer|automation\s*engineer|director|product\s*manager|engineering\s*manager|it\s*manager|it\s*project|staff\s*engineer|principal\s*engineer|cto|vp\s*eng)\b/i;
const DE_CONTENT_RE = /\b(Sie|Ihre?|Ihnen|Wir|wir|und\s|für|mit\s|von\s|bei\s|das\s|ist\s|sind\s|haben\s|werden\s|können|suchen|bieten|Kenntnisse|Erfahrung|Anforderungen|Aufgaben|Stellenbeschreibung|Stellenangebot|Bewerbung|Ausbildung|Berufserfahrung|Teamarbeit|Vollzeit|Teilzeit|unbefristet|Abschluss|gesucht|Deutsch|Englisch|Schweizer)\b/g; const DE_CONTENT_RE = /\b(Sie|Ihre?|Ihnen|Wir|wir|und\s|für|mit\s|von\s|bei\s|das\s|ist\s|sind\s|haben\s|werden\s|können|suchen|bieten|Kenntnisse|Erfahrung|Anforderungen|Aufgaben|Stellenbeschreibung|Stellenangebot|Bewerbung|Ausbildung|Berufserfahrung|Teamarbeit|Vollzeit|Teilzeit|unbefristet|Abschluss|gesucht|Deutsch|Englisch|Schweizer)\b/g;
const FR_CONTENT_RE = /\b(vous|nous|votre|notre|pour\s|avec\s|profil|poste|emploi|entreprise|compétences|expérience|formation|équipe|rejoindre|recherchons|offrons|missions|candidature|CDI|CDD|temps\s+plein|temps\s+partiel)\b/g; const FR_CONTENT_RE = /\b(vous|nous|votre|notre|pour\s|avec\s|profil|poste|emploi|entreprise|compétences|expérience|formation|équipe|rejoindre|recherchons|offrons|missions|candidature|CDI|CDD|temps\s+plein|temps\s+partiel)\b/g;
+3 -1
View File
@@ -2,7 +2,7 @@
import { searches } from "./config.ts"; import { searches } from "./config.ts";
import { parseCsvLine } from "./csv.ts"; import { parseCsvLine } from "./csv.ts";
import { fetchJobs, isViableContent, isViableUrl } from "./fetch.ts"; import { fetchJobs, isViableContent, isViableUrl } from "./fetch.ts";
import { fetchDataCareer, fetchStellenBasel, fetchItJobsCh, fetchJobRoom, fetchBaselIntlOrgs, fetchJobBasel, fetchEnglishJobSearch, fetchBaselJobs, fetchSciCore, fetchSwisscom, fetchNovartisBasel, fetchMoreJobs, fetchStartupJobs } from "./fetch-other.ts"; import { fetchDataCareer, fetchStellenBasel, fetchItJobsCh, fetchJobRoom, fetchBaselIntlOrgs, fetchJobBasel, fetchEnglishJobSearch, fetchBaselJobs, fetchSciCore, fetchSwisscom, fetchNovartisBasel, fetchMoreJobs, fetchStartupJobs, fetchArbeitnow } from "./fetch-other.ts";
const CSV = "./results/jobs.csv"; const CSV = "./results/jobs.csv";
const OLD_HEADER = "date_found,posted_at,search,title,company,link"; const OLD_HEADER = "date_found,posted_at,search,title,company,link";
@@ -193,6 +193,8 @@ const extraSources: Array<{ label: string; fetch: () => ReturnType<typeof fetchD
{ label: "more-jobs.ch", fetch: fetchMoreJobs, dedupeByTitle: true }, { label: "more-jobs.ch", fetch: fetchMoreJobs, dedupeByTitle: true },
// Aggregator of company ATS feeds (Wise, Stripe, Avaloq…) that LinkedIn also carries. // Aggregator of company ATS feeds (Wise, Stripe, Avaloq…) that LinkedIn also carries.
{ label: "startup.jobs", fetch: fetchStartupJobs, dedupeByTitle: true }, { label: "startup.jobs", fetch: fetchStartupJobs, dedupeByTitle: true },
// ATS-fed board that overlaps ~50% with LinkedIn (measured 2026-11-03).
{ label: "arbeitnow.ch", fetch: fetchArbeitnow, dedupeByTitle: true },
]; ];
async function processExtraSource({ label, fetch: fetchFn, dedupeByTitle }: (typeof extraSources)[number]): Promise<void> { async function processExtraSource({ label, fetch: fetchFn, dedupeByTitle }: (typeof extraSources)[number]): Promise<void> {
+26
View File
@@ -11,6 +11,32 @@ Scope: `linkme/cv/`, `linkme/motivation*.md`, `rafapolo/` portfolio + `rafapolo/
- [ ] Reconsider German-first corporates (Coop, Valora, Helvetia) while at B1 — screening - [ ] Reconsider German-first corporates (Coop, Valora, Helvetia) while at B1 — screening
failure precedes any engineering assessment. failure precedes any engineering assessment.
## Referência da Swisscom (aberto, ação do Rafael)
- [ ] Pedir à Swisscom uma confirmação de emprego (Arbeitsbestätigung, Art. 330a al. 2 CO),
só com função e datas, sem avaliação. Fallback caso a correção da carta de março de
2025 ("moderate results") seja recusada: a confirmação pesa menos que a carta, mas
também chama atenção.
- Para: Swisscom (Switzerland) Ltd, Group Human Resources, HR Advisory, CH-3050 Berne
- Dados: DevOps Engineer III, Transport IP Network, Zürich, 01/03/2024 a 28/02/2025
- Ao receber: salvar em `rafapolo/job/credentials/references/` e usar no dossiê no
lugar da carta.
Rascunho:
> Dear HR Advisory team,
>
> I was employed by Swisscom (Switzerland) Ltd from March 1, 2024 until
> February 28, 2025 as DevOps Engineer III in the Transport IP Network unit in Zürich.
>
> Under Art. 330a para. 2 CO, I would like to request a confirmation of employment
> stating only the nature and duration of my employment, without any assessment of
> performance or conduct.
>
> Thank you in advance.
>
> Rafael Gomes de Oliveira Polo
## Not changing ## Not changing
Scraper is fine — `fetch.ts`, filter pipeline, `index.ts` all working. No changes needed. Scraper is fine — `fetch.ts`, filter pipeline, `index.ts` all working. No changes needed.