feat: revive datacareer.ch and integrate basel-jobs.ch

datacareer.ch had been returning 403 on every page for months. The cause
was our own User-Agent: its nginx inverts the usual bot rule and blocks
browser-like UAs while serving honest ones (no UA, curl, Googlebot).
Give it PLAIN_HEADERS instead of the shared spoofed-Chrome HEADERS.
Basel has 8 jobs on a single page with no pagination, so page 2 returning
0 items is correct rather than truncation.

basel-jobs.ch was documented as un-integrable because DataDome blocks
plain curl. Only half of that was true. Measured: plain fetch 403s on any
UA, headless Chrome over CDP 403s, Playwright's own launch 403s even
headful — but a real Chrome launched normally with --remote-debugging-port
and attached over CDP afterwards gets 200. Clearance then lives entirely
in a datadome cookie that replays fine over plain fetch with a matching
UA, so a browser is touched only to mint, never to scrape. The earlier
"cookie can't be extracted" note was true of the browser extension, not
of CDP.

Minting needs a visible window, so launchd runs can't do it; a stale
cookie makes the source log and yield nothing instead of failing the run.
Refresh by hand with `bun run mint-datadome`, or set LINKME_NO_BROWSER=1.

Three passes, all needed — they cover different sets. The informatik
category is a curated taxonomy, not the whole board, and the two search
modes query different indexes: websearch=1 finds role words and postings
filed outside IT (+147 over the category), while the bare search is the
only mode matching stack terms — react, frontend and backend return zero
under websearch=1. Language checks run against div.plain-content only;
against the whole page the German site chrome alone clears the DE/FR
threshold and drops every job, English ones included.

Also fix a pre-existing double-wrapped Promise in the extraSources type.
This commit is contained in:
2026-09-01 12:23:59 +02:00
parent b61a75eca0
commit d5e6605176
5 changed files with 446 additions and 10 deletions
+3 -2
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env bun
import { searches } from "./config.ts";
import { fetchJobs, isViableContent, isViableUrl } from "./fetch.ts";
import { fetchDataCareer, fetchStellenBasel, fetchItJobsCh, fetchJobRoom, fetchBaselIntlOrgs, fetchJobBasel, fetchEnglishJobSearch } from "./fetch-other.ts";
import { fetchDataCareer, fetchStellenBasel, fetchItJobsCh, fetchJobRoom, fetchBaselIntlOrgs, fetchJobBasel, fetchEnglishJobSearch, fetchBaselJobs } from "./fetch-other.ts";
const CSV = "./results/jobs.csv";
const OLD_HEADER = "date_found,posted_at,search,title,company,link";
@@ -124,7 +124,7 @@ async function processLinkedInSearch(config: (typeof searches)[number]): Promise
console.log(` [${config.label}] ${jobs.length} found, ${added} new, ${langDropped} non-English dropped`);
}
const extraSources: Array<{ label: string; fetch: () => Promise<ReturnType<typeof fetchDataCareer>> }> = [
const extraSources: Array<{ label: string; fetch: () => ReturnType<typeof fetchDataCareer> }> = [
{ label: "datacareer.ch", fetch: fetchDataCareer },
{ label: "stellen-basel.ch", fetch: fetchStellenBasel },
{ label: "itjobs.ch", fetch: fetchItJobsCh },
@@ -132,6 +132,7 @@ const extraSources: Array<{ label: string; fetch: () => Promise<ReturnType<typeo
{ label: "basel intl orgs", fetch: fetchBaselIntlOrgs },
{ label: "jobbasel.ch", fetch: fetchJobBasel },
{ label: "englishjobsearch.ch", fetch: fetchEnglishJobSearch },
{ label: "basel-jobs.ch", fetch: fetchBaselJobs },
];
async function processExtraSource({ label, fetch: fetchFn }: (typeof extraSources)[number]): Promise<void> {