feat: revive datacareer.ch and integrate basel-jobs.ch
datacareer.ch had been returning 403 on every page for months. The cause was our own User-Agent: its nginx inverts the usual bot rule and blocks browser-like UAs while serving honest ones (no UA, curl, Googlebot). Give it PLAIN_HEADERS instead of the shared spoofed-Chrome HEADERS. Basel has 8 jobs on a single page with no pagination, so page 2 returning 0 items is correct rather than truncation. basel-jobs.ch was documented as un-integrable because DataDome blocks plain curl. Only half of that was true. Measured: plain fetch 403s on any UA, headless Chrome over CDP 403s, Playwright's own launch 403s even headful — but a real Chrome launched normally with --remote-debugging-port and attached over CDP afterwards gets 200. Clearance then lives entirely in a datadome cookie that replays fine over plain fetch with a matching UA, so a browser is touched only to mint, never to scrape. The earlier "cookie can't be extracted" note was true of the browser extension, not of CDP. Minting needs a visible window, so launchd runs can't do it; a stale cookie makes the source log and yield nothing instead of failing the run. Refresh by hand with `bun run mint-datadome`, or set LINKME_NO_BROWSER=1. Three passes, all needed — they cover different sets. The informatik category is a curated taxonomy, not the whole board, and the two search modes query different indexes: websearch=1 finds role words and postings filed outside IT (+147 over the category), while the bare search is the only mode matching stack terms — react, frontend and backend return zero under websearch=1. Language checks run against div.plain-content only; against the whole page the German site chrome alone clears the DE/FR threshold and drops every job, English ones included. Also fix a pre-existing double-wrapped Promise in the extraSources type.
This commit is contained in:
@@ -0,0 +1,15 @@
|
||||
/**
|
||||
* Refresh the basel-jobs.ch DataDome cookie.
|
||||
*
|
||||
* Opens a real Chrome window for a few seconds — headless and Playwright-launched
|
||||
* browsers are both fingerprinted and blocked, so this cannot run unattended.
|
||||
* Run it by hand when `bun fetch` reports basel-jobs.ch as unavailable.
|
||||
*/
|
||||
import { mintDataDomeSession } from "../fetch-other.ts";
|
||||
|
||||
const session = await mintDataDomeSession();
|
||||
if (!session) {
|
||||
console.error("failed to mint a DataDome cookie");
|
||||
process.exit(1);
|
||||
}
|
||||
console.log(`ok — minted ${session.mintedAt}`);
|
||||
Reference in New Issue
Block a user