feat: revive datacareer.ch and integrate basel-jobs.ch
datacareer.ch had been returning 403 on every page for months. The cause was our own User-Agent: its nginx inverts the usual bot rule and blocks browser-like UAs while serving honest ones (no UA, curl, Googlebot). Give it PLAIN_HEADERS instead of the shared spoofed-Chrome HEADERS. Basel has 8 jobs on a single page with no pagination, so page 2 returning 0 items is correct rather than truncation. basel-jobs.ch was documented as un-integrable because DataDome blocks plain curl. Only half of that was true. Measured: plain fetch 403s on any UA, headless Chrome over CDP 403s, Playwright's own launch 403s even headful — but a real Chrome launched normally with --remote-debugging-port and attached over CDP afterwards gets 200. Clearance then lives entirely in a datadome cookie that replays fine over plain fetch with a matching UA, so a browser is touched only to mint, never to scrape. The earlier "cookie can't be extracted" note was true of the browser extension, not of CDP. Minting needs a visible window, so launchd runs can't do it; a stale cookie makes the source log and yield nothing instead of failing the run. Refresh by hand with `bun run mint-datadome`, or set LINKME_NO_BROWSER=1. Three passes, all needed — they cover different sets. The informatik category is a curated taxonomy, not the whole board, and the two search modes query different indexes: websearch=1 finds role words and postings filed outside IT (+147 over the category), while the bare search is the only mode matching stack terms — react, frontend and backend return zero under websearch=1. Language checks run against div.plain-content only; against the whole page the German site chrome alone clears the DE/FR threshold and drops every job, English ones included. Also fix a pre-existing double-wrapped Promise in the extraSources type.
This commit is contained in:
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env bun
|
||||
import { searches } from "./config.ts";
|
||||
import { fetchJobs, isViableContent, isViableUrl } from "./fetch.ts";
|
||||
import { fetchDataCareer, fetchStellenBasel, fetchItJobsCh, fetchJobRoom, fetchBaselIntlOrgs, fetchJobBasel, fetchEnglishJobSearch } from "./fetch-other.ts";
|
||||
import { fetchDataCareer, fetchStellenBasel, fetchItJobsCh, fetchJobRoom, fetchBaselIntlOrgs, fetchJobBasel, fetchEnglishJobSearch, fetchBaselJobs } from "./fetch-other.ts";
|
||||
|
||||
const CSV = "./results/jobs.csv";
|
||||
const OLD_HEADER = "date_found,posted_at,search,title,company,link";
|
||||
@@ -124,7 +124,7 @@ async function processLinkedInSearch(config: (typeof searches)[number]): Promise
|
||||
console.log(` [${config.label}] ${jobs.length} found, ${added} new, ${langDropped} non-English dropped`);
|
||||
}
|
||||
|
||||
const extraSources: Array<{ label: string; fetch: () => Promise<ReturnType<typeof fetchDataCareer>> }> = [
|
||||
const extraSources: Array<{ label: string; fetch: () => ReturnType<typeof fetchDataCareer> }> = [
|
||||
{ label: "datacareer.ch", fetch: fetchDataCareer },
|
||||
{ label: "stellen-basel.ch", fetch: fetchStellenBasel },
|
||||
{ label: "itjobs.ch", fetch: fetchItJobsCh },
|
||||
@@ -132,6 +132,7 @@ const extraSources: Array<{ label: string; fetch: () => Promise<ReturnType<typeo
|
||||
{ label: "basel intl orgs", fetch: fetchBaselIntlOrgs },
|
||||
{ label: "jobbasel.ch", fetch: fetchJobBasel },
|
||||
{ label: "englishjobsearch.ch", fetch: fetchEnglishJobSearch },
|
||||
{ label: "basel-jobs.ch", fetch: fetchBaselJobs },
|
||||
];
|
||||
|
||||
async function processExtraSource({ label, fetch: fetchFn }: (typeof extraSources)[number]): Promise<void> {
|
||||
|
||||
Reference in New Issue
Block a user