feat: revive datacareer.ch and integrate basel-jobs.ch
datacareer.ch had been returning 403 on every page for months. The cause was our own User-Agent: its nginx inverts the usual bot rule and blocks browser-like UAs while serving honest ones (no UA, curl, Googlebot). Give it PLAIN_HEADERS instead of the shared spoofed-Chrome HEADERS. Basel has 8 jobs on a single page with no pagination, so page 2 returning 0 items is correct rather than truncation. basel-jobs.ch was documented as un-integrable because DataDome blocks plain curl. Only half of that was true. Measured: plain fetch 403s on any UA, headless Chrome over CDP 403s, Playwright's own launch 403s even headful — but a real Chrome launched normally with --remote-debugging-port and attached over CDP afterwards gets 200. Clearance then lives entirely in a datadome cookie that replays fine over plain fetch with a matching UA, so a browser is touched only to mint, never to scrape. The earlier "cookie can't be extracted" note was true of the browser extension, not of CDP. Minting needs a visible window, so launchd runs can't do it; a stale cookie makes the source log and yield nothing instead of failing the run. Refresh by hand with `bun run mint-datadome`, or set LINKME_NO_BROWSER=1. Three passes, all needed — they cover different sets. The informatik category is a curated taxonomy, not the whole board, and the two search modes query different indexes: websearch=1 finds role words and postings filed outside IT (+147 over the category), while the bare search is the only mode matching stack terms — react, frontend and backend return zero under websearch=1. Language checks run against div.plain-content only; against the whole page the German site chrome alone clears the DE/FR threshold and drops every job, English ones included. Also fix a pre-existing double-wrapped Promise in the extraSources type.
This commit is contained in:
+389
-1
@@ -8,6 +8,19 @@ const HEADERS = {
|
||||
Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
};
|
||||
|
||||
// datacareer.ch inverts the usual bot rule: its nginx 403s anything claiming to
|
||||
// be a browser (a spoofed Chrome UA is the one thing guaranteed to fail) while
|
||||
// letting honest clients — no UA, curl, Googlebot — straight through. Presumably
|
||||
// real browser traffic arrives via a path that also carries matching TLS/CH
|
||||
// fingerprints, so a bare Chrome UA over plain HTTP reads as a scraper. Sending
|
||||
// an honest identifier is both what unblocks it and the polite thing to do.
|
||||
// This is what killed the source for months; it was never a hard block.
|
||||
const PLAIN_HEADERS = {
|
||||
"User-Agent": "linkme-jobbot/1.0 (personal job search; contact via basel)",
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
};
|
||||
|
||||
function decodeHtml(s: string): string {
|
||||
return s
|
||||
.replace(/&/g, "&")
|
||||
@@ -63,7 +76,7 @@ export async function fetchDataCareer(): Promise<Job[]> {
|
||||
|
||||
const res = await fetch(
|
||||
`https://www.datacareer.ch/jobs/?city=Basel&page=${page}`,
|
||||
{ headers: HEADERS }
|
||||
{ headers: PLAIN_HEADERS }
|
||||
);
|
||||
if (!res.ok) { console.error(`failed (${res.status})`); break; }
|
||||
|
||||
@@ -708,3 +721,378 @@ export async function fetchStellenBasel(): Promise<Job[]> {
|
||||
|
||||
return allJobs;
|
||||
}
|
||||
|
||||
// basel-jobs.ch — regional board on the JobChannel network (same syndication
|
||||
// network whose ?source=jobchannel links show up on Siemens/jobs.ch postings).
|
||||
//
|
||||
// The board is protected by DataDome, which is why this source sat documented
|
||||
// as "not integrable" for months. The block is real but narrower than it looked:
|
||||
//
|
||||
// - Plain curl / bun fetch → 403 on every UA, browser-like or not.
|
||||
// - Chrome --headless=new, driven over CDP → 403. Headless is detected.
|
||||
// - Playwright's own chromium.launch(), even headful → 403. The CDP-launch
|
||||
// path leaves automation artefacts DataDome fingerprints.
|
||||
// - A real Chrome launched *normally* (with --remote-debugging-port so we can
|
||||
// attach afterwards) → 200. Attaching over CDP after the fact is invisible.
|
||||
//
|
||||
// The unlock is that DataDome's clearance lives entirely in a `datadome` cookie
|
||||
// (Max-Age one year) which replays fine over plain fetch with a matching
|
||||
// User-Agent. So we mint a cookie once with a real browser window, cache it, and
|
||||
// scrape normally thereafter — a browser is touched only when the cookie is
|
||||
// missing or has been rejected, not on every run. An earlier note in CLAUDE.md
|
||||
// claimed the cookie "can't be extracted for replay"; that was true of the
|
||||
// browser-extension tooling, not of CDP, which hands it over directly.
|
||||
//
|
||||
// Operational caveat: minting needs a *visible* Chrome window for ~10s, so an
|
||||
// unattended `bun daily` under launchd cannot mint. When the cookie is stale in
|
||||
// that context the source logs and yields nothing rather than failing the run;
|
||||
// refresh it deliberately with `bun run scripts/mint-datadome.ts`.
|
||||
|
||||
const DATADOME_CACHE = "results/.datadome.json";
|
||||
const CHROME_BIN = "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome";
|
||||
const BASELJOBS_ORIGIN = "https://basel-jobs.ch";
|
||||
|
||||
interface DataDomeSession {
|
||||
cookie: string;
|
||||
ua: string;
|
||||
mintedAt: string;
|
||||
}
|
||||
|
||||
async function loadDataDomeSession(): Promise<DataDomeSession | null> {
|
||||
try {
|
||||
const raw = await Bun.file(DATADOME_CACHE).text();
|
||||
const s = JSON.parse(raw) as DataDomeSession;
|
||||
return s.cookie && s.ua ? s : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async function waitForCdp(port: number, timeoutMs: number): Promise<boolean> {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
try {
|
||||
const r = await fetch(`http://127.0.0.1:${port}/json/version`);
|
||||
if (r.ok) return true;
|
||||
} catch { /* not up yet */ }
|
||||
await Bun.sleep(300);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Launch a real, visible Chrome, let it clear DataDome, then attach over CDP and
|
||||
* lift the resulting cookie. Headless and Playwright-launched browsers are both
|
||||
* detected — only a normally-launched window gets through, which is the whole
|
||||
* reason this is a separate, rarely-run step instead of the fetch path itself.
|
||||
*/
|
||||
export async function mintDataDomeSession(): Promise<DataDomeSession | null> {
|
||||
if (process.env.LINKME_NO_BROWSER === "1") {
|
||||
console.log(" LINKME_NO_BROWSER=1 — skipping DataDome mint");
|
||||
return null;
|
||||
}
|
||||
if (!(await Bun.file(CHROME_BIN).exists())) {
|
||||
console.error(` Chrome not found at ${CHROME_BIN} — cannot mint DataDome cookie`);
|
||||
return null;
|
||||
}
|
||||
|
||||
let chromium: typeof import("playwright-core").chromium;
|
||||
try {
|
||||
({ chromium } = await import("playwright-core"));
|
||||
} catch {
|
||||
console.error(" playwright-core not installed — run `bun add playwright-core`");
|
||||
return null;
|
||||
}
|
||||
|
||||
// A port unlikely to collide with a debugging session the user opened.
|
||||
const port = 9333;
|
||||
const profile = "results/.datadome-profile";
|
||||
const proc = Bun.spawn(
|
||||
[
|
||||
CHROME_BIN,
|
||||
`--remote-debugging-port=${port}`,
|
||||
`--user-data-dir=${profile}`,
|
||||
"--no-first-run",
|
||||
"--no-default-browser-check",
|
||||
"--window-size=1200,860",
|
||||
BASELJOBS_ORIGIN,
|
||||
],
|
||||
{ stdout: "ignore", stderr: "ignore" }
|
||||
);
|
||||
|
||||
try {
|
||||
if (!(await waitForCdp(port, 20_000))) {
|
||||
console.error(" Chrome CDP endpoint never came up");
|
||||
return null;
|
||||
}
|
||||
const browser = await chromium.connectOverCDP(`http://127.0.0.1:${port}`);
|
||||
const ctx = browser.contexts()[0];
|
||||
const page =
|
||||
ctx.pages().find((p) => p.url().includes("basel-jobs")) ?? ctx.pages()[0];
|
||||
|
||||
// The interstitial resolves itself; give it room, then confirm we're through.
|
||||
let cookie = "";
|
||||
let ua = "";
|
||||
for (let attempt = 0; attempt < 12; attempt++) {
|
||||
await page.waitForTimeout(1500);
|
||||
const html = await page.content();
|
||||
if (/Please enable JS|geo\.captcha/.test(html)) continue;
|
||||
const cookies = await ctx.cookies(BASELJOBS_ORIGIN);
|
||||
const dd = cookies.find((c) => c.name === "datadome");
|
||||
if (!dd) continue;
|
||||
cookie = dd.value;
|
||||
ua = await page.evaluate(() => navigator.userAgent);
|
||||
break;
|
||||
}
|
||||
await browser.close();
|
||||
|
||||
if (!cookie || !ua) {
|
||||
console.error(" DataDome challenge did not clear — no cookie minted");
|
||||
return null;
|
||||
}
|
||||
|
||||
const session: DataDomeSession = { cookie, ua, mintedAt: new Date().toISOString() };
|
||||
await Bun.write(DATADOME_CACHE, JSON.stringify(session, null, 2));
|
||||
console.log(` minted DataDome cookie (${cookie.length} chars)`);
|
||||
return session;
|
||||
} catch (e) {
|
||||
console.error(` DataDome mint failed: ${e}`);
|
||||
return null;
|
||||
} finally {
|
||||
proc.kill();
|
||||
}
|
||||
}
|
||||
|
||||
function baselJobsHeaders(s: DataDomeSession): Record<string, string> {
|
||||
return {
|
||||
"User-Agent": s.ua,
|
||||
"Accept-Language": "de-CH,de;q=0.9,en;q=0.8",
|
||||
Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
Cookie: `datadome=${s.cookie}`,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch with the cached cookie, re-minting once if the cookie has gone stale.
|
||||
* DataDome rotates the cookie on ordinary responses too, so a Set-Cookie on the
|
||||
* way past is folded back into the session rather than ignored.
|
||||
*/
|
||||
async function baselJobsFetch(
|
||||
url: string,
|
||||
state: { session: DataDomeSession | null; minted: boolean }
|
||||
): Promise<string | null> {
|
||||
for (let attempt = 0; attempt < 2; attempt++) {
|
||||
if (!state.session) {
|
||||
if (state.minted) return null;
|
||||
state.minted = true;
|
||||
state.session = await mintDataDomeSession();
|
||||
if (!state.session) return null;
|
||||
}
|
||||
|
||||
const res = await fetch(url, { headers: baselJobsHeaders(state.session) });
|
||||
|
||||
const rotated = res.headers
|
||||
.getSetCookie?.()
|
||||
.map((c) => /(?:^|;\s*)datadome=([^;]+)/.exec(c)?.[1])
|
||||
.find(Boolean);
|
||||
if (rotated && state.session) {
|
||||
state.session = { ...state.session, cookie: rotated };
|
||||
await Bun.write(DATADOME_CACHE, JSON.stringify(state.session, null, 2));
|
||||
}
|
||||
|
||||
if (res.ok) return res.text();
|
||||
|
||||
if (res.status === 403 && !state.minted) {
|
||||
console.log(" cookie rejected — re-minting");
|
||||
state.session = null;
|
||||
continue;
|
||||
}
|
||||
console.error(` failed (${res.status}) ${url}`);
|
||||
return null;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// Terms swept through the board's search. No keyword-union endpoint exists, so
|
||||
// — like job-room.ch and jobbasel.ch — this is a fixed list, one request each
|
||||
// per mode, deduped by vacancy id. German stems are in deliberately: the search
|
||||
// matches description text, so "entwickler" surfaces postings whose English
|
||||
// title the category never filed under IT. The title and description filters
|
||||
// drop the German-language ones afterwards.
|
||||
//
|
||||
// Role words and stack words behave differently across the two search modes
|
||||
// (see fetchBaselJobs), which is why both modes get the whole list rather than
|
||||
// a tuned subset — `websearch=1` finds no stack terms at all, and the bare
|
||||
// search is the only mode that matches "react"/"frontend"/"backend".
|
||||
const BASELJOBS_TERMS = [
|
||||
// role / domain words — these carry the `websearch=1` pass
|
||||
"software", "engineer", "developer", "entwickler", "informatik", "data",
|
||||
"cloud", "devops", "platform", "architect", "security", "ai",
|
||||
// stack words — only the bare search matches these
|
||||
"python", "javascript", "typescript", "react", "node",
|
||||
"fullstack", "full stack", "backend", "frontend", "kubernetes", "linux",
|
||||
];
|
||||
|
||||
function buildBaselJobsLocation(city: string, canton: string): string {
|
||||
if (!city) return canton || "Basel";
|
||||
if (!canton) return city;
|
||||
// data-gtm-jp is one or more codes ("BS", "BL,BS"); keep only what the city
|
||||
// string doesn't already name.
|
||||
const missing = canton
|
||||
.split(",")
|
||||
.map((c) => c.trim())
|
||||
.filter((c) => c && !new RegExp(`\\b${c}\\b`).test(city));
|
||||
return missing.length ? `${city}, ${missing.join(",")}` : city;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pull every vacancy card out of a listing page. Returns how many ids were
|
||||
* new so callers can use "no new postings" as their end condition — past the
|
||||
* last real page the board still serves its two sponsored slots, so an empty
|
||||
* list never arrives.
|
||||
*/
|
||||
function parseBaselJobsCards(
|
||||
html: string,
|
||||
seen: Set<string>,
|
||||
out: Job[],
|
||||
today: string
|
||||
): { items: number; added: number } {
|
||||
const items = [...html.matchAll(/<li class="item"(.*?)<\/li>/gs)].map((m) => m[1]);
|
||||
let added = 0;
|
||||
|
||||
for (const card of items) {
|
||||
const jid = /data-gtm-jid="(\d+)"/.exec(card)?.[1];
|
||||
const linkM = /href="(https:\/\/basel-jobs\.ch\/job\/[^"]+)"/.exec(card);
|
||||
const titleM = /<span class="jobtitle">(.*?)<\/span>/s.exec(card);
|
||||
if (!jid || !linkM || !titleM) continue;
|
||||
|
||||
// Sponsored slots repeat the same posting within a page, between pages,
|
||||
// and across the category and search sweeps.
|
||||
const id = `bj-${jid}`;
|
||||
if (seen.has(id)) continue;
|
||||
seen.add(id);
|
||||
added++;
|
||||
|
||||
const title = decodeHtml(titleM[1]);
|
||||
if (!applyFilters(title)) continue;
|
||||
|
||||
const companyM =
|
||||
/<span class="company">\s*<a[^>]*>(.*?)<\/a>/s.exec(card) ??
|
||||
/<span class="company">(.*?)<\/span>/s.exec(card);
|
||||
const locationM = /<span class="location">(.*?)<\/span>/s.exec(card);
|
||||
const canton = /data-gtm-jp="([^"]*)"/.exec(card)?.[1] ?? "";
|
||||
|
||||
out.push({
|
||||
id,
|
||||
title,
|
||||
company: companyM ? decodeHtml(companyM[1]) : "",
|
||||
// Cards carry a city; the canton attribute disambiguates BS vs BL,
|
||||
// which is what daily.ts groups on. Some cards already end in the
|
||||
// canton code ("Basel, BS, 4030") — don't append it twice.
|
||||
location: buildBaselJobsLocation(
|
||||
locationM ? decodeHtml(locationM[1]) : "",
|
||||
canton
|
||||
),
|
||||
// No date anywhere on the card or detail page — discovery date is the
|
||||
// best available, same as any board that hides posting age.
|
||||
postedAt: today,
|
||||
link: linkM[1],
|
||||
search: "basel-jobs.ch",
|
||||
source: "basel-jobs.ch",
|
||||
});
|
||||
}
|
||||
|
||||
return { items: items.length, added };
|
||||
}
|
||||
|
||||
export async function fetchBaselJobs(): Promise<Job[]> {
|
||||
const allJobs: Job[] = [];
|
||||
const seen = new Set<string>();
|
||||
const state = { session: await loadDataDomeSession(), minted: false };
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
|
||||
// Pass 1 — the board's IT category, ~300 postings over 3 pages.
|
||||
for (let page = 1; page <= 10; page++) {
|
||||
if (page > 1) await Bun.sleep(400);
|
||||
process.stdout.write(` informatik page ${page} … `);
|
||||
|
||||
const html = await baselJobsFetch(
|
||||
`${BASELJOBS_ORIGIN}/informatik-jobs?page=${page}`,
|
||||
state
|
||||
);
|
||||
if (html === null) { console.log("unavailable"); break; }
|
||||
|
||||
const { items, added } = parseBaselJobsCards(html, seen, allJobs, today);
|
||||
if (items === 0) { console.log("0 items"); break; }
|
||||
console.log(`${items} items, ${added} new`);
|
||||
if (added === 0) break;
|
||||
}
|
||||
|
||||
// Pass 2 — full-text search. The category is a *curated taxonomy*, not the
|
||||
// whole board: it only holds what the board filed under IT, so tech roles
|
||||
// parked under Industrie/Pharma/Bau, and roles whose relevance lives in the
|
||||
// description rather than the title, never appear in pass 1. Measured on
|
||||
// 2026-09-01 this pass added **147 postings the category missed**, including
|
||||
// "Senior AI Infrastructure & Platform Engineer (Basel)" and "Senior DevOps /
|
||||
// Cloud Engineer (Kubernetes, CI/CD, Linux)". The reverse is also true (the
|
||||
// category held 257 the search missed), so both passes are needed.
|
||||
//
|
||||
// `websearch=1` is the mode that does this — the bare `?terms=` search
|
||||
// returned nothing that wasn't already in the category. It ignores `page`
|
||||
// entirely and returns the whole result set on one response, so there is no
|
||||
// pagination to walk.
|
||||
// Both search modes are swept, because they query different indexes and each
|
||||
// is blind to what the other finds:
|
||||
//
|
||||
// term websearch=1 bare
|
||||
// engineer 90 260 found
|
||||
// frontend 0 2
|
||||
// backend 0 4
|
||||
// react 0 2
|
||||
//
|
||||
// `websearch=1` matches role/domain words and reaches postings filed outside
|
||||
// the IT category; the bare search is the *only* mode that matches stack
|
||||
// terms. Neither paginates — both ignore `page` and return their whole result
|
||||
// set in one response (bare caps around 100 cards). Measured 2026-09-01
|
||||
// against the category: websearch added 147, bare added a further 139 raw
|
||||
// (23 surviving the title filters).
|
||||
for (const mode of ["websearch", "bare"] as const) {
|
||||
for (const term of BASELJOBS_TERMS) {
|
||||
await Bun.sleep(400);
|
||||
process.stdout.write(` ${mode} "${term}" … `);
|
||||
|
||||
const qs = `terms=${encodeURIComponent(term)}${mode === "websearch" ? "&page=1&websearch=1" : ""}`;
|
||||
const html = await baselJobsFetch(`${BASELJOBS_ORIGIN}/search?${qs}`, state);
|
||||
if (html === null) { console.log("unavailable"); break; }
|
||||
|
||||
const { items, added } = parseBaselJobsCards(html, seen, allJobs, today);
|
||||
console.log(`${items} items, ${added} new`);
|
||||
}
|
||||
}
|
||||
|
||||
// Detail pages carry the full description inline, so the DE/FR and
|
||||
// German-requirement checks run here instead of costing index.ts a fetch that
|
||||
// would hit DataDome without the cookie and be dropped as unreachable.
|
||||
const viable: Job[] = [];
|
||||
for (const job of allJobs) {
|
||||
await Bun.sleep(300);
|
||||
process.stdout.write(` checking ${job.title} … `);
|
||||
const page = await baselJobsFetch(job.link, state);
|
||||
if (page === null) { console.log("unreachable, keeping"); viable.push(job); continue; }
|
||||
|
||||
// Only the posting's own text, never the page. basel-jobs.ch is a German
|
||||
// site, so its nav/footer/category chrome alone clears the ≥ 6 DE/FR word
|
||||
// threshold — checking the whole page dropped *every* job, English ones
|
||||
// included. The mobile `plain-content` block is the description as plain
|
||||
// text; the desktop view loads the same thing into an iframe, so this is
|
||||
// the only copy present in the server-rendered HTML.
|
||||
const descM = /<div class="box plain-content[^"]*">(.*?)<\/div>/s.exec(page);
|
||||
if (!descM) { console.log("no description, keeping"); viable.push(job); continue; }
|
||||
const check = isViableDescription(descM[1].replace(/<br\s*\/?>/gi, " "));
|
||||
if (!check.ok) { console.log(`dropped (${check.reason})`); continue; }
|
||||
console.log("ok");
|
||||
viable.push({ ...job, contentChecked: true });
|
||||
}
|
||||
|
||||
return viable;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user