Compare commits
4
Commits
aeb16de5ed
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5a2d053ca7 | ||
|
|
b05bddb73d | ||
|
|
b5cf8931ea | ||
|
|
73840cad04 |
@@ -21,7 +21,7 @@ Temp scripts: `bun run scripts/<name>-temp.ts` — never edit `config.ts` for on
|
||||
```
|
||||
config.ts # search keywords and areas
|
||||
fetch.ts # scraper, HTML parser, all filters
|
||||
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch, startup.jobs)
|
||||
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel intl orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch + data-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch, startup.jobs, arbeitnow.ch)
|
||||
index.ts # orchestrator: runs searches, writes CSV
|
||||
recheck.ts # raw-mode fetch, lists filtered-out jobs
|
||||
daily.ts # digest: fetch → match against profile.ts → send to Telegram
|
||||
@@ -40,10 +40,10 @@ plans/ # LinkedIn post drafts
|
||||
## Architecture
|
||||
|
||||
- **`config.ts`** — `KEYWORDS` string (joined with ` OR `) and `searches: SearchConfig[]`. Edit here to add/remove search areas or keywords.
|
||||
- Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform) and `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform and Field add a Switzerland-wide remote pass.
|
||||
- Also exports `PLATFORM_KEYWORDS` (architect/SRE/infra/security/MLOps), `RESEARCH_KEYWORDS` (research & scientific software, HPC, data platform), `FIELD_KEYWORDS` (forward-deployed, customer/sales/integration, founding/product) and `FRONTEND_KEYWORDS` (frontend/React/web, data visualisation, geospatial/GIS, WebGL — added 2026-10-20 after a one-off search found 14 of 18 relevant frontend/viz roles missing from the CSV). Kept as separate OR-strings rather than folded into `KEYWORDS` — LinkedIn ranks a focused query far better than one 50-term disjunction. The `sweep()` helper generates one search per area per family; Research skips Bern, Platform, Field and Frontend add a Switzerland-wide remote pass.
|
||||
- `RESEARCH_KEYWORDS` deliberately contains no bioinformatics/computational-biology terms — see `BIO_DOMAIN_RE`.
|
||||
- **`fetch.ts`** — HTTP scraping, HTML parsing, and all filters. Exports `fetchJobs(config, { raw? })` and `isEnglishContent(id)`.
|
||||
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch, startup.jobs). Imported by `index.ts`.
|
||||
- **`fetch-other.ts`** — scrapes extra sources (datacareer.ch, stellen-basel.ch, itjobs.ch, job-room.ch, Basel international orgs, jobbasel.ch, englishjobsearch.ch, basel-jobs.ch + data-jobs.ch, scicore.unibas.ch, swisscom.com, novartis.com, more-jobs.ch, startup.jobs, arbeitnow.ch). Imported by `index.ts`.
|
||||
- **novartis.com** (`fetchNovartisBasel`, added 2026-09-21) — Rafael has an internal contact at Novartis Basel who can refer him, so this source exists purely to catch a genuine software/engineering opening there, not to widen the CSV generally. Its career-search page (`novartis.com/careers/career-search?country[]=LOC_CH`) is a server-rendered Drupal view — no headless browser needed — but only at the **locale-less English path** (`/careers/career-search`); the `ch-de` German path's job-detail links 302 straight back to the search page for both locales when hit without a session, so detail pages only resolve through the English path. Pagination is 0-indexed (`page=0` implicit, `page=1` is the second page of 10) and the page's own "Showing N results" line gives the exact total, so the page-loop bound is computed rather than probed — the listing has also been observed to **repeat rows across pages within one sweep** (measured 2026-09-21: several titles appeared twice across 5 pages), so results are deduped by `req-<id>` before filtering, not trusted from the pager.
|
||||
|
||||
Novartis Basel's titling is almost entirely non-technical "Director, `<business function>`" roles (Portfolio Analytics, Therapeutic Area Strategy, Policy Communications, ...) plus device/hardware "Engineering" roles (autoinjector/drug-delivery hardware, not software) — the generic `SOFTWARE_TITLE_RE` used by every other source in this file matched **13 of 47** CH listings on title alone in an initial run, and every single one was a false positive (no engineering-team role among them). `NOVARTIS_TITLE_RE` replaces it here: it requires an explicit software/IT engineering job-family term (`software engineer`, `platform engineer`, `data engineer`, `devops`, `sre`, `solutions architect`, ...), not just a bare seniority or business-function word. `BIO_DOMAIN_RE` and job-room.ch's `PHARMA_ROLE_RE` still run on top to catch domain-science titles the stricter regex wouldn't otherwise need to worry about. Narrowed to Basel/Muttenz specifically (`NOVARTIS_BASEL_RE`) — the CH listing also includes Stein (Aargau), a different commute the contact couldn't refer into. As of 2026-09-21 this source keeps **zero** listings — expected; it's watching for the rare opening, not sweeping volume. Each detail page carries the full description inline (`job_details_content_center`…`job_details_content_bottom`), so the DE/FR + `requiresGerman` check runs in-process per candidate rather than costing `index.ts` a page fetch.
|
||||
@@ -78,6 +78,8 @@ plans/ # LinkedIn post drafts
|
||||
|
||||
Pagination/dedup: page 4+ of the category still serves 2 sponsored slots, so **"no new ids" is the end condition**, not an empty list. Sponsored cards repeat within a page, across pages, and across both passes — dedupe on `data-gtm-jid`. Cards carry no date, so `postedAt` is the discovery date. Canton comes from `data-gtm-jp` (sometimes `"BL,BS"`).
|
||||
|
||||
**Pass 3: data-jobs.ch** (added 2026-10-22) — the same jobchannel network's data/analytics board, swept for BS/BL/ZH/BE/LU (`DATAJOBS_PROVINCES`) inside `fetchBaselJobs`: same card markup and parser, and the same DataDome cookie clears both hosts (a cookie rotated by one host works on the other). `/search?provinces=…&lang=en-ch&page=n` walks the board's own listings (100 per page; past the end it still serves a few sponsored cards, so "no new ids" ends the walk), then one `&page=1&websearch=1` request returns all the "further jobs from the web search" (it ignores `page`). Measured 2026-10-22: 166 own + 98 web search = 264 unique for the five cantons. Vacancy ids are network-wide, so it runs last and shares `seen` — a posting on both boards keeps its basel-jobs.ch link. Its inventory is **not** a subset of basel-jobs.ch (data-only postings 404 there): measured 2026-10-22, 26 ids the basel-jobs passes missed. A posting on both boards keeps the same `/job/<slug>/<id>` path, so `knownJobchannelTails()` skips any card whose path is already in the CSV under either host — before its detail fetch, not after. Re-checking every known posting on every run is what got the cookie blocked on the first five-canton test (2026-10-22). For the same reason a detail page that fails to load is now **skipped** (retried next run) rather than kept unchecked: a blocked cookie fails every remaining page at once. Rows get `source`/`search` = `data-jobs.ch`.
|
||||
|
||||
**Language-check the `div.plain-content` block only, never the whole page.** The site chrome is German and on its own clears the ≥ 6 DE/FR word threshold — checking the full page dropped *every* job, English ones included. That block is the mobile rendering of the description; the desktop view iframes the same text, so it's the only copy in the server-rendered HTML.
|
||||
- **Basel international orgs** (`fetchBaselIntlOrgs`) sweeps three boards under one label, each only a handful of postings:
|
||||
- **BIS** — the career site is JS-rendered with no usable API, so we read the UNjobs mirror at `unjobs.org/offices/bis_182320855689338880`. Titles arrive as `Role, City, Country` and get split back apart. BIS also cycles Senior Software / Data / Data & Analytics Engineer roles through this board.
|
||||
@@ -93,10 +95,11 @@ plans/ # LinkedIn post drafts
|
||||
- **startup.jobs** (`fetchStartupJobs`, added 2026-10-06) — global startup board, swept Switzerland-wide. **Every path is behind a Cloudflare managed challenge** (403 on plain HTTP for any UA, feeds and `/api` included), and unlike basel-jobs.ch's DataDome cookie, `cf_clearance` **does not replay** over Bun's fetch — measured 403 with the exact cookie + UA; it's bound to Chrome's TLS fingerprint. What works: Chrome `--headless=new` with a throwaway profile clears the challenge **only if the UA has no "HeadlessChrome" token** (the version is read from `Chrome --version` so it doesn't drift). So this source launches a headless browser on every run — unlike basel-jobs.ch's minting, that works fine under launchd. `LINKME_NO_BROWSER=1` disables it; any failure logs `unavailable` and returns `[]`.
|
||||
|
||||
Listings come from **Algolia** (app `4CQMTMMK73`, index `Post_production`), not the HTML: the page embeds a *secured* search key in `meta[name="current-algolia-api-key-search"]`, and algolia.net is not behind Cloudflare, so the queries run over plain fetch. The key's `validUntil` is **~3h after page load** — read fresh every run, never cached. The key rejects every country filter (`country_code`/`country_id` → 0 hits), so the sweep is a geo bounding box around Switzerland with `country_code === "CH"` checked locally (the box also catches DE/FR/AT/LI/IT), limited to the last 60 days. Algolia caps a query at **500 retrievable hits** (100 × 5 pages) — a box over 500 is split into quadrants recursively (all ~1400 CH-box postings vs 556 in the 60-day window, measured 2026-10-06). `tagFilters` on `_tags` (Developer/Engineer) was tried and dropped: it covered only 365 of them. Records carry no description: each title-filtered candidate's detail page is fetched *inside* the browser (`page.evaluate(fetch)`, same TLS) and its JSON-LD `JobPosting.description` goes through `isViableDescription` (`contentChecked: true`). Links already in `results/jobs.csv` are skipped before that fetch. It aggregates company ATS feeds (Wise, Stripe, Avaloq, Nebius…) that LinkedIn also carries, so its `extraSources` entry sets `dedupeByTitle` like more-jobs.ch. The user's original URL was `?q=Developer&w=hybrid` around Basel; the source deliberately keeps on-site/remote too and uses `SOFTWARE_TITLE_RE` instead of the `Developer` query, matching every other source.
|
||||
- **arbeitnow.ch** (`fetchArbeitnow`, added 2026-11-03) — free public JSON API (`/api/job-board-api?page=n`, 100/page, no auth), fed straight from company ATS feeds. Rolling ~7-day window (~195 jobs), so 2 requests cover it. Full description inline → in-process DE/FR + `requiresGerman` check. `dedupeByTitle` (overlaps ~50% with LinkedIn). **pilotjob.ch was evaluated and rejected**: its listing only offers `src=arbeitnow|jobroom`, i.e. it re-serves this feed plus job-room.ch. englishjobsearch.ch/jobs/remote shares only 1 of arbeitnow's 19 remote postings.
|
||||
- **`index.ts`** — orchestrator: loads existing IDs from `results/jobs.csv`, calls `fetchJobs`, calls `isEnglishContent` for each new job, appends survivors to CSV. Writes both `search` (the query/source that found the job) and `location` (the posting's own location, as parsed off its card by `fetch.ts`/`fetch-other.ts` — no extra page fetch) to the CSV. Migrates old CSVs on startup: adds `source` if missing, then `location` (backfilled `""` for rows written before the column existed).
|
||||
- Every LinkedIn search and every `fetch-other.ts` source hits a different host, so they run **concurrently** (bounded: 3 at a time within each group, both groups started together) instead of one after another — cut a full `bun fetch` from ~16-20 min to well under 2 min. `existingLinks.add()` happens synchronously right after the `.has()` check and before any `await`, which is what makes concurrent workers safe against double-claiming the same link. Concurrency surfaced a real gap: LinkedIn started returning occasional 429s under parallel load, which `fetchJobs` used to treat as end-of-results (silently truncating that search) — `fetchJobs` now retries a 429 up to 3× with backoff (1s/2s/4s) before giving up.
|
||||
- **`recheck.ts`** — calls `fetchJobs(..., { raw: true })` to bypass `SOFTWARE_TITLE_RE`, then lists every new job that *would* be filtered out and prints word-frequency suggestions.
|
||||
- **`daily.ts`** — the notification layer on top of `index.ts`. Snapshots the CSV's links, shells out to `bun fetch`, then diffs to find what's new. Survivors are filtered through `matchesProfile` and grouped by area (Basel / Remote / Zürich / Bern) so the message scans on a phone. `results/sent.txt` is written **only after a successful send**, so a Telegram failure retries next run instead of silently losing jobs. A non-zero exit from `bun fetch` sends a failure notice rather than failing quietly.
|
||||
- **`daily.ts`** — the notification layer on top of `index.ts`. Snapshots the CSV's links, shells out to `bun fetch`, then diffs to find what's new. Survivors are filtered through `matchesProfile` and grouped by area (Basel / Remote / Zürich / Bern / Luzern) so the message scans on a phone. `results/sent.txt` is written **only after a successful send**, so a Telegram failure retries next run instead of silently losing jobs. A non-zero exit from `bun fetch` sends a failure notice rather than failing quietly.
|
||||
- The area grouping matches against the CSV's `location` column (the posting's real location) first, falling back to `search` (the query/source label) only when `location` is empty — i.e. for rows written before the `location` column existed. This fixed two real mislabels: a LinkedIn "Bern" sweep can surface a job actually based in Lausanne (search label ≠ real location), and `fetch-other.ts` sources (job-room.ch, itjobs.ch, stellen-basel.ch, datacareer.ch) write the source name into `search` instead of a city, which used to dump their jobs into "⚪️ Other" regardless of where they actually were.
|
||||
- **`profile.ts`** — `matchesProfile(title, company, minScore = 4)`. Hard rejects run first (Microsoft stack, Java/C++/SAP, life-science, non-software functions, blocked companies Jobgether and ALLPS), then `SIGNALS` scores stack and seniority terms from `cv/polo.md`. This is a **second, stricter gate than the fetch filters**: `fetch.ts` decides what lands in the CSV, `profile.ts` decides what's worth a phone buzz. Widen `SOFTWARE_TITLE_RE` to catch more in the CSV; tune `SIGNALS`/`minScore` to change what gets sent.
|
||||
- **`telegram.ts`** — reads `.env` directly (no dotenv dep). Accepts `TELEGRAM_BOT` or `TELEGRAM_TOKEN` for the token. Splits at 3900 chars on line boundaries (Telegram hard-caps at 4096) and sleeps 400 ms between parts to stay under the rate limit. Messages are HTML — run titles/companies through `escapeHtml`.
|
||||
|
||||
@@ -184,6 +184,21 @@ export const searches: SearchConfig[] = [
|
||||
},
|
||||
];
|
||||
|
||||
/** Frontend, web and data-visualisation / geospatial — the Cantography lane. */
|
||||
export const FRONTEND_KEYWORDS = [
|
||||
"Frontend Engineer",
|
||||
"Front-end Developer",
|
||||
"React",
|
||||
"React Developer",
|
||||
"Full-Stack Developer",
|
||||
"Web Developer",
|
||||
"Data Visualization",
|
||||
"Geospatial",
|
||||
"GIS Developer",
|
||||
"WebGL",
|
||||
"Creative Technologist",
|
||||
].join(" OR ");
|
||||
|
||||
// --- Role-family sweeps ---------------------------------------------------
|
||||
// Same filters as the core searches, run across a narrower set of areas.
|
||||
// Research is Basel/Zürich only (that's where the institutes are); platform
|
||||
@@ -226,5 +241,6 @@ function sweep(
|
||||
searches.push(
|
||||
...sweep("Platform", PLATFORM_KEYWORDS, CORE_AREAS, { remote: true }),
|
||||
...sweep("Research", RESEARCH_KEYWORDS, RESEARCH_AREAS),
|
||||
...sweep("Field", FIELD_KEYWORDS, CORE_AREAS, { remote: true })
|
||||
...sweep("Field", FIELD_KEYWORDS, CORE_AREAS, { remote: true }),
|
||||
...sweep("Frontend", FRONTEND_KEYWORDS, CORE_AREAS, { remote: true })
|
||||
);
|
||||
|
||||
@@ -133,9 +133,10 @@ const area = (r: Row) => {
|
||||
if (text.includes("basel")) return "🟢 Basel";
|
||||
if (text.includes("zürich") || text.includes("zurich")) return "🔵 Zürich";
|
||||
if (text.includes("bern")) return "🟣 Bern";
|
||||
if (text.includes("luzern") || text.includes("lucerne")) return "🟠 Luzern";
|
||||
return "⚪️ Other";
|
||||
};
|
||||
const order = ["🟢 Basel", "🌍 Remote", "🔵 Zürich", "🟣 Bern", "⚪️ Other"];
|
||||
const order = ["🟢 Basel", "🌍 Remote", "🔵 Zürich", "🟣 Bern", "🟠 Luzern", "⚪️ Other"];
|
||||
const groups = new Map<string, Row[]>();
|
||||
for (const r of matched) {
|
||||
const k = area(r);
|
||||
|
||||
+121
-6
@@ -221,6 +221,60 @@ export async function fetchItJobsCh(): Promise<Job[]> {
|
||||
return allJobs;
|
||||
}
|
||||
|
||||
// arbeitnow.ch — Swiss instance of arbeitnow, which pulls postings straight from
|
||||
// company ATS feeds (Personio, Greenhouse, …). Free public JSON API, no auth:
|
||||
// /api/job-board-api?page=n, 100 per page, newest first, `links.next` null on
|
||||
// the last page. It's a rolling ~7-day window (measured 2026-11-03: 195 jobs,
|
||||
// oldest 7 days old), so 2 requests cover the whole board. Each item carries
|
||||
// the full HTML description, so the DE/FR + requiresGerman check runs here.
|
||||
// pilotjob.ch was evaluated first and is NOT a source in its own right: its
|
||||
// listing only offers `src=arbeitnow|jobroom`, i.e. it re-serves this feed plus
|
||||
// job-room.ch, which we already sweep.
|
||||
export async function fetchArbeitnow(): Promise<Job[]> {
|
||||
const allJobs: Job[] = [];
|
||||
const seen = new Set<string>();
|
||||
|
||||
for (let page = 1; page <= 20; page++) {
|
||||
if (page > 1) await Bun.sleep(500);
|
||||
process.stdout.write(` page ${page} … `);
|
||||
const res = await fetch(`https://www.arbeitnow.ch/api/job-board-api?page=${page}`, {
|
||||
headers: { ...PLAIN_HEADERS, Accept: "application/json" },
|
||||
});
|
||||
if (!res.ok) { console.log(`end (${res.status})`); break; }
|
||||
const data = (await res.json()) as {
|
||||
data: Array<{ slug: string; company_name: string; title: string; description?: string; remote: boolean; url: string; location: string; created_at: number }>;
|
||||
links: { next: string | null };
|
||||
};
|
||||
|
||||
let kept = 0;
|
||||
let langDropped = 0;
|
||||
for (const item of data.data) {
|
||||
const title = decodeHtml(item.title);
|
||||
if (!applyFilters(title) || MS_STACK_RE.test(title) || BIO_DOMAIN_RE.test(title)) continue;
|
||||
const id = `an-${item.slug}`;
|
||||
if (seen.has(id)) continue;
|
||||
seen.add(id);
|
||||
if (!isViableDescription(item.description).ok) { langDropped++; continue; }
|
||||
const loc = decodeHtml(item.location ?? "");
|
||||
allJobs.push({
|
||||
id,
|
||||
title,
|
||||
company: decodeHtml(item.company_name ?? ""),
|
||||
location: /^remote/i.test(loc) || (item.remote && !loc) ? "Remote" : loc || "Switzerland",
|
||||
postedAt: item.created_at ? new Date(item.created_at * 1000).toISOString().slice(0, 10) : "",
|
||||
link: item.url,
|
||||
search: "arbeitnow.ch",
|
||||
source: "arbeitnow.ch",
|
||||
contentChecked: true,
|
||||
});
|
||||
kept++;
|
||||
}
|
||||
console.log(`${data.data.length} items, ${kept} kept, ${langDropped} non-English`);
|
||||
if (!data.links?.next || data.data.length === 0) break;
|
||||
}
|
||||
return allJobs;
|
||||
}
|
||||
|
||||
// swisscom.com — Swisscom's own Workday career site. Small board (~47 openings,
|
||||
// Switzerland-wide), so no keyword sweep — pull the full listing and filter
|
||||
// locally, like itjobs.ch. The listing page itself is a bare JS shell (no jobs
|
||||
@@ -1054,6 +1108,12 @@ export async function fetchStellenBasel(): Promise<Job[]> {
|
||||
const DATADOME_CACHE = "results/.datadome.json";
|
||||
const CHROME_BIN = "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome";
|
||||
const BASELJOBS_ORIGIN = "https://basel-jobs.ch";
|
||||
// Sister board on the same jobchannel network: same markup, same DataDome
|
||||
// customer (one cookie, rotated or not, clears both hosts), same `/job/<slug>/<id>`
|
||||
// path for a posting listed on both. Its inventory is not a subset of
|
||||
// basel-jobs.ch, though — data-only postings 404 on the basel-jobs.ch host.
|
||||
const DATAJOBS_ORIGIN = "https://data-jobs.ch";
|
||||
const DATAJOBS_PROVINCES = "BS,BL,ZH,BE,LU";
|
||||
|
||||
interface DataDomeSession {
|
||||
cookie: string;
|
||||
@@ -1258,14 +1318,16 @@ function parseBaselJobsCards(
|
||||
html: string,
|
||||
seen: Set<string>,
|
||||
out: Job[],
|
||||
today: string
|
||||
today: string,
|
||||
host = "basel-jobs.ch",
|
||||
knownTails = new Map<string, string>()
|
||||
): { items: number; added: number } {
|
||||
const items = [...html.matchAll(/<li class="item"(.*?)<\/li>/gs)].map((m) => m[1]);
|
||||
let added = 0;
|
||||
|
||||
for (const card of items) {
|
||||
const jid = /data-gtm-jid="(\d+)"/.exec(card)?.[1];
|
||||
const linkM = /href="(https:\/\/basel-jobs\.ch\/job\/[^"]+)"/.exec(card);
|
||||
const linkM = /href="(https:\/\/(?:basel|data)-jobs\.ch\/job\/[^"]+)"/.exec(card);
|
||||
const titleM = /<span class="jobtitle">(.*?)<\/span>/s.exec(card);
|
||||
if (!jid || !linkM || !titleM) continue;
|
||||
|
||||
@@ -1276,6 +1338,14 @@ function parseBaselJobsCards(
|
||||
seen.add(id);
|
||||
added++;
|
||||
|
||||
// Already in the CSV, under either host (a posting carried by both boards
|
||||
// shares its `/job/<slug>/<id>` tail, which index.ts's link dedupe can't
|
||||
// see). Skipping here also spares the detail fetch below: re-checking every
|
||||
// known posting on every run is what got the cookie blocked once
|
||||
// data-jobs.ch's five cantons were added (measured 2026-10-22).
|
||||
const tail = linkM[1].replace(/^https:\/\/[^/]+/, "");
|
||||
if (knownTails.has(tail)) continue;
|
||||
|
||||
const title = decodeHtml(titleM[1]);
|
||||
if (!applyFilters(title)) continue;
|
||||
|
||||
@@ -1300,8 +1370,8 @@ function parseBaselJobsCards(
|
||||
// best available, same as any board that hides posting age.
|
||||
postedAt: today,
|
||||
link: linkM[1],
|
||||
search: "basel-jobs.ch",
|
||||
source: "basel-jobs.ch",
|
||||
search: host,
|
||||
source: host,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -1463,11 +1533,24 @@ export async function fetchNovartisBasel(): Promise<Job[]> {
|
||||
return jobs;
|
||||
}
|
||||
|
||||
/** `/job/<slug>/<id>` → stored link, for every basel-jobs.ch/data-jobs.ch row in the CSV. */
|
||||
async function knownJobchannelTails(): Promise<Map<string, string>> {
|
||||
const tails = new Map<string, string>();
|
||||
try {
|
||||
const text = await Bun.file("results/jobs.csv").text();
|
||||
for (const link of text.match(/https:\/\/(?:basel|data)-jobs\.ch\/job\/[^\s",]+/g) ?? []) {
|
||||
tails.set(link.replace(/^https:\/\/[^/]+/, ""), link);
|
||||
}
|
||||
} catch { /* no CSV yet */ }
|
||||
return tails;
|
||||
}
|
||||
|
||||
export async function fetchBaselJobs(): Promise<Job[]> {
|
||||
const allJobs: Job[] = [];
|
||||
const seen = new Set<string>();
|
||||
const state = { session: await loadDataDomeSession(), minted: false };
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
const knownTails = await knownJobchannelTails();
|
||||
|
||||
// Pass 1 — the board's IT category, ~300 postings over 3 pages.
|
||||
for (let page = 1; page <= 10; page++) {
|
||||
@@ -1523,11 +1606,40 @@ export async function fetchBaselJobs(): Promise<Job[]> {
|
||||
const html = await baselJobsFetch(`${BASELJOBS_ORIGIN}/search?${qs}`, state);
|
||||
if (html === null) { console.log("unavailable"); break; }
|
||||
|
||||
const { items, added } = parseBaselJobsCards(html, seen, allJobs, today);
|
||||
const { items, added } = parseBaselJobsCards(html, seen, allJobs, today, "basel-jobs.ch", knownTails);
|
||||
console.log(`${items} items, ${added} new`);
|
||||
}
|
||||
}
|
||||
|
||||
// Pass 3 — data-jobs.ch, the network's data/analytics board, filtered to
|
||||
// the cantons Rafael would work in. Runs after the basel-jobs.ch passes and
|
||||
// shares `seen` (vacancy ids are network-wide), so a posting on both boards
|
||||
// keeps its basel-jobs.ch link. "N jobs found" is the board's own listings
|
||||
// plus the "further jobs from the web search" behind `websearch=1`. The own
|
||||
// listings paginate (`page=n`, 100 per page) and past the end still serve a
|
||||
// few sponsored cards, so "no new ids" ends the walk; the web search ignores
|
||||
// `page` and returns everything at once. Measured 2026-10-22 for
|
||||
// BS,BL,ZH,BE,LU: 166 own + 98 web search = 264 unique; for BS,BL alone only
|
||||
// 7 of 24 own listings were already in the CSV via basel-jobs.ch.
|
||||
const dataJobsSearch = `${DATAJOBS_ORIGIN}/search?provinces=${DATAJOBS_PROVINCES}&lang=en-ch`;
|
||||
for (let page = 1; page <= 10; page++) {
|
||||
await Bun.sleep(400);
|
||||
process.stdout.write(` data-jobs.ch page ${page} … `);
|
||||
const html = await baselJobsFetch(`${dataJobsSearch}&page=${page}`, state);
|
||||
if (html === null) { console.log("unavailable"); break; }
|
||||
const { items, added } = parseBaselJobsCards(html, seen, allJobs, today, "data-jobs.ch", knownTails);
|
||||
console.log(`${items} items, ${added} new`);
|
||||
if (items === 0 || added === 0) break;
|
||||
}
|
||||
await Bun.sleep(400);
|
||||
process.stdout.write(` data-jobs.ch websearch … `);
|
||||
const webHtml = await baselJobsFetch(`${dataJobsSearch}&page=1&websearch=1`, state);
|
||||
if (webHtml === null) console.log("unavailable");
|
||||
else {
|
||||
const { items, added } = parseBaselJobsCards(webHtml, seen, allJobs, today, "data-jobs.ch", knownTails);
|
||||
console.log(`${items} items, ${added} new`);
|
||||
}
|
||||
|
||||
// Detail pages carry the full description inline, so the DE/FR and
|
||||
// German-requirement checks run here instead of costing index.ts a fetch that
|
||||
// would hit DataDome without the cookie and be dropped as unreachable.
|
||||
@@ -1536,7 +1648,10 @@ export async function fetchBaselJobs(): Promise<Job[]> {
|
||||
await Bun.sleep(300);
|
||||
process.stdout.write(` checking ${job.title} … `);
|
||||
const page = await baselJobsFetch(job.link, state);
|
||||
if (page === null) { console.log("unreachable, keeping"); viable.push(job); continue; }
|
||||
// Not kept: a blocked cookie fails every remaining detail page at once, and
|
||||
// keeping those would land them in the CSV unchecked for good. Left out,
|
||||
// they're simply retried on the next run.
|
||||
if (page === null) { console.log("unreachable, skipping"); continue; }
|
||||
|
||||
// Only the posting's own text, never the page. basel-jobs.ch is a German
|
||||
// site, so its nav/footer/category chrome alone clears the ≥ 6 DE/FR word
|
||||
|
||||
@@ -161,7 +161,7 @@ export const BIO_DOMAIN_RE = new RegExp(
|
||||
"i"
|
||||
);
|
||||
|
||||
export const SOFTWARE_TITLE_RE = /\b(software|developer|architect|devops|devsecops|platform|data|backend|frontend|fullstack|full.stack|tech\s*lead|technical|typescript|python|rails|node\.?js|ai\b|agentic|llm|machine\s*learning|cloud|security|analytics|sre|reliability|infrastructure|mlops|solutions?|consultant|applications?\s*engineer|system[\s-]*engineer|informatics|scientific\s*(?:software|comput\w*|programmer)|research\s*(?:software|engineer)|hpc|forward\s*deployed|fde|palantir|field\s*engineer|customer\s*engineer|sales\s*engineer|pre.?sales|integration\s*engineer|api\s*engineer|founding\s*engineer|product\s*engineer|automation\s*engineer|director|product\s*manager|engineering\s*manager|it\s*manager|it\s*project|staff\s*engineer|principal\s*engineer|cto|vp\s*eng)\b/i;
|
||||
export const SOFTWARE_TITLE_RE = /\b(software|developer|architect|devops|devsecops|platform|data|backend|frontend|front.end|fullstack|full.stack|react|web\s*developer|gis|geospatial|visuali[sz]ation|webgl|graphics|creative\s*technologist|cartograph\w*|tech\s*lead|technical|typescript|python|rails|node\.?js|ai\b|agentic|llm|machine\s*learning|cloud|security|analytics|sre|reliability|infrastructure|mlops|solutions?|consultant|applications?\s*engineer|system[\s-]*engineer|informatics|scientific\s*(?:software|comput\w*|programmer)|research\s*(?:software|engineer)|hpc|forward\s*deployed|fde|palantir|field\s*engineer|customer\s*engineer|sales\s*engineer|pre.?sales|integration\s*engineer|api\s*engineer|founding\s*engineer|product\s*engineer|automation\s*engineer|director|product\s*manager|engineering\s*manager|it\s*manager|it\s*project|staff\s*engineer|principal\s*engineer|cto|vp\s*eng)\b/i;
|
||||
|
||||
const DE_CONTENT_RE = /\b(Sie|Ihre?|Ihnen|Wir|wir|und\s|für|mit\s|von\s|bei\s|das\s|ist\s|sind\s|haben\s|werden\s|können|suchen|bieten|Kenntnisse|Erfahrung|Anforderungen|Aufgaben|Stellenbeschreibung|Stellenangebot|Bewerbung|Ausbildung|Berufserfahrung|Teamarbeit|Vollzeit|Teilzeit|unbefristet|Abschluss|gesucht|Deutsch|Englisch|Schweizer)\b/g;
|
||||
const FR_CONTENT_RE = /\b(vous|nous|votre|notre|pour\s|avec\s|profil|poste|emploi|entreprise|compétences|expérience|formation|équipe|rejoindre|recherchons|offrons|missions|candidature|CDI|CDD|temps\s+plein|temps\s+partiel)\b/g;
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
import { searches } from "./config.ts";
|
||||
import { parseCsvLine } from "./csv.ts";
|
||||
import { fetchJobs, isViableContent, isViableUrl } from "./fetch.ts";
|
||||
import { fetchDataCareer, fetchStellenBasel, fetchItJobsCh, fetchJobRoom, fetchBaselIntlOrgs, fetchJobBasel, fetchEnglishJobSearch, fetchBaselJobs, fetchSciCore, fetchSwisscom, fetchNovartisBasel, fetchMoreJobs, fetchStartupJobs } from "./fetch-other.ts";
|
||||
import { fetchDataCareer, fetchStellenBasel, fetchItJobsCh, fetchJobRoom, fetchBaselIntlOrgs, fetchJobBasel, fetchEnglishJobSearch, fetchBaselJobs, fetchSciCore, fetchSwisscom, fetchNovartisBasel, fetchMoreJobs, fetchStartupJobs, fetchArbeitnow } from "./fetch-other.ts";
|
||||
|
||||
const CSV = "./results/jobs.csv";
|
||||
const OLD_HEADER = "date_found,posted_at,search,title,company,link";
|
||||
@@ -193,6 +193,8 @@ const extraSources: Array<{ label: string; fetch: () => ReturnType<typeof fetchD
|
||||
{ label: "more-jobs.ch", fetch: fetchMoreJobs, dedupeByTitle: true },
|
||||
// Aggregator of company ATS feeds (Wise, Stripe, Avaloq…) that LinkedIn also carries.
|
||||
{ label: "startup.jobs", fetch: fetchStartupJobs, dedupeByTitle: true },
|
||||
// ATS-fed board that overlaps ~50% with LinkedIn (measured 2026-11-03).
|
||||
{ label: "arbeitnow.ch", fetch: fetchArbeitnow, dedupeByTitle: true },
|
||||
];
|
||||
|
||||
async function processExtraSource({ label, fetch: fetchFn, dedupeByTitle }: (typeof extraSources)[number]): Promise<void> {
|
||||
|
||||
@@ -11,6 +11,32 @@ Scope: `linkme/cv/`, `linkme/motivation*.md`, `rafapolo/` portfolio + `rafapolo/
|
||||
- [ ] Reconsider German-first corporates (Coop, Valora, Helvetia) while at B1 — screening
|
||||
failure precedes any engineering assessment.
|
||||
|
||||
## Referência da Swisscom (aberto, ação do Rafael)
|
||||
|
||||
- [ ] Pedir à Swisscom uma confirmação de emprego (Arbeitsbestätigung, Art. 330a al. 2 CO),
|
||||
só com função e datas, sem avaliação. Fallback caso a correção da carta de março de
|
||||
2025 ("moderate results") seja recusada: a confirmação pesa menos que a carta, mas
|
||||
também chama atenção.
|
||||
- Para: Swisscom (Switzerland) Ltd, Group Human Resources, HR Advisory, CH-3050 Berne
|
||||
- Dados: DevOps Engineer III, Transport IP Network, Zürich, 01/03/2024 a 28/02/2025
|
||||
- Ao receber: salvar em `rafapolo/job/credentials/references/` e usar no dossiê no
|
||||
lugar da carta.
|
||||
|
||||
Rascunho:
|
||||
|
||||
> Dear HR Advisory team,
|
||||
>
|
||||
> I was employed by Swisscom (Switzerland) Ltd from March 1, 2024 until
|
||||
> February 28, 2025 as DevOps Engineer III in the Transport IP Network unit in Zürich.
|
||||
>
|
||||
> Under Art. 330a para. 2 CO, I would like to request a confirmation of employment
|
||||
> stating only the nature and duration of my employment, without any assessment of
|
||||
> performance or conduct.
|
||||
>
|
||||
> Thank you in advance.
|
||||
>
|
||||
> Rafael Gomes de Oliveira Polo
|
||||
|
||||
## Not changing
|
||||
|
||||
Scraper is fine — `fetch.ts`, filter pipeline, `index.ts` all working. No changes needed.
|
||||
|
||||
Reference in New Issue
Block a user