Add content-level language filter, recheck command, expanded keywords

- isEnglishContent() fetches each job page and drops descriptions with ≥6 German or French indicator words; called for every new job in index.ts before writing to CSV
- New `bun recheck` command (recheck.ts) runs searches in raw mode, lists filtered-out jobs, and prints word-frequency suggestions for SOFTWARE_TITLE_RE
- Expanded KEYWORDS and SOFTWARE_TITLE_RE to cover ML/AI, system/applications engineer, director, informatics, and more title variants
- Added `raw` flag to fetchJobs() to bypass the software-role filter
- Add CLAUDE.md with project workflow docs; add CV files and city temp scripts
This commit is contained in:
2026-05-12 13:42:27 +02:00
parent ffe2b4467a
commit bcb6ae5ea3
19 changed files with 1201 additions and 14 deletions
+49 -6
View File
@@ -5,7 +5,8 @@ Scrapes LinkedIn job listings into a local CSV. Run it daily, delete rows you've
## Usage
```bash
bun fetch
bun fetch # scrape and append new jobs to jobs.csv
bun recheck # show jobs currently filtered out + keyword suggestions
```
No setup needed. Results are appended to `jobs.csv` — reruns are safe, already-seen jobs are skipped by ID.
@@ -30,24 +31,48 @@ Open in any spreadsheet app. Delete rows as you apply or dismiss.
- **Paginated scraping** — fetches up to 6 pages per search (150 jobs) with a 400 ms delay between pages to avoid rate limiting
- **Card-boundary-safe parsing** — each job card is isolated before extracting fields, preventing content from adjacent cards bleeding into company or date values
- **Language filter** — drops German (`m/w/d`, umlauts, gender-marked vocabulary) and French (`H/F`, accented characters) job titles automatically
- **Software-role filter** — requires the title to contain a relevant term (software, developer, data, AI, backend, devops, platform, architect, technical, solutions, consultant, product/IT/engineering manager, etc.) — eliminates aviation, mechanical, and pharma-process roles that match "Engineer" broadly
- **Seniority filter** — drops junior, intern, trainee, and apprentice titles
- **Global deduplication** — job IDs are indexed from the existing CSV on startup; jobs you've already seen are never re-added, even if you delete and re-run
- **Software-role filter** — requires the title to contain a relevant term (software, developer, data, AI, backend, frontend, devops, platform, architect, tech lead, typescript, python, rails, node.js, ML, cloud, security, SRE, solutions, consultant, applications/system engineer, director, product/IT/engineering manager, staff/principal engineer, CTO, VP engineering, etc.) — eliminates aviation, mechanical, and pharma-process roles that match "Engineer" broadly
- **Seniority filter** — drops junior, jr., intern, trainee, apprentice, werkstudent, and entry-level titles
- **Global deduplication** — job IDs are indexed from the existing CSV on startup; jobs you've already seen are never re-added, even if you re-run
- **30-day recency** — only fetches jobs posted in the last month (`f_TPR=r2592000`)
## recheck
`bun recheck` fetches all three search areas in raw mode (bypassing the software-role filter) and lists every job that would be filtered out — useful for spotting false negatives. It also prints a word-frequency table of the most common terms in filtered titles so you can decide what to add to `SOFTWARE_TITLE_RE`.
## Configure searches
Edit `config.ts`:
```ts
const KEYWORDS = [
export const KEYWORDS = [
"Software Engineer",
"Solutions Engineer",
"Full Stack Engineer",
"Backend Engineer",
"Data Engineer",
"AI Engineer",
"Applications Engineer",
"Platform Engineer",
"DevOps Engineer",
"Staff Engineer",
"Principal Engineer",
"Tech Lead",
"Developer",
"Ruby on Rails",
"Python",
// add or remove freely
"Node.js",
"TypeScript",
"Artificial Intelligence",
"Machine Learning",
"Data Science",
"AI Consultant",
"ML Engineer",
"Director of Engineering",
"Engineering Manager",
"Lead Developer",
"System Engineer",
"Solution Engineer",
].join(" OR ");
export const searches: SearchConfig[] = [
@@ -59,6 +84,22 @@ export const searches: SearchConfig[] = [
workType: ["onsite", "hybrid"],
timePosted: "month",
},
{
label: "Basel Metropolitan Area",
keywords: KEYWORDS,
geoId: "90009885",
location: "Basel Metropolitan Area",
workType: ["onsite", "hybrid"],
timePosted: "month",
},
{
label: "Basel-Country",
keywords: KEYWORDS,
geoId: "107117530",
location: "Basel-Country",
workType: ["onsite", "hybrid"],
timePosted: "month",
},
];
```
@@ -68,3 +109,5 @@ export const searches: SearchConfig[] = [
- `107117530` — Basel-Country
You can also paste any LinkedIn search URL directly as `url:` instead of building from filters.
`SearchConfig` also supports `jobType` (full-time, part-time, contract, temporary, internship) and `seniority` (internship, entry, associate, mid-senior, director, executive) to pass LinkedIn's native API filters.