Add content-level language filter, recheck command, expanded keywords
- isEnglishContent() fetches each job page and drops descriptions with ≥6 German or French indicator words; called for every new job in index.ts before writing to CSV - New `bun recheck` command (recheck.ts) runs searches in raw mode, lists filtered-out jobs, and prints word-frequency suggestions for SOFTWARE_TITLE_RE - Expanded KEYWORDS and SOFTWARE_TITLE_RE to cover ML/AI, system/applications engineer, director, informatics, and more title variants - Added `raw` flag to fetchJobs() to bypass the software-role filter - Add CLAUDE.md with project workflow docs; add CV files and city temp scripts
This commit is contained in:
@@ -5,7 +5,8 @@ Scrapes LinkedIn job listings into a local CSV. Run it daily, delete rows you've
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
bun fetch
|
||||
bun fetch # scrape and append new jobs to jobs.csv
|
||||
bun recheck # show jobs currently filtered out + keyword suggestions
|
||||
```
|
||||
|
||||
No setup needed. Results are appended to `jobs.csv` — reruns are safe, already-seen jobs are skipped by ID.
|
||||
@@ -30,24 +31,48 @@ Open in any spreadsheet app. Delete rows as you apply or dismiss.
|
||||
- **Paginated scraping** — fetches up to 6 pages per search (150 jobs) with a 400 ms delay between pages to avoid rate limiting
|
||||
- **Card-boundary-safe parsing** — each job card is isolated before extracting fields, preventing content from adjacent cards bleeding into company or date values
|
||||
- **Language filter** — drops German (`m/w/d`, umlauts, gender-marked vocabulary) and French (`H/F`, accented characters) job titles automatically
|
||||
- **Software-role filter** — requires the title to contain a relevant term (software, developer, data, AI, backend, devops, platform, architect, technical, solutions, consultant, product/IT/engineering manager, etc.) — eliminates aviation, mechanical, and pharma-process roles that match "Engineer" broadly
|
||||
- **Seniority filter** — drops junior, intern, trainee, and apprentice titles
|
||||
- **Global deduplication** — job IDs are indexed from the existing CSV on startup; jobs you've already seen are never re-added, even if you delete and re-run
|
||||
- **Software-role filter** — requires the title to contain a relevant term (software, developer, data, AI, backend, frontend, devops, platform, architect, tech lead, typescript, python, rails, node.js, ML, cloud, security, SRE, solutions, consultant, applications/system engineer, director, product/IT/engineering manager, staff/principal engineer, CTO, VP engineering, etc.) — eliminates aviation, mechanical, and pharma-process roles that match "Engineer" broadly
|
||||
- **Seniority filter** — drops junior, jr., intern, trainee, apprentice, werkstudent, and entry-level titles
|
||||
- **Global deduplication** — job IDs are indexed from the existing CSV on startup; jobs you've already seen are never re-added, even if you re-run
|
||||
- **30-day recency** — only fetches jobs posted in the last month (`f_TPR=r2592000`)
|
||||
|
||||
## recheck
|
||||
|
||||
`bun recheck` fetches all three search areas in raw mode (bypassing the software-role filter) and lists every job that would be filtered out — useful for spotting false negatives. It also prints a word-frequency table of the most common terms in filtered titles so you can decide what to add to `SOFTWARE_TITLE_RE`.
|
||||
|
||||
## Configure searches
|
||||
|
||||
Edit `config.ts`:
|
||||
|
||||
```ts
|
||||
const KEYWORDS = [
|
||||
export const KEYWORDS = [
|
||||
"Software Engineer",
|
||||
"Solutions Engineer",
|
||||
"Full Stack Engineer",
|
||||
"Backend Engineer",
|
||||
"Data Engineer",
|
||||
"AI Engineer",
|
||||
"Applications Engineer",
|
||||
"Platform Engineer",
|
||||
"DevOps Engineer",
|
||||
"Staff Engineer",
|
||||
"Principal Engineer",
|
||||
"Tech Lead",
|
||||
"Developer",
|
||||
"Ruby on Rails",
|
||||
"Python",
|
||||
// add or remove freely
|
||||
"Node.js",
|
||||
"TypeScript",
|
||||
"Artificial Intelligence",
|
||||
"Machine Learning",
|
||||
"Data Science",
|
||||
"AI Consultant",
|
||||
"ML Engineer",
|
||||
"Director of Engineering",
|
||||
"Engineering Manager",
|
||||
"Lead Developer",
|
||||
"System Engineer",
|
||||
"Solution Engineer",
|
||||
].join(" OR ");
|
||||
|
||||
export const searches: SearchConfig[] = [
|
||||
@@ -59,6 +84,22 @@ export const searches: SearchConfig[] = [
|
||||
workType: ["onsite", "hybrid"],
|
||||
timePosted: "month",
|
||||
},
|
||||
{
|
||||
label: "Basel Metropolitan Area",
|
||||
keywords: KEYWORDS,
|
||||
geoId: "90009885",
|
||||
location: "Basel Metropolitan Area",
|
||||
workType: ["onsite", "hybrid"],
|
||||
timePosted: "month",
|
||||
},
|
||||
{
|
||||
label: "Basel-Country",
|
||||
keywords: KEYWORDS,
|
||||
geoId: "107117530",
|
||||
location: "Basel-Country",
|
||||
workType: ["onsite", "hybrid"],
|
||||
timePosted: "month",
|
||||
},
|
||||
];
|
||||
```
|
||||
|
||||
@@ -68,3 +109,5 @@ export const searches: SearchConfig[] = [
|
||||
- `107117530` — Basel-Country
|
||||
|
||||
You can also paste any LinkedIn search URL directly as `url:` instead of building from filters.
|
||||
|
||||
`SearchConfig` also supports `jobType` (full-time, part-time, contract, temporary, internship) and `seniority` (internship, entry, associate, mid-senior, director, executive) to pass LinkedIn's native API filters.
|
||||
|
||||
Reference in New Issue
Block a user