chore: reorganize repo structure and update docs
- move temp scripts to scripts/, cv files to cv/, jobs.csv to results/ - update index.ts and recheck.ts CSV path to results/jobs.csv - fix imports in scripts/ to use ../config.ts and ../fetch.ts - update CLAUDE.md and README.md to reflect new structure - add plans/ with LinkedIn post drafts - update .gitignore: ignore results/, jobs/, cv binaries, .DS_Store
This commit is contained in:
@@ -5,75 +5,73 @@ Scrapes LinkedIn job listings into a local CSV. Run it daily, delete rows you've
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
bun fetch # scrape and append new jobs to jobs.csv
|
||||
bun fetch # scrape and append new jobs to results/jobs.csv
|
||||
bun recheck # show jobs currently filtered out + keyword suggestions
|
||||
```
|
||||
|
||||
No setup needed. Results are appended to `jobs.csv` — reruns are safe, already-seen jobs are skipped by ID.
|
||||
No setup needed. Results are appended to `results/jobs.csv` — reruns are safe, already-seen jobs are skipped by ID.
|
||||
|
||||
## Structure
|
||||
|
||||
```
|
||||
config.ts # search keywords and areas
|
||||
fetch.ts # scraper, HTML parser, filters
|
||||
fetch-other.ts # extra sources (datacareer.ch, stellen-basel.ch)
|
||||
index.ts # orchestrator: runs searches, writes CSV
|
||||
recheck.ts # raw-mode fetch, lists filtered-out jobs
|
||||
scripts/ # one-off search scripts (temp, not in config.ts)
|
||||
cv/ # polo.md (profile) + CV files
|
||||
results/ # jobs.csv output
|
||||
jobs/ # notes on specific job leads
|
||||
plans/ # LinkedIn post drafts
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
`jobs.csv` — one row per job:
|
||||
`results/jobs.csv` — one row per job:
|
||||
|
||||
| column | description |
|
||||
|---|---|
|
||||
| `date_found` | date you ran the fetch |
|
||||
| `posted_at` | date the job was posted on LinkedIn |
|
||||
| `posted_at` | date the job was posted |
|
||||
| `search` | which search config matched it |
|
||||
| `title` | job title |
|
||||
| `company` | hiring company |
|
||||
| `link` | canonical LinkedIn job URL |
|
||||
| `link` | canonical job URL |
|
||||
| `source` | site the job came from |
|
||||
|
||||
Open in any spreadsheet app. Delete rows as you apply or dismiss.
|
||||
|
||||
## How it works
|
||||
|
||||
- **Paginated scraping** — fetches up to 6 pages per search (150 jobs) with a 400 ms delay between pages to avoid rate limiting
|
||||
- **Card-boundary-safe parsing** — each job card is isolated before extracting fields, preventing content from adjacent cards bleeding into company or date values
|
||||
- **Language filter** — drops German (`m/w/d`, umlauts, gender-marked vocabulary) and French (`H/F`, accented characters) job titles automatically
|
||||
- **Software-role filter** — requires the title to contain a relevant term (software, developer, data, AI, backend, frontend, devops, platform, architect, tech lead, typescript, python, rails, node.js, ML, cloud, security, SRE, solutions, consultant, applications/system engineer, director, product/IT/engineering manager, staff/principal engineer, CTO, VP engineering, etc.) — eliminates aviation, mechanical, and pharma-process roles that match "Engineer" broadly
|
||||
- **Seniority filter** — drops junior, jr., intern, trainee, apprentice, werkstudent, and entry-level titles
|
||||
- **Global deduplication** — job IDs are indexed from the existing CSV on startup; jobs you've already seen are never re-added, even if you re-run
|
||||
- **30-day recency** — only fetches jobs posted in the last month (`f_TPR=r2592000`)
|
||||
- **Paginated scraping** — fetches up to 6 pages per search (150 jobs) with a 400 ms delay between pages
|
||||
- **Card-boundary-safe parsing** — each job card is isolated before extracting fields
|
||||
- **Language filter** — drops German (`m/w/d`, umlauts, gender markers) and French (`H/F`, accents) titles
|
||||
- **Software-role filter** — requires a relevant term in the title (software, developer, data, AI, backend, frontend, devops, platform, architect, tech lead, typescript, python, rails, node.js, ML, cloud, security, SRE, etc.)
|
||||
- **Seniority filter** — drops junior, intern, trainee, apprentice, werkstudent, entry-level
|
||||
- **Global deduplication** — IDs indexed from existing CSV on startup; already-seen jobs are never re-added
|
||||
- **30-day recency** — only jobs posted in the last month
|
||||
|
||||
## recheck
|
||||
|
||||
`bun recheck` fetches all three search areas in raw mode (bypassing the software-role filter) and lists every job that would be filtered out — useful for spotting false negatives. It also prints a word-frequency table of the most common terms in filtered titles so you can decide what to add to `SOFTWARE_TITLE_RE`.
|
||||
`bun recheck` fetches all searches in raw mode (bypassing the software-role filter) and lists every job that would be filtered out — useful for spotting false negatives. Also prints a word-frequency table of filtered titles to help decide what to add to `SOFTWARE_TITLE_RE`.
|
||||
|
||||
## Temporary searches
|
||||
|
||||
Write a throwaway script in `scripts/` instead of editing `config.ts`:
|
||||
|
||||
```bash
|
||||
bun run scripts/bern-temp.ts
|
||||
```
|
||||
|
||||
Scripts in `scripts/` import from `../config.ts` and `../fetch.ts`.
|
||||
|
||||
## Configure searches
|
||||
|
||||
Edit `config.ts`:
|
||||
|
||||
```ts
|
||||
export const KEYWORDS = [
|
||||
"Software Engineer",
|
||||
"Solutions Engineer",
|
||||
"Full Stack Engineer",
|
||||
"Backend Engineer",
|
||||
"Data Engineer",
|
||||
"AI Engineer",
|
||||
"Applications Engineer",
|
||||
"Platform Engineer",
|
||||
"DevOps Engineer",
|
||||
"Staff Engineer",
|
||||
"Principal Engineer",
|
||||
"Tech Lead",
|
||||
"Developer",
|
||||
"Ruby on Rails",
|
||||
"Python",
|
||||
"Node.js",
|
||||
"TypeScript",
|
||||
"Artificial Intelligence",
|
||||
"Machine Learning",
|
||||
"Data Science",
|
||||
"AI Consultant",
|
||||
"ML Engineer",
|
||||
"Director of Engineering",
|
||||
"Engineering Manager",
|
||||
"Lead Developer",
|
||||
"System Engineer",
|
||||
"Solution Engineer",
|
||||
].join(" OR ");
|
||||
export const KEYWORDS = "Software Engineer OR Data Engineer OR ...";
|
||||
|
||||
export const searches: SearchConfig[] = [
|
||||
{
|
||||
@@ -84,30 +82,17 @@ export const searches: SearchConfig[] = [
|
||||
workType: ["onsite", "hybrid"],
|
||||
timePosted: "month",
|
||||
},
|
||||
{
|
||||
label: "Basel Metropolitan Area",
|
||||
keywords: KEYWORDS,
|
||||
geoId: "90009885",
|
||||
location: "Basel Metropolitan Area",
|
||||
workType: ["onsite", "hybrid"],
|
||||
timePosted: "month",
|
||||
},
|
||||
{
|
||||
label: "Basel-Country",
|
||||
keywords: KEYWORDS,
|
||||
geoId: "107117530",
|
||||
location: "Basel-Country",
|
||||
workType: ["onsite", "hybrid"],
|
||||
timePosted: "month",
|
||||
},
|
||||
];
|
||||
```
|
||||
|
||||
`geoId` values for the Basel area:
|
||||
Known Swiss geoIds:
|
||||
- `103383283` — Basel (city)
|
||||
- `90009885` — Basel Metropolitan Area
|
||||
- `107117530` — Basel-Country
|
||||
- `102436504` — Zürich (city)
|
||||
- `107814425` — Zürich Metropolitan Area
|
||||
- `90009888` — Zürich Area
|
||||
|
||||
You can also paste any LinkedIn search URL directly as `url:` instead of building from filters.
|
||||
|
||||
`SearchConfig` also supports `jobType` (full-time, part-time, contract, temporary, internship) and `seniority` (internship, entry, associate, mid-senior, director, executive) to pass LinkedIn's native API filters.
|
||||
`SearchConfig` also supports `jobType` (full-time, part-time, contract, temporary, internship) and `seniority` (internship, entry, associate, mid-senior, director, executive).
|
||||
|
||||
Reference in New Issue
Block a user