perf: run fetch sources concurrently instead of sequentially

Every LinkedIn search and every fetch-other.ts source hits a different
host, so there was no real reason for the old for-loop to wait on each
one in turn - that's what made a full run take 30-40+ minutes. Runs
them concurrently now (bounded: 3 at a time within each group, both
groups started together), which brought a real run down to under 90
seconds.

existingLinks.add() now happens synchronously right after the .has()
check and before any await, which is what makes concurrent workers
safe against double-claiming the same link.

Concurrency surfaced a real regression: LinkedIn started returning
occasional 429s under parallel load, and fetchJobs treated that the
same as end-of-results, silently truncating the rest of that search.
Added retry-with-backoff (3 attempts, 1s/2s/4s) so a rate limit no
longer costs real listings.
This commit is contained in:
2026-08-30 13:07:32 +02:00
parent d195d67c2a
commit ba3039150f
3 changed files with 96 additions and 48 deletions
+9 -1
View File
@@ -368,7 +368,15 @@ export async function fetchJobs(config: SearchConfig, { raw = false } = {}): Pro
const url = `${baseUrl}&start=${page * PAGE_SIZE}`;
process.stdout.write(` page ${page + 1} … `);
const res = await fetch(url, { headers: HEADERS });
// 429s became common once index.ts started running several searches
// concurrently — a transient rate-limit, not a real end-of-results
// signal, so retry with backoff instead of silently truncating the
// search (losing every page after this one).
let res = await fetch(url, { headers: HEADERS });
for (let attempt = 0; res.status === 429 && attempt < 3; attempt++) {
await Bun.sleep(1000 * 2 ** attempt); // 1s, 2s, 4s
res = await fetch(url, { headers: HEADERS });
}
if (!res.ok) {
console.error(`failed (${res.status})`);
break;