perf: run fetch sources concurrently instead of sequentially
Every LinkedIn search and every fetch-other.ts source hits a different host, so there was no real reason for the old for-loop to wait on each one in turn - that's what made a full run take 30-40+ minutes. Runs them concurrently now (bounded: 3 at a time within each group, both groups started together), which brought a real run down to under 90 seconds. existingLinks.add() now happens synchronously right after the .has() check and before any await, which is what makes concurrent workers safe against double-claiming the same link. Concurrency surfaced a real regression: LinkedIn started returning occasional 429s under parallel load, and fetchJobs treated that the same as end-of-results, silently truncating the rest of that search. Added retry-with-backoff (3 attempts, 1s/2s/4s) so a rate limit no longer costs real listings.
This commit is contained in:
@@ -368,7 +368,15 @@ export async function fetchJobs(config: SearchConfig, { raw = false } = {}): Pro
|
||||
const url = `${baseUrl}&start=${page * PAGE_SIZE}`;
|
||||
process.stdout.write(` page ${page + 1} … `);
|
||||
|
||||
const res = await fetch(url, { headers: HEADERS });
|
||||
// 429s became common once index.ts started running several searches
|
||||
// concurrently — a transient rate-limit, not a real end-of-results
|
||||
// signal, so retry with backoff instead of silently truncating the
|
||||
// search (losing every page after this one).
|
||||
let res = await fetch(url, { headers: HEADERS });
|
||||
for (let attempt = 0; res.status === 429 && attempt < 3; attempt++) {
|
||||
await Bun.sleep(1000 * 2 ** attempt); // 1s, 2s, 4s
|
||||
res = await fetch(url, { headers: HEADERS });
|
||||
}
|
||||
if (!res.ok) {
|
||||
console.error(`failed (${res.status})`);
|
||||
break;
|
||||
|
||||
Reference in New Issue
Block a user