rodado/linkme
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: back off longer on 429s in description fetch

Browse Source 
fetchDescription and isEnglishContent retried rate-limited (429) requests
after only 350/700/1050ms, too short under concurrent load, so retries kept
hitting 429, exhausted, and fell back to the "can't read it, so keep"
default. That let an all-German CLEEVEN posting (job 4465075383) through
undetected and out to Telegram. Match the 1s/2s/4s backoff already used for
listing-page 429 retries.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011zgfzuFy3bAbs8GTh9apFc
This commit is contained in:
rodado
2026-09-10 12:15:58 +02:00
co-authored by Claude Sonnet 5
parent 821bd19ff7
commit c7b86e06f4
1 changed files with 9 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
fetch.ts
+9 -2
View File
@@ -275,7 +275,13 @@ export function requiresGerman(text: string): boolean {
export async function fetchDescription(id: string): Promise<string | null> { export async function fetchDescription(id: string): Promise<string | null> {
for (let attempt = 0; attempt < 3; attempt++) { for (let attempt = 0; attempt < 3; attempt++) {
try { try {
await Bun.sleep(350 * (attempt + 1)); // 429s need the same backoff as the listing-page retry (1s/2s/4s) — the
// old 350/700/1050ms wait was too short under concurrent load, so this
// exhausted its retries while still rate-limited and fell through to the
// "can't read it, so keep" default below, letting German postings (e.g.
// CLEEVEN, job 4465075383, all-German with zero English content) slip
// past isViableContent entirely on 2026-09-10.
await Bun.sleep(1000 * 2 ** attempt);
const res = await fetch(`https://www.linkedin.com/jobs/view/${id}/`, { headers: HEADERS }); const res = await fetch(`https://www.linkedin.com/jobs/view/${id}/`, { headers: HEADERS });
const html = await res.text(); const html = await res.text();
const m = html.match(/class="show-more-less-html__markup[^"]*">([\s\S]{0,8000}?)<\/div>/); const m = html.match(/class="show-more-less-html__markup[^"]*">([\s\S]{0,8000}?)<\/div>/);
@@ -307,7 +313,8 @@ export async function isEnglishContent(id: string): Promise<boolean> {
// false "English" result. Back off and try again before falling back. // false "English" result. Back off and try again before falling back.
for (let attempt = 0; attempt < 3; attempt++) { for (let attempt = 0; attempt < 3; attempt++) {
try { try {
await Bun.sleep(350 * (attempt + 1)); // Same 1s/2s/4s backoff as fetchDescription — see its comment.
await Bun.sleep(1000 * 2 ** attempt);
const res = await fetch(`https://www.linkedin.com/jobs/view/${id}/`, { headers: HEADERS }); const res = await fetch(`https://www.linkedin.com/jobs/view/${id}/`, { headers: HEADERS });
const html = await res.text(); const html = await res.text();
const m = html.match(/class="show-more-less-html__markup[^"]*">([\s\S]{0,8000}?)<\/div>/); const m = html.match(/class="show-more-less-html__markup[^"]*">([\s\S]{0,8000}?)<\/div>/);