fix: catch German descriptions by umlaut, not just DE word count
The >=6 DE_CONTENT_RE word threshold needs a reasonably long snippet to trip. An Expleo (Bern) posting slipped through because a concurrent fetch under load returned a truncated description with too few matches, even though the full text (72 matches) was entirely German. A bare umlaut or eszett is a near-certain German signal and survives truncation — one German sentence almost always carries one. Zurich/ Zurcher is excluded as the one umlaut word that shows up constantly in genuinely English-language Swiss postings, as a place name. Applied in isViableContent, isEnglishContent, isViableUrl and the fetch-other.ts sources, so every description path gets the same check.
This commit is contained in:
+2
-1
@@ -1,4 +1,4 @@
|
||||
import { NON_ENGLISH_RE, JUNIOR_TITLE_RE, SOFTWARE_TITLE_RE, requiresGerman } from "./fetch.ts";
|
||||
import { NON_ENGLISH_RE, JUNIOR_TITLE_RE, SOFTWARE_TITLE_RE, requiresGerman, hasNonEnglishUmlaut } from "./fetch.ts";
|
||||
import type { Job } from "./fetch.ts";
|
||||
|
||||
const HEADERS = {
|
||||
@@ -60,6 +60,7 @@ const DE_FR_WORDS_RE =
|
||||
function isViableDescription(description?: string): { ok: boolean; reason?: string } {
|
||||
if (!description) return { ok: true }; // nothing to judge — let the URL check in index.ts decide
|
||||
const text = description.replace(/<[^>]+>/g, " ").replace(/&[a-z]+;/g, " ");
|
||||
if (hasNonEnglishUmlaut(text)) return { ok: false, reason: "non-English (umlaut)" };
|
||||
if ((text.match(DE_FR_WORDS_RE) ?? []).length >= 6) return { ok: false, reason: "non-English" };
|
||||
if (requiresGerman(text)) return { ok: false, reason: "requires German > B1" };
|
||||
return { ok: true };
|
||||
|
||||
Reference in New Issue
Block a user