fix: catch German descriptions by umlaut, not just DE word count

The >=6 DE_CONTENT_RE word threshold needs a reasonably long snippet to
trip. An Expleo (Bern) posting slipped through because a concurrent fetch
under load returned a truncated description with too few matches, even
though the full text (72 matches) was entirely German.

A bare umlaut or eszett is a near-certain German signal and survives
truncation — one German sentence almost always carries one. Zurich/
Zurcher is excluded as the one umlaut word that shows up constantly in
genuinely English-language Swiss postings, as a place name.

Applied in isViableContent, isEnglishContent, isViableUrl and the
fetch-other.ts sources, so every description path gets the same check.
This commit is contained in:
2026-09-05 18:04:22 +02:00
parent afddbe496d
commit 2b7bf3b5f7
2 changed files with 20 additions and 1 deletions
+2 -1
View File
@@ -1,4 +1,4 @@
import { NON_ENGLISH_RE, JUNIOR_TITLE_RE, SOFTWARE_TITLE_RE, requiresGerman } from "./fetch.ts";
import { NON_ENGLISH_RE, JUNIOR_TITLE_RE, SOFTWARE_TITLE_RE, requiresGerman, hasNonEnglishUmlaut } from "./fetch.ts";
import type { Job } from "./fetch.ts";
const HEADERS = {
@@ -60,6 +60,7 @@ const DE_FR_WORDS_RE =
function isViableDescription(description?: string): { ok: boolean; reason?: string } {
if (!description) return { ok: true }; // nothing to judge — let the URL check in index.ts decide
const text = description.replace(/<[^>]+>/g, " ").replace(/&[a-z]+;/g, " ");
if (hasNonEnglishUmlaut(text)) return { ok: false, reason: "non-English (umlaut)" };
if ((text.match(DE_FR_WORDS_RE) ?? []).length >= 6) return { ok: false, reason: "non-English" };
if (requiresGerman(text)) return { ok: false, reason: "requires German > B1" };
return { ok: true };