fix: systematic sweep of NON_ENGLISH_RE gaps in gender/compound notation
Audited all 2949 titles in jobs.csv against broader German-signal heuristics than the regex itself uses, to find what it still misses: - gender suffix generalized beyond slash/colon to also catch */_ and the adjectival -r form (Engineer*in, Mitarbeiter_in, Verantwortliche:r) - 'und'/'oder' were missing entirely — basic conjunctions, zero English overlap - occupational nouns (Mitarbeiter, Ingenieur, Berater, Leiter, Informatiker, Verantwortliche) now also match as compound suffixes (Aussendienstmitarbeiter, Vertriebsingenieur) instead of only standalone, since German compounds have no separator - Berater, Leiter(in), Informatiker, Verantwortliche, unbefristet/ befristet added — words that weren't on the list at all Verified: 72 titles in jobs.csv history now correctly flag as non-English that didn't before, all genuinely German on inspection; re-ran the broad-heuristic audit after the fix and every remaining candidate was a false positive from the heuristic itself (ambiguous 'Team', an English/German loanword).
This commit is contained in:
@@ -75,21 +75,38 @@ export const NON_ENGLISH_RE = new RegExp(
|
||||
String.raw`\s\(a\)(\s*[\d%]|\s*$)`,
|
||||
// German/Swiss umlauts and ß
|
||||
String.raw`[äöüÄÖÜß]`,
|
||||
// /in word-ending (e.g. Mitarbeiter/in) — excludes m/f, n/a, etc. Also
|
||||
// catches the gender-colon variant (Entwickler:in) — ZHAW's "Senior Full
|
||||
// Stack Entwickler:in 80-100 %" (via job-room.ch) used ":in" and slipped
|
||||
// past both this rule (slash-only) and the word list ("Entwickler" wasn't
|
||||
// on it either).
|
||||
String.raw`[a-zäöü][\/:]in\b`,
|
||||
// Gender-neutral suffix, in all four notations Swiss postings use
|
||||
// (Mitarbeiter/in, Entwickler:in, Engineer*in, Mitarbeiter_in) — excludes
|
||||
// m/f, n/a, etc. Also covers the adjectival-noun form, which appends "r"
|
||||
// instead of "in" (Verantwortliche:r, Angestellte:r, Studierende:r) since
|
||||
// the base word already ends in "-e". ZHAW's "Senior Full Stack
|
||||
// Entwickler:in 80-100 %" used ":in" and slipped past both this rule
|
||||
// (slash-only, at the time) and the word list ("Entwickler" wasn't on it
|
||||
// either); the audit below found the *in/_in/:r variants were equally
|
||||
// unhandled (OT-System Engineer*in, Applikationsverantwortliche:r).
|
||||
String.raw`[a-zäöü][\/:*_](in|r)\b`,
|
||||
// German-only words (case-insensitive handled by flag)
|
||||
String.raw`\b(Deine|Mitarbeiter|Pflegehelfer|Köchin|Koch EFZ|Kellner|Sozialpädagog|Lehrperson|Produktionsmitarbeiter|Produktionsplaner|Geschäftsführer|Spontanbewerbung|Arztsekretär|Physiotherapeut|Empfangsmitarbeiter|Automobilmechatroniker|Verkäufer|Projektleiter|Crewmitarbeiter|Crewmember|Systemgastronomie|Studienteilnehmende|Teilnehmende|Bauingenieur|Tiefbau|Hochbau|Elektroingenieur|Maschinenbau|gesucht|EFZ|Schweiz|Filiale|Ingenieur|Spezialist|Perspektive|Stellvertretung|Leitung|Beratung|Entwicklung|Betreuung|Verwaltung)\b`,
|
||||
// German prepositions/articles in title context (must appear as whole words) —
|
||||
// none of these exist as English words, so they're safe unambiguous markers.
|
||||
// Added after "Software Engineer auf dem Weg zum Data / AI Engineer" (ti&m,
|
||||
// via job-room.ch) sailed through: none of the German-word list above
|
||||
// matched, and the job-room description was too short (~1 hit) to clear the
|
||||
// separate ≥6 DE/FR word-count threshold either.
|
||||
String.raw`\bmit\b|\bauf\b|\bzum\b|\bzur\b|\bvom\b|\bfür\b|\bbeim\b`,
|
||||
String.raw`\b(Deine|Mitarbeiter|Pflegehelfer|Köchin|Koch EFZ|Kellner|Sozialpädagog|Lehrperson|Produktionsmitarbeiter|Produktionsplaner|Geschäftsführer|Spontanbewerbung|Arztsekretär|Physiotherapeut|Empfangsmitarbeiter|Automobilmechatroniker|Verkäufer|Projektleiter|Crewmitarbeiter|Crewmember|Systemgastronomie|Studienteilnehmende|Teilnehmende|Bauingenieur|Tiefbau|Hochbau|Elektroingenieur|Maschinenbau|gesucht|EFZ|Schweiz|Filiale|Ingenieur|Spezialist|Perspektive|Stellvertretung|Leitung|Beratung|Entwicklung|Betreuung|Verwaltung|unbefristet|befristet)\b`,
|
||||
// German occupational suffixes — matched with NO leading \b, deliberately,
|
||||
// because German compounds glue words together with no separator
|
||||
// (Aussendienstmitarbeiter, Vertriebsingenieur, Applikationsverantwortliche)
|
||||
// and a leading \b would only ever match the bare word. Found by auditing
|
||||
// jobs.csv for titles containing these strings that the rest of the regex
|
||||
// still let through: "Software Berater Fokus Lohn/HR in Bern",
|
||||
// "Leiter Business Applikationen, CRM & Analytics", "Leiterin Trading
|
||||
// Data, Integration & Reporting", "ICT System Engineer / Informatiker
|
||||
// Systemtechnik". All are German-only strings with no English overlap.
|
||||
String.raw`(mitarbeiter|ingenieur|berater|leiterin|leiter|informatiker|verantwortliche)\b`,
|
||||
// German prepositions/articles/conjunctions in title context (must appear
|
||||
// as whole words) — none of these exist as English words, so they're safe
|
||||
// unambiguous markers. Added after "Software Engineer auf dem Weg zum
|
||||
// Data / AI Engineer" (ti&m, via job-room.ch) sailed through: none of the
|
||||
// German-word list above matched, and the job-room description was too
|
||||
// short (~1 hit) to clear the separate ≥6 DE/FR word-count threshold
|
||||
// either. "und"/"oder" added after the same audit found "System Engineer
|
||||
// Microsoft und OnPrem Infrastruktur" slipping through — even the most
|
||||
// basic German conjunctions weren't covered.
|
||||
String.raw`\bmit\b|\bauf\b|\bzum\b|\bzur\b|\bvom\b|\bfür\b|\bbeim\b|\bund\b|\boder\b`,
|
||||
// French gender markers
|
||||
String.raw`\(H\/F\)|\(F\/H\)|\(h\/f\)|\(f\/h\)`,
|
||||
// French-only accented characters
|
||||
|
||||
Reference in New Issue
Block a user