fix: systematic sweep of NON_ENGLISH_RE gaps in gender/compound notation
Audited all 2949 titles in jobs.csv against broader German-signal heuristics than the regex itself uses, to find what it still misses: - gender suffix generalized beyond slash/colon to also catch */_ and the adjectival -r form (Engineer*in, Mitarbeiter_in, Verantwortliche:r) - 'und'/'oder' were missing entirely — basic conjunctions, zero English overlap - occupational nouns (Mitarbeiter, Ingenieur, Berater, Leiter, Informatiker, Verantwortliche) now also match as compound suffixes (Aussendienstmitarbeiter, Vertriebsingenieur) instead of only standalone, since German compounds have no separator - Berater, Leiter(in), Informatiker, Verantwortliche, unbefristet/ befristet added — words that weren't on the list at all Verified: 72 titles in jobs.csv history now correctly flag as non-English that didn't before, all genuinely German on inspection; re-ran the broad-heuristic audit after the fix and every remaining candidate was a false positive from the heuristic itself (ambiguous 'Team', an English/German loanword).
This commit is contained in:
@@ -75,21 +75,38 @@ export const NON_ENGLISH_RE = new RegExp(
|
|||||||
String.raw`\s\(a\)(\s*[\d%]|\s*$)`,
|
String.raw`\s\(a\)(\s*[\d%]|\s*$)`,
|
||||||
// German/Swiss umlauts and ß
|
// German/Swiss umlauts and ß
|
||||||
String.raw`[äöüÄÖÜß]`,
|
String.raw`[äöüÄÖÜß]`,
|
||||||
// /in word-ending (e.g. Mitarbeiter/in) — excludes m/f, n/a, etc. Also
|
// Gender-neutral suffix, in all four notations Swiss postings use
|
||||||
// catches the gender-colon variant (Entwickler:in) — ZHAW's "Senior Full
|
// (Mitarbeiter/in, Entwickler:in, Engineer*in, Mitarbeiter_in) — excludes
|
||||||
// Stack Entwickler:in 80-100 %" (via job-room.ch) used ":in" and slipped
|
// m/f, n/a, etc. Also covers the adjectival-noun form, which appends "r"
|
||||||
// past both this rule (slash-only) and the word list ("Entwickler" wasn't
|
// instead of "in" (Verantwortliche:r, Angestellte:r, Studierende:r) since
|
||||||
// on it either).
|
// the base word already ends in "-e". ZHAW's "Senior Full Stack
|
||||||
String.raw`[a-zäöü][\/:]in\b`,
|
// Entwickler:in 80-100 %" used ":in" and slipped past both this rule
|
||||||
|
// (slash-only, at the time) and the word list ("Entwickler" wasn't on it
|
||||||
|
// either); the audit below found the *in/_in/:r variants were equally
|
||||||
|
// unhandled (OT-System Engineer*in, Applikationsverantwortliche:r).
|
||||||
|
String.raw`[a-zäöü][\/:*_](in|r)\b`,
|
||||||
// German-only words (case-insensitive handled by flag)
|
// German-only words (case-insensitive handled by flag)
|
||||||
String.raw`\b(Deine|Mitarbeiter|Pflegehelfer|Köchin|Koch EFZ|Kellner|Sozialpädagog|Lehrperson|Produktionsmitarbeiter|Produktionsplaner|Geschäftsführer|Spontanbewerbung|Arztsekretär|Physiotherapeut|Empfangsmitarbeiter|Automobilmechatroniker|Verkäufer|Projektleiter|Crewmitarbeiter|Crewmember|Systemgastronomie|Studienteilnehmende|Teilnehmende|Bauingenieur|Tiefbau|Hochbau|Elektroingenieur|Maschinenbau|gesucht|EFZ|Schweiz|Filiale|Ingenieur|Spezialist|Perspektive|Stellvertretung|Leitung|Beratung|Entwicklung|Betreuung|Verwaltung)\b`,
|
String.raw`\b(Deine|Mitarbeiter|Pflegehelfer|Köchin|Koch EFZ|Kellner|Sozialpädagog|Lehrperson|Produktionsmitarbeiter|Produktionsplaner|Geschäftsführer|Spontanbewerbung|Arztsekretär|Physiotherapeut|Empfangsmitarbeiter|Automobilmechatroniker|Verkäufer|Projektleiter|Crewmitarbeiter|Crewmember|Systemgastronomie|Studienteilnehmende|Teilnehmende|Bauingenieur|Tiefbau|Hochbau|Elektroingenieur|Maschinenbau|gesucht|EFZ|Schweiz|Filiale|Ingenieur|Spezialist|Perspektive|Stellvertretung|Leitung|Beratung|Entwicklung|Betreuung|Verwaltung|unbefristet|befristet)\b`,
|
||||||
// German prepositions/articles in title context (must appear as whole words) —
|
// German occupational suffixes — matched with NO leading \b, deliberately,
|
||||||
// none of these exist as English words, so they're safe unambiguous markers.
|
// because German compounds glue words together with no separator
|
||||||
// Added after "Software Engineer auf dem Weg zum Data / AI Engineer" (ti&m,
|
// (Aussendienstmitarbeiter, Vertriebsingenieur, Applikationsverantwortliche)
|
||||||
// via job-room.ch) sailed through: none of the German-word list above
|
// and a leading \b would only ever match the bare word. Found by auditing
|
||||||
// matched, and the job-room description was too short (~1 hit) to clear the
|
// jobs.csv for titles containing these strings that the rest of the regex
|
||||||
// separate ≥6 DE/FR word-count threshold either.
|
// still let through: "Software Berater Fokus Lohn/HR in Bern",
|
||||||
String.raw`\bmit\b|\bauf\b|\bzum\b|\bzur\b|\bvom\b|\bfür\b|\bbeim\b`,
|
// "Leiter Business Applikationen, CRM & Analytics", "Leiterin Trading
|
||||||
|
// Data, Integration & Reporting", "ICT System Engineer / Informatiker
|
||||||
|
// Systemtechnik". All are German-only strings with no English overlap.
|
||||||
|
String.raw`(mitarbeiter|ingenieur|berater|leiterin|leiter|informatiker|verantwortliche)\b`,
|
||||||
|
// German prepositions/articles/conjunctions in title context (must appear
|
||||||
|
// as whole words) — none of these exist as English words, so they're safe
|
||||||
|
// unambiguous markers. Added after "Software Engineer auf dem Weg zum
|
||||||
|
// Data / AI Engineer" (ti&m, via job-room.ch) sailed through: none of the
|
||||||
|
// German-word list above matched, and the job-room description was too
|
||||||
|
// short (~1 hit) to clear the separate ≥6 DE/FR word-count threshold
|
||||||
|
// either. "und"/"oder" added after the same audit found "System Engineer
|
||||||
|
// Microsoft und OnPrem Infrastruktur" slipping through — even the most
|
||||||
|
// basic German conjunctions weren't covered.
|
||||||
|
String.raw`\bmit\b|\bauf\b|\bzum\b|\bzur\b|\bvom\b|\bfür\b|\bbeim\b|\bund\b|\boder\b`,
|
||||||
// French gender markers
|
// French gender markers
|
||||||
String.raw`\(H\/F\)|\(F\/H\)|\(h\/f\)|\(f\/h\)`,
|
String.raw`\(H\/F\)|\(F\/H\)|\(h\/f\)|\(f\/h\)`,
|
||||||
// French-only accented characters
|
// French-only accented characters
|
||||||
|
|||||||
Reference in New Issue
Block a user