Flags, Lines, and Unicode: Where Correct Patterns Go Wrong

A pattern that passes every unit test can still break the first week in production. The input arrives as a Windows-exported CSV, or a username contains an emoji, or a "digits only" field accepts Arabic-Indic numerals. None of these are exotic. They're Tuesday. This lesson is the checklist I run before I trust a regex with real-world text.

Flags that change meaning

• i: case-insensitive (re.I)
• m: ^ and $ match at line breaks, not just at input edges (re.M)
• s: . matches newlines too (re.S, "dotall")
• x: verbose mode with whitespace and comments (re.X; not in JS)
• u / v (JS): Unicode code-point mode, \p{...} property escapes
• g / y (JS): global and sticky; both make the regex object stateful
• re.A (Python): restrict \w, \d, \s to ASCII

$ and the newline you forgot

In JavaScript without m, $ means end of input. In Python, PCRE, and Java, $ also matches just before a trailing newline:

import re
re.match(r"^[a-z0-9]+$", "admin\n")     # matches. The newline sneaks through validation
re.fullmatch(r"[a-z0-9]+", "admin\n")   # None
re.match(r"^[a-z0-9]+\Z", "admin\n")    # None. \Z is Python's absolute end (\z in PCRE/Java/Ruby)

Ruby is worse: ^ and $ are always line anchors, so /^https?:\/\// happily validates "javascript:alert(1)\nhttp://x". Rails refuses ^/$ in format validators for exactly this reason. Use \A and \z.

CRLF: the invisible \r

const csv = "id,name\r\n1,alice\r\n2,bob\r\n";
console.log(csv.split("\n").map(l => /^\d+,\w+$/.test(l)));    // [false, false, false, false]
console.log(csv.split(/\r?\n/).map(l => /^\d+,\w+$/.test(l))); // [false, true, true, false]

Every line from Excel or a Windows service ends in \r\n. Split on \n and a trailing \r breaks every $. It's invisible in most terminals. cat -A shows it as ^M.

Code units versus code points

console.log(/^.$/.test("😀"));             // false: UTF-16 sees two code units
console.log(/^.$/u.test("😀"));            // true
console.log("café".match(/\w+/)[0]);        // "caf": \w is ASCII-only in JS
console.log("café".match(/\p{L}+/u)[0]);    // "café"
console.log("cafe\u0301".match(/\p{L}+/u)[0]);  // "cafe": combining accent is \p{M}, not \p{L}

Normalize first (s.normalize("NFC")) and then match. Python goes the other way: \d and \w are Unicode-aware by default for str, so re.fullmatch(r"\d+", "١٢٣") succeeds. That's fine for prose and wrong for account numbers. Use [0-9] or re.ASCII for identifiers.

The g flag makes a regex stateful

const re = /error/g;
console.log(re.test("error: disk full"));   // true  (lastIndex is now 5)
console.log(re.test("error: oom killed"));  // false (search started at index 5)

A module-level /.../g used with test() alternates true and false across calls. Drop g for boolean checks.

Field notes

• In Python, validate with re.fullmatch. Never with ^...$.
• [\s\S] was the pre-s-flag idiom for "any character including newline." You'll still see it in old code.
• LC_ALL=C makes grep and sed work on bytes: faster, but . no longer matches a multi-byte character as one unit.
• Case-insensitive matching uses simple case folding. Turkish ı/İ and German ß won't behave the way your users expect.