Unicode Anomalies
Scans for invisible and unusual Unicode characters that indicate AI generation, copy-paste from large language model (LLM) interfaces, or adversarial text manipulation. Detects homoglyph substitutions, zero-width characters, bidirectional overrides, and Unicode normalisation issues.
Technical description
E1 performs a per-character scan of the entire document across 21 detection categories. It detects AI-characteristic typographic choices (em-dashes, smart quotes, Unicode ellipsis), invisible characters used in prompt injection and text smuggling (zero-width spaces, bidirectional overrides, soft hyphens, tag characters), and homoglyph substitutions where Cyrillic or Greek letters masquerade as Latin characters. The indicator runs at Layer 1 with no external dependencies beyond the Python standard library.
How it works
1. Em-dash / En-dash ratio. Counts em-dashes (U+2014), en-dashes (U+2013), and normal hyphens (U+002D). A text with exclusively em/en-dashes and no normal hyphens is a strong LLM signal (+1.5). A ratio above 0.5 (dashes > half of normal hyphens) contributes +1.0.
2. Smart quotes vs straight quotes. Detects smart/curly quotes (U+201C, U+201D, U+2018, U+2019) and straight quotes (U+0022, U+0027). Text with smart quotes only and no straight quotes contributes +1.0. LLMs systematically use smart quotes.
3-4. Non-breaking and thin spaces. Non-breaking spaces (U+00A0) above 5 occurrences (+0.5). Thin spaces (U+2009) at any occurrence (+0.5 each).
5. Zero-width characters. Zero-width space (U+200B), non-joiner (U+200C), joiner (U+200D), and word joiner (U+2060). Each occurrence is critical (+1.0 each, severity: error). Used in prompt injection attacks and AI copy-paste artifacts.
6. Soft hyphens. U+00AD in mid-text positions (+0.3 each).
7. Directional marks. left-to-right (LTR) mark (U+200E) and right-to-left (RTL) mark (U+200F) (+0.3 each).
8. Bidirectional override characters. Left-to-Right Embedding (LRE), Right-to-Left Embedding (RLE), Pop Directional Formatting (PDF), Left-to-Right Override (LRO), Right-to-Left Override (RLO), Left-to-Right Isolate (LRI), Right-to-Left Isolate (RLI), First Strong Isolate (FSI), Pop Directional Isolate (PDI) (U+202A-U+202E, U+2066-U+2069). Critical severity (+1.0 each). Used in Unicode smuggling attacks to manipulate text direction and bypass content filters (Sarabamoun, 2025; Microsoft apm, 2025).
9. Misplaced BOM. Byte order mark (U+FEFF) at any position other than index 0 (+0.5).
10. Exotic bullets. Unicode bullet characters (U+2022, U+2023, U+25CF, U+25CB) common in LLM-generated lists (+0.5).
11. Unicode ellipsis. U+2026 used instead of three American Standard Code for Information Interchange (ASCII) dots (+0.3).
12. Homoglyph detection. Cyrillic and Greek letters visually identical to Latin characters (Cyrillic а→a, е→e, о→o, р→p, с→c, х→x; Greek Α→A, Β→B, Ε→E, Η→H, Ι→I, Κ→K, Μ→M, Ν→N, Ο→O, Ρ→P, Τ→T, Υ→Y, Χ→X, etc.). Each detected homoglyph contributes +0.3, capped at +2.0 total. Severity: error. This is a primary vector for bypassing content filters and AI detection (Sarabamoun, 2025; Parikh PromptSonar, 2025).
13. Unicode tag characters. U+E0000-U+E007F range. Used in steganography and prompt file supply-chain attacks (+1.0, severity: error). Microsoft apm (2025) found 12% of sampled shared prompt files contained hidden tag characters.
14. Unicode normalisation check. Verifies text is in Normalization Form C (NFC). Text in decomposed form, Normalization Form D (NFD), indicates multi-tool assembly and can mask hidden characters (+0.5, severity: info).
15. Variation selectors. Variation selectors (U+FE00-U+FE0F) and the supplement range (U+E0100-U+E01EF). Each occurrence is critical (+1.5, severity: error). Appended invisibly to characters in imperceptible LLM jailbreaking attacks (Imperceptible Jailbreaking, 2025).
16. Invisible operators. Invisible times (U+2062), invisible separator (U+2063), invisible plus (U+2064) (+1.0 each, severity: error). Used in "Sneaky Bits" data-smuggling encodings (Embrace The Red, 2025).
17. Fullwidth characters. Fullwidth Latin forms (U+FF01-U+FF5E) used as confusable substitutes that evade keyword filters (+0.3 each, capped at +2.0).
18. Private Use Area. PUA codepoints (U+E000-U+F8FF) carry no standard meaning and can hide payloads (+0.3 each, capped at +1.5).
19. Line / paragraph separators. Unicode line separator (U+2028) and paragraph separator (U+2029) used instead of standard newlines (+0.3 each, severity: info).
20. Ideographic / narrow no-break space. Ideographic space (U+3000) and narrow no-break space (U+202F) (+0.3 each, severity: info). NNBSP is a recurring GPT-5/o3/o4 output artifact (2025); because it is also legitimate in French typography (before ; : ! ? and inside guillemets), it is scored conservatively to avoid false positives.
21. Hair space / em space. Hair space (U+200A) and em space (U+2003) are uncommon spacing characters typical of copy-paste or LLM output (+0.3 each, severity: info).
Score thresholds
| Score | Meaning |
|---|---|
| 0 to 1 | Clean ASCII or well-normalised Unicode. No invisible characters, no homoglyphs, normal typographic conventions. |
| 2 to 3 | Minor anomalies: some smart quotes, a Unicode ellipsis, moderate em-dash usage. Likely copy-paste from a word processor or LLM interface. |
| 4 to 5 | Critical anomalies present: zero-width characters, bidirectional overrides, homoglyph substitutions, variation selectors, or tag characters. Strong evidence of AI generation, adversarial manipulation, or multi-tool text assembly. |
Limitations
The homoglyph mapping covers the most common Cyrillic and Greek letters that visually match Latin but is not exhaustive. Mathematical alphanumeric symbols (U+1D400-U+1D7FF) and other script homoglyphs (Armenian, extended Greek) are not currently detected; fullwidth Latin forms (U+FF01-U+FF5E) are covered separately by category 17. The normalisation check only distinguishes NFC from non-NFC forms; it does not detect Normalization Form KC (NFKC) level issues.
Em-dash and smart-quote detection may produce false positives on text that was legitimately typeset with professional publishing software (e.g., LaTeX with proper Unicode output). The ratio thresholds are calibrated for typical academic manuscripts and may be overly sensitive for formally published articles.
References
- Sarabamoun E. Special-character adversarial attacks on open-source language models. arXiv preprint arXiv:2508.14070. 2025. https://arxiv.org/abs/2508.14070
- Parikh M. Detecting Unicode homoglyph and zero-width character evasion in LLM prompt injection attacks (PromptSonar). 2025.
- Microsoft apm. Content security scanning for prompt files: hidden Unicode detection. GitHub Issue #312. 2025. https://github.com/microsoft/apm/issues/312
- Imperceptible jailbreaking against large language models. arXiv preprint arXiv:2510.05025. 2025. https://arxiv.org/abs/2510.05025
- Embrace The Red. Sneaky bits: advanced data smuggling and ASCII smuggler updates. 2025. https://embracethered.com/blog/posts/2025/sneaky-bits-and-ascii-smuggler/