qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Qemu-devel] [PATCH] checkpatch: detect doubly-encoded UTF-8


From: Paolo Bonzini
Subject: [Qemu-devel] [PATCH] checkpatch: detect doubly-encoded UTF-8
Date: Fri, 17 May 2019 15:19:00 +0200

Copy and pasting from Thunderbird's "view source" window results in double
encoding of multibyte UTF-8 sequences.  The appearance of those sequences is
very peculiar, so detect it and give an error despite the (low) possibility
of false positives.

As the major offender, I am also adding the same check to my applypatch-msg
and commit-msg hooks, but this will also cause patchew to croak loudly when
this mistake happens.

Signed-off-by: Paolo Bonzini <address@hidden>
---
 scripts/checkpatch.pl | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl
index 88682cb..b27e1de 100755
--- a/scripts/checkpatch.pl
+++ b/scripts/checkpatch.pl
@@ -262,6 +262,19 @@ our $UTF8  = qr{
        | $NON_ASCII_UTF8
 }x;
 
+# some readers default to ISO-8859-1 when showing email source. detect
+# when UTF-8 is incorrectly interpreted as ISO-8859-1 and reencoded back.
+# False positives are possible but very unlikely.
+our $UTF8_MOJIBAKE = qr{
+       \xC3[\x82-\x9F] \xC2[\x80-\xBF]                    # c2-df 80-bf
+       | \xC3\xA0 \xC2[\xA0-\xBF] \xC2[\x80-\xBF]         # e0 a0-bf 80-bf
+       | \xC3[\xA1-\xAC\xAE\xAF] (?: \xC2[\x80-\xBF]){2}  # e1-ec/ee/ef 80-bf 
80-bf
+       | \xC3\xAD \xC2[\x80-\x9F] \xC2[\x80-\xBF]         # ed 80-9f 80-bf
+       | \xC3\xB0 \xC2[\x90-\xBF] (?: \xC2[\x80-\xBF]){2} # f0 90-bf 80-bf 
80-bf
+       | \xC3[\xB1-\xB3] (?: \xC2[\x80-\xBF]){3}          # f1-f3 80-bf 80-bf 
80-bf
+       | \xC3\xB4 \xC2[\x80-\x8F] (?: \xC2[\x80-\xBF]){2} # f4 80-b8 80-bf 
80-bf
+}x;
+
 # There are still some false positives, but this catches most
 # common cases.
 our $typeTypedefs = qr{(?x:
@@ -1506,6 +1519,9 @@ sub process {
                        ERROR("Invalid UTF-8, patch and commit message should 
be encoded in UTF-8\n" . $hereptr);
                }
 
+               if ($rawline =~ m/$UTF8_MOJIBAKE/) {
+                       ERROR("Doubly-encoded UTF-8\n" . $herecurr);
+               }
 # Check if it's the start of a commit log
 # (not a header line and we haven't seen the patch filename)
                if ($in_header_lines && $realfile =~ /^$/ &&
-- 
1.8.3.1




reply via email to

[Prev in Thread] Current Thread [Next in Thread]