[6920] 8-bit input for XeTeX and LuaTeX

[Gavin D. Smith]

Revision: 6920
          http://svn.sv.gnu.org/viewvc/?view=rev&root=texinfo&revision=6920
Author:   gavin
Date:     2016-01-04 20:39:43 +0000 (Mon, 04 Jan 2016)
Log Message:
-----------
8-bit input for XeTeX and LuaTeX

Modified Paths:
--------------
    trunk/ChangeLog
    trunk/NEWS
    trunk/doc/texinfo.tex

Modified: trunk/ChangeLog
===================================================================
--- trunk/ChangeLog     2016-01-04 20:19:54 UTC (rev 6919)
+++ trunk/ChangeLog     2016-01-04 20:39:43 UTC (rev 6920)
@@ -1,5 +1,11 @@
 2016-01-04  Gavin Smith  <address@hidden>
 
+       * doc/texinfo.tex, NEWS: For XeTeX and LuaTeX, use input by 
+       bytes instead of by UTF-8 codepoints.  Documents encoded in 
+       UTF-8 reported broken by Werner Lemberg.
+
+2016-01-04  Gavin Smith  <address@hidden>
+
        * tp/Texinfo/Convert/HTML.pm (%css_map): Use 'font-family: initial'
        instead of 'font-family: serif' for @r.  Suggestion from Karl Berry.
 

Modified: trunk/NEWS
===================================================================
--- trunk/NEWS  2016-01-04 20:19:54 UTC (rev 6919)
+++ trunk/NEWS  2016-01-04 20:39:43 UTC (rev 6920)
@@ -68,6 +68,8 @@
     can extend slightly into the margin instead of being broken across 
     two lines, and text is split more evenly across lines.  Reduce 
     chance of an orphaned index entry appearing at the top of a column.
+  . Support character encodings beyond ASCII for XeTeX and LuaTeX by
+    reading file input byte-by-byte.
 
 * texi2dvi
   . Support of getting output files using the `-recorder' option to TeX,

Modified: trunk/doc/texinfo.tex
===================================================================
--- trunk/doc/texinfo.tex       2016-01-04 20:19:54 UTC (rev 6919)
+++ trunk/doc/texinfo.tex       2016-01-04 20:39:43 UTC (rev 6920)
@@ -3,11 +3,11 @@
 % Load plain if necessary, i.e., if running under initex.
 \expandafter\ifx\csname fmtname\endcsname\relax\input plain\fi
 %
-\def\texinfoversion{2015-12-20.12}
+\def\texinfoversion{2016-01-04.20}
 %
 % Copyright 1985, 1986, 1988, 1990, 1991, 1992, 1993, 1994, 1995,
 % 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006,
-% 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015
+% 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016
 % Free Software Foundation, Inc.
 %
 % This texinfo.tex file is free software: you can redistribute it and/or
@@ -9428,6 +9428,31 @@
   \global\righthyphenmin = #3\relax
 }
 
+% Get input by bytes instead of by UTF-8 codepoints for XeTeX and LuaTeX, 
+% otherwise the encoding support is completely broken.
+\ifx\XeTeXrevision\thisisundefined
+\else
+\XeTeXinputencoding "bytes"
+\fi
+
+\ifx\luatexversion\thisisundefined
+\else
+\directlua{
+local utf8_char, byte, gsub = unicode.utf8.char, string.byte, string.gsub
+
+local function convert_char (char)
+  return utf8_char(byte(char))
+end
+
+local function convert_line (line)
+  return gsub(line, ".", convert_char)
+end
+
+callback.register("process_input_buffer", convert_line)
+}
+\fi
+
+
 % Helpers for encodings.
 % Set the catcode of characters 128 through 255 to the specified number.
 %