[bug-gettext] [PATCH] msgfilter: Add --newline option

[Daiki Ueno]

The filter program was supposed to handle input without ending
with a newline character.  This was causing portability problems
with standard text processing programs on some platforms (BSD Sed,
for instance) and not friendly towards POSIX, where a "text file"
is required to have an ending newline.  The --newline option
controls the behavior.  If it is given, both filter input and
output are assumed to end with a newline character.
* gettext-tools/src/msgfilter.c (newline): New variable.
(long_options): Add --newline option.
(main): Handle --newline option.
(usage): Document --newline option.
(process_string_with_newline): New function which wraps
process_string.
(process_message): Use process_string_with_newline instead of
process_string if --newline is specified.

* gettext-tools/tests/msgfilter-8: New file.
* gettext-tools/tests/Makefile.am (TESTS): Add new test.

* gettext-tools/doc/msgfilter.texi: Document --newline option.
---
 gettext-tools/doc/ChangeLog      |   4 ++
 gettext-tools/doc/msgfilter.texi |  23 +++++---
 gettext-tools/src/ChangeLog      |  19 +++++++
 gettext-tools/src/msgfilter.c    |  48 ++++++++++++++++-
 gettext-tools/tests/ChangeLog    |   5 ++
 gettext-tools/tests/Makefile.am  |   2 +-
 gettext-tools/tests/msgfilter-8  | 114 +++++++++++++++++++++++++++++++++++++++
 7 files changed, 206 insertions(+), 9 deletions(-)
 create mode 100644 gettext-tools/tests/msgfilter-8

diff --git a/gettext-tools/doc/ChangeLog b/gettext-tools/doc/ChangeLog
index 109505b..5f618a0 100644
--- a/gettext-tools/doc/ChangeLog
+++ b/gettext-tools/doc/ChangeLog
@@ -1,3 +1,7 @@
+2015-01-28  Daiki Ueno  <address@hidden>
+
+       * msgfilter.texi: Document --newline option.
+
 2015-01-05  Daiki Ueno  <address@hidden>
 
        * gettext.texi (Vala): New section.
diff --git a/gettext-tools/doc/msgfilter.texi b/gettext-tools/doc/msgfilter.texi
index 1fae251..758f7a4 100644
--- a/gettext-tools/doc/msgfilter.texi
+++ b/gettext-tools/doc/msgfilter.texi
@@ -73,6 +73,14 @@ input and writes a modified translation to standard output.  
A frequently
 used filter is @samp{sed}.  A few particular built-in filters are also
 recognized.
 
address@hidden @samp
address@hidden --newline
address@hidden address@hidden, @code{msgfilter} option}
+Add newline at the end of each input line and also strip the ending
+newline from the output line.
+
address@hidden table
+
 @cindex @code{msgfilter} filter and catalog encoding
 Note: If the filter is not a built-in filter, you have to care about encodings:
 It is your responsibility to ensure that the @var{filter} can cope
@@ -86,13 +94,14 @@ you can first convert the translation catalog to UTF-8 
using the
 locale, by using the @code{LC_ALL} environment variable.
 
 @cindex portability problems with @code{sed}
-Note: Most translations in a translation catalog don't end with a newline
-character.  For this reason, it is important that the @var{filter}
-recognizes its last input line even if it ends without a newline, and that
-it doesn't add an undesired trailing newline at the end.  The @samp{sed}
-program on some platforms is known to ignore the last line of input if it
-is not terminated with a newline.  You can use GNU @code{sed} instead; it
-does not have this limitation.
+Note: Most translations in a translation catalog don't end with a
+newline character.  For this reason, unless the @code{--newline}
+option is used, it is important that the @var{filter} recognizes its
+last input line even if it ends without a newline, and that it doesn't
+add an undesired trailing newline at the end.  The @samp{sed} program on
+some platforms is known to ignore the last line of input if it is not
+terminated with a newline.  You can use GNU @code{sed} instead; it does
+not have this limitation.
 
 @subsection Useful @var{filter-option}s when the @var{filter} is @samp{sed}
 
diff --git a/gettext-tools/src/ChangeLog b/gettext-tools/src/ChangeLog
index af08eda..bc48fad 100644
--- a/gettext-tools/src/ChangeLog
+++ b/gettext-tools/src/ChangeLog
@@ -1,3 +1,22 @@
+2015-01-28  Daiki Ueno  <address@hidden>
+
+       msgfilter: Add --newline option
+       The filter program was supposed to handle input without ending
+       with a newline character.  This was causing portability problems
+       with standard text processing programs on some platforms (BSD Sed,
+       for instance) and not friendly towards POSIX, where a "text file"
+       is required to have an ending newline.  The --newline option
+       controls the behavior.  If it is given, both filter input and
+       output are assumed to end with a newline character.
+       * msgfilter.c (newline): New variable.
+       (long_options): Add --newline option.
+       (main): Handle --newline option.
+       (usage): Document --newline option.
+       (process_string_with_newline): New function which wraps
+       process_string.
+       (process_message): Use process_string_with_newline instead of
+       process_string if --newline is specified.
+
 2015-01-24  Daiki Ueno  <address@hidden>
 
        xgettext, msgmerge: Avoid undefined non-null argument behavior
diff --git a/gettext-tools/src/msgfilter.c b/gettext-tools/src/msgfilter.c
index f9cb1cc..43fdc52 100644
--- a/gettext-tools/src/msgfilter.c
+++ b/gettext-tools/src/msgfilter.c
@@ -81,6 +81,8 @@ static const char *sub_path;
 static const char **sub_argv;
 static int sub_argc;
 
+static bool newline;
+
 /* Filter function.  */
 static void (*filter) (const char *str, size_t len, char **resultp, size_t 
*lengthp);
 
@@ -88,6 +90,7 @@ static void (*filter) (const char *str, size_t len, char 
**resultp, size_t *leng
 static const struct option long_options[] =
 {
   { "add-location", optional_argument, NULL, 'n' },
+  { "newline", no_argument, NULL, CHAR_MAX + 9 },
   { "color", optional_argument, NULL, CHAR_MAX + 6 },
   { "directory", required_argument, NULL, 'D' },
   { "escape", no_argument, NULL, 'E' },
@@ -269,6 +272,10 @@ main (int argc, char **argv)
         message_print_style_filepos (filepos_comment_none);
         break;
 
+      case CHAR_MAX + 9: /* --newline */
+        newline = true;
+        break;
+
       default:
         usage (EXIT_FAILURE);
         break;
@@ -438,6 +445,12 @@ and writes a modified translation to standard output.\n\
 "));
       printf ("\n");
       printf (_("\
+Filter input and output:\n"));
+      printf (_("\
+  --newline                   add newline at the end of each translation and\n\
+                                expect the output ends with a newline"));
+      printf ("\n");
+      printf (_("\
 Useful FILTER-OPTIONs when the FILTER is 'sed':\n"));
       printf (_("\
   -e, --expression=SCRIPT     add SCRIPT to the commands to be executed\n"));
@@ -628,6 +641,33 @@ process_string (const char *str, size_t len, char 
**resultp, size_t *lengthp)
 }
 
 
+/* Do the same thing as process_string but appends a newline to STR
+   before processing, and remove a newline from the result.
+ */
+static void
+process_string_with_newline (const char *str, size_t len, char **resultp,
+                             size_t *lengthp)
+{
+  char *newstr;
+  char *result;
+  size_t length;
+
+  newstr = XNMALLOC (len + 1, char);
+  memcpy (newstr, str, len);
+  newstr[len] = '\n';
+
+  process_string (newstr, len + 1, &result, &length);
+
+  free (newstr);
+
+  if (length > 0 && result[length - 1] == '\n')
+    result[--length] = '\0';
+
+  *resultp = result;
+  *lengthp = length;
+}
+
+
 static void
 process_message (message_ty *mp)
 {
@@ -693,6 +733,7 @@ process_message (message_ty *mp)
   substrings = XNMALLOC (nsubstrings, char *);
   for (p = msgstr, k = 0, total_len = 0; k < nsubstrings; k++)
     {
+      char *string;
       char *result;
       size_t length;
 
@@ -705,7 +746,12 @@ process_message (message_ty *mp)
         }
       else
         unsetenv ("MSGFILTER_PLURAL_FORM");
-      process_string (p, strlen (p), &result, &length);
+
+      if (newline)
+        process_string_with_newline (p, strlen (p), &result, &length);
+      else
+        process_string (p, strlen (p), &result, &length);
+
       result = (char *) xrealloc (result, length + 1);
       result[length] = '\0';
       substrings[k] = result;
diff --git a/gettext-tools/tests/ChangeLog b/gettext-tools/tests/ChangeLog
index a08f91d..c60f35a 100644
--- a/gettext-tools/tests/ChangeLog
+++ b/gettext-tools/tests/ChangeLog
@@ -1,3 +1,8 @@
+2015-01-28  Daiki Ueno  <address@hidden>
+
+       * msgfilter-8: New file.
+       * Makefile.am (TESTS): Add new test.
+
 2015-01-16  Daiki Ueno  <address@hidden>
 
        * xgettext-desktop-1: Check "invalid non-blank character" warning.
diff --git a/gettext-tools/tests/Makefile.am b/gettext-tools/tests/Makefile.am
index 5a0d3c0..c098a6f 100644
--- a/gettext-tools/tests/Makefile.am
+++ b/gettext-tools/tests/Makefile.am
@@ -42,7 +42,7 @@ TESTS = gettext-1 gettext-2 gettext-3 gettext-4 gettext-5 
gettext-6 gettext-7 \
        msgen-1 msgen-2 msgen-3 msgen-4 \
        msgexec-1 msgexec-2 msgexec-3 msgexec-4 msgexec-5 \
        msgfilter-1 msgfilter-2 msgfilter-3 msgfilter-4 msgfilter-5 \
-       msgfilter-6 msgfilter-7 \
+       msgfilter-6 msgfilter-7 msgfilter-8 \
        msgfilter-sr-latin-1 msgfilter-quote-1 \
        msgfmt-1 msgfmt-2 msgfmt-3 msgfmt-4 msgfmt-5 msgfmt-6 msgfmt-7 \
        msgfmt-8 msgfmt-9 msgfmt-10 msgfmt-11 msgfmt-12 msgfmt-13 msgfmt-14 \
diff --git a/gettext-tools/tests/msgfilter-8 b/gettext-tools/tests/msgfilter-8
new file mode 100644
index 0000000..2fec6ef
--- /dev/null
+++ b/gettext-tools/tests/msgfilter-8
@@ -0,0 +1,114 @@
+#! /bin/sh
+. "${srcdir=.}/init.sh"; path_prepend_ . ../src
+
+# Test -add-newline option.
+
+cat <<\EOF > mfi-test8.po
+# HEADER.
+#
+msgid ""
+msgstr ""
+"Project-Id-Version: Bonnie Tyler\n"
+"Content-Type: text/plain; charset=ISO-8859-1\n"
+"Content-Transfer-Encoding: 8bit\n"
+
+#: married-men:4
+#, fuzzy
+msgid "The world is full of married men"
+msgstr "So viele verheiratete M�nner"
+
+#: married-men:5
+msgid "with wives who never understand"
+msgstr "und ihre Frauen verstehen sie nicht"
+
+#: married-men:6
+msgid "They're looking for someone to share"
+msgstr ""
+
+# schwer zu �bersetzen...
+#: married-men:7
+msgid "the excitement of a love affair"
+msgstr ""
+
+#: married-men:8
+msgid "Just as soon as they find you"
+msgstr ""
+
+#: married-men:9
+msgid "They warn you and darn you"
+msgstr ""
+
+#~ msgid "You fly on the wings of romance"
+#~ msgstr "Die Fl�gel der frischen Liebe heben dich zum Himmel"
+
+#, fuzzy
+#~ msgid "In the eyes of the world"
+#~ msgstr "F�r die anderen"
+
+# Etwas freie �bersetzung.
+#~ msgid "You're just another crazy girl"
+#~ msgstr "bist du blo� ein verr�cktes dummes Ding"
+
+#~ msgid "Who loves a married man"
+#~ msgstr "das einen verheirateten Mann liebt"
+EOF
+
+: ${MSGFILTER=msgfilter}
+LC_ALL=C ${MSGFILTER} --newline -i mfi-test8.po -o mfi-test8.out \
+      echo testing >mfi-test8.err 2>&1
+result=$?
+cat mfi-test8.err | grep -v 'warning: Locale charset' | grep -v '^ '
+test $result = 0 || { exit 1; }
+
+cat <<\EOF > mfi-test8.ok
+# HEADER.
+#
+msgid ""
+msgstr "testing"
+
+#: married-men:4
+#, fuzzy
+msgid "The world is full of married men"
+msgstr "testing"
+
+#: married-men:5
+msgid "with wives who never understand"
+msgstr "testing"
+
+#: married-men:6
+msgid "They're looking for someone to share"
+msgstr "testing"
+
+# schwer zu �bersetzen...
+#: married-men:7
+msgid "the excitement of a love affair"
+msgstr "testing"
+
+#: married-men:8
+msgid "Just as soon as they find you"
+msgstr "testing"
+
+#: married-men:9
+msgid "They warn you and darn you"
+msgstr "testing"
+
+#~ msgid "You fly on the wings of romance"
+#~ msgstr "testing"
+
+#, fuzzy
+#~ msgid "In the eyes of the world"
+#~ msgstr "testing"
+
+# Etwas freie �bersetzung.
+#~ msgid "You're just another crazy girl"
+#~ msgstr "testing"
+
+#~ msgid "Who loves a married man"
+#~ msgstr "testing"
+EOF
+
+: ${DIFF=diff}
+${DIFF} mfi-test8.ok mfi-test8.out
+result=$?
+
+exit $result
-- 
2.1.0