[bug-gnu-libiconv] iconv tool buffers entire input file

[Yongzheng Wu]

Hi,

The iconv tool buffers the entire input file in memory. This causes a
problem with big files. This problem has been raised by others before.
http://serverfault.com/questions/50919/how-to-run-command-on-parts-of-input-file
and http://keithdevens.com/weblog/archive/2011/Sep/22/iconv. I found a
comment in iconv_prog.c, "Since we have to deal with arbitrary
encodings we must read the whole text in a buffer and process it in
one step."

My question is, can this be fixed? A trivial approach is to process
the input in fixed-size blocks. The remaining unprocessed bytes of
each block is feed to the next block. I think this approach works with
all prefix code encodings including UTF-8, UTF-16 and GBK.

Wu Yongzheng