[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Huge file adventure (+patch)
From: |
Dmitry Antipov |
Subject: |
Huge file adventure (+patch) |
Date: |
Mon, 07 Oct 2013 13:47:26 +0400 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.0 |
Recently I have got installed 16GB RAM in my laptop and, of course,
have tried to open 8GB ASCII text file to see what happens (now it's
just 1/2 of RAM, so why not?). After opening, I did some editing,
then wait for auto-save and ... got "Memory exhausted" message. Short
investigation quickly shows that I need ... 24GB of RAM to handle
such a file :-(. Here is why:
1) Fdo_auto_save calls Fwrite_region for the whole file, which
issues e_write (.., start=1, end=8G, ...).
2) CODING_REQUIRE_ENCODING decides that it's time to do some
encoding, and encode_coding_object allocates 8G destination
buffer (from coding.c):
8335 else if (EQ (dst_object, Qt))
8336 {
8337 ptrdiff_t dst_bytes = max (1, coding->src_chars);
8338 coding->dst_object = Qnil;
8339 coding->destination = xmalloc (dst_bytes); /* HERE */
8340 coding->dst_bytes = dst_bytes;
8341 coding->dst_multibyte = 0;
8342 }
3) Finally encode_coding_object tries to create 8G Lisp string
to hold the result (from coding.c):
8351 if (EQ (dst_object, Qt))
8352 {
8353 if (BUFFERP (coding->dst_object))
8354 coding->dst_object = Fbuffer_string ();
8355 else
8356 {
8357 coding->dst_object
8358 = make_unibyte_string ((char *) coding->destination,
8359 coding->produced); /* HERE */
8360 xfree (coding->destination);
8361 }
8362 }
So 8G for buffer text + 8G for encoding buffer + 8G for the result.
Since `coding->destination' is freed immediately after creating Lisp
string, I need 16G for some period of time but with short 24G peak.
And, of course, there is a patch to address an issues described above.
Comments are very welcome because I'm not hooked too much in coding
machinery (yet).
Dmitry
e_write_and_encode.patch
Description: Text Data
- Huge file adventure (+patch),
Dmitry Antipov <=