Re: lynx-dev [PATCH 2.8.5-dev14] *Really* large tables

[Leonid Pauzner]

7-Apr-2003 22:40 Ilya Zakharevich wrote:
> over 56Kb/sec line without any slowdown).  I tried several different possible
> modifications, but could not improve the footprint.  You may want to
> experiment
> with CELLS_GROWBY_FACTOR, ROWS_GROWBY_DIVISOR and REUSE_ROWS_AS_CELLS_POOLS if
> you are interested.


I think a main footprint is not due to (re)allocation of memory
but copying it from one place to another.
Just a guess. Tables have two kind of memory:

"array of cells + something" (==row), and
"array of pointers to rows"

Rows can be of different size (usually <<80), we need not realloc rows:
we can work nicely with a pair (array of cells, num of cells in this array).
And cell is a fixed-size record.

"Array of pointers to rows" can be of arbitrary size, 500000 entries in your
example. Instead of reallocating this huge array we can add another level of
indirection: split this array into chunks (pools) of a fixed size, and
maintain a small array of pointers to that chunks (VM usually does this:).

Everything except the small array may utilize HTPool
(say, HText->Stbl->pool, a parse-time temporary pool).

Does this naive picture have something with the reality?


>  [Note that #undef'ing SAVE_TIME_NOT_SPACE would lead to *larger* memory
>   consumption (due to realloc()s most probably not able to reallocate in
>   place); I think this #ifdef should be removed.]

> Enjoy,
> Ilya

> P.S.  Note that there is the third place where addmem_rowinfo() should be
> used:
>       in Stbl_fakeFinishCellInTable().  But I could not quickly follow the
>       logic of this function (I think the logic is mine :-[).  This means
>       that tables with <br> and <p> in cells may still show quadratic
>       behaviour.

> P.P.S. I tried the above URL, but it is not shown with partial-display-while-
>        downloading.  I have zlib used by lynx; why it would not gunzip on the
>        fly?  (It would download the file, then would open the copy.)

Try with -trace (just few first pages, stop with `z' key).


> --- ./src/TRSTable.c-ini      Sat Jul  7 17:30:12 2001
> +++ ./src/TRSTable.c  Mon Apr  7 22:12:46 2003
> @@ -17,11 +17,17 @@
>  #ifdef SAVE_TIME_NOT_SPACE
>  #define CELLS_GROWBY 16
>  #define ROWS_GROWBY 16
> -#else
> +#define ROWS_GROWBY_DIVISOR 2
> +#else  /* This is very silly, and leads to *larger* memory consumption... */
>  #define CELLS_GROWBY 2
>  #define ROWS_GROWBY 2
> +#define ROWS_GROWBY_DIVISOR 10
>  #endif

> +/* Experiments show that 2 is better than 1.5 is better than 1.25 (all by
> +   a small margin only)??? */
> +#define CELLS_GROWBY_FACTOR 2
> +



; To UNSUBSCRIBE: Send "unsubscribe lynx-dev" to address@hidden