Re: [bug-gawk] Make awk data structure reside in memory

[Peng Yu]

Why the shared memory thing (ipcs, ipcrm) can not be done? It sounds like
the most relevant solution.

On Fri, Apr 12, 2019 at 12:51 PM david kerns <address@hidden>
wrote:

> your other options include:
> 1) creating a server that keeps running, thus the dataset in memory, and
> service requests as they come
> 2) using an existing awk extension (@load "rwarray") to quickly
> save/restore an array (the dataset)
> 3) write your own extension to solve it another way
> 4) or choose a different language (C/Python) more suitable for the task ;)
>
> On Fri, Apr 12, 2019 at 8:35 AM Peng Yu <address@hidden> wrote:
>
>> No. This means that the same data will be loaded multiple times, which is
>> not acceptable as the data is too large to fit multiple of them in  memory.
>> A solution must load the data once in memory.
>>
>> On Fri, Apr 12, 2019 at 10:08 AM david kerns <address@hidden>
>> wrote:
>>
>>> If you're running on Linux, (and you have the memory available) you
>>> might consider using /dev/shm (or another RAM disk)
>>> awk 'BEGIN{f="/dev/shm/fred";for(i=0;i<100000;i++)print i, rand() >
>>> f;close(f)}' # store your DB/dataset
>>> time awk 'BEGIN{f="/dev/shm/fred";while ((getline
>>> l<f)>0){split(l,a);db[a[1]]=a[2]};close(f)} /* other processing here */'
>>> re-reading from RAM disk should be significantly faster than normal disk
>>> (assuming you have the RAM and don't start swapping)
>>>
>>> On Fri, Apr 12, 2019 at 7:20 AM Peng Yu <address@hidden> wrote:
>>>
>>>> Hi,
>>>>
>>>> I need to load a large dataset into memory before the process of some
>>>> other small data can be done. This will be inefficient if I need to
>>>> load the data again and again whenever I need to process different
>>>> small data. Is there a way to make the large data in memory so that
>>>> different awk processes can read them without having to reload them
>>>> into the memory? Thanks.
>>>>
>>>> --
>>>> Regards,
>>>> Peng
>>>>
>>>> --
>> Regards,
>> Peng
>>
> --
Regards,
Peng