Re: [Bug-apl] Performance issue when manipulating arrays

[Elias Mårtenson]

Update:

I just did some more testing, and it seems as my clone() optimisation was a bit too agressive.

The "temp" system needs to be changed a bit into a "copy on write" flag instead. In looking at the code, it seems as though that should be something that would be fairly trivial to do, and I've done some tests already.

However, I would much prefer to see an analysis from you before I do anything else. Simply see my previous hack (I refuse to call it a patch since it actually breaks things :-) ) as an indication as to what kind of performance improvements we might be able to see if we get this right.

Regards,

Elias

On 25 April 2014 13:08, Elias Mårtenson <address@hidden> wrote:

I ran the program through Callgrind and I found the performance bottleneck. The program spends most of its time cloning values that are only used once.

I applied the following patch that reduces the run time from a few minutes to a fraction of a second.

Jürgen, could you take a look at it and see if the fix is sound? The performance difference is really remarkable, and if it's correct, it's really worth applying.

Regards,

Elias

On 25 April 2014 00:21, Elias Mårtenson <address@hidden> wrote:

In writing a function that uses lib_file_io to load the content of an entire file into an array of strings, I came across a bad performance problem that I am having trouble narrowing down.

Here is my current version of the function: https://github.com/lokedhs/apl-tools/blob/e3e81816f3ccb4d8c56acc8e4012d53f05de96d6/io.apl#L8

The first version did not do blocked reads and resized the array after each row was read. That was terribly slow, so I preallocate a block of 1000 elements, and resize every 1000 lines, giving the version you can see linked above.

I was testing with a text file containing almost 14000 rows, and on my laptop it takes many minutes to load the file. One would expect that the time taken to load such a small file should not take any noticeable time at all.

One interesting aspect of this is that it takes longer and longer to load each row as the loading proceeds. I have no explanation as to why that is the case. It's not the resizing that takes time, I was measuring the time taken to load a block of rows excluding the array resize.

Any ideas?

Regards,

Elias