[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[sr #110636] asttable is very slow with many columns
From: |
Mohammad Akhlaghi |
Subject: |
[sr #110636] asttable is very slow with many columns |
Date: |
Sun, 10 Apr 2022 19:41:04 -0400 (EDT) |
Follow-up Comment #1, sr #110636 (project gnuastro):
Thanks a lot for reporting this.
I was able to reproduce the problem with this manually created table of 10e5
columns:
echo 1 100000 | awk '{for(i=0;i<$2;++i) printf "%d ", $1+i; print ""}' >
many-columns.txt
asttable many-columns.txt -i
Increasing the column count to 1e6 did indeed take very long and forced me to
manually close it too.
The problem with plain-text tables is that the reading can't be easily
parallelized (with FITS tables, since we know the number a-priori, in
Gnuastro, we parallelize the column reading).
If you do come up with a good solution for plain-text tables in NEMO, please
share it here.
In Gnuastro, we do have a format
<https://www.gnu.org/software/gnuastro/manual/html_node/Gnuastro-text-table-format.html>
to give metadata to plain-text tables. But that also won't necessarily help
unless all columns are string (their number of characters are known
a-priori).
Generally, besides theoretical checking of code, I am curious if there is any
real-world scenario that would require so many columns? Usually the workflow
can be re-designed (optimized) to avoid so many columns in one file. For
example FITS only accepts 999 columns.
_______________________________________________________
Reply to this item at:
<https://savannah.gnu.org/support/?110636>
_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/