[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH] sort: Add --threads option, which parallelizes internal sort
|
From: |
Jim Meyering |
|
Subject: |
Re: [PATCH] sort: Add --threads option, which parallelizes internal sort. |
|
Date: |
Thu, 28 May 2009 21:33:21 +0200 |
Glen Lenker wrote:
> On Thu, Mar 26, 2009 at 09:50:08PM +0000, Ralf Wildenhues wrote:
>> Hi Paul, all,
>>
>> Paul Eggert writes:
>> >
>> > This patch is by Glen Lenker, Matt Pham, Benjamin Nuernberger, Sky
>> > Lin, TaeSung Roh, and Paul Eggert. It adds support for parallelism
>> > within an internal sort. On our simple tests on a 2-core desktop x86,
>> > overall performance improved by roughly a factor of 1.6.
>>
>> This is too interesting to pass up.
>>
>> Example run, on an 8-way, and with cat'ed instances of the dictionary,
>> on tmpfs, timings best of three:
>
> Hey Ralf, did you happen to specify the amount of RAM sort should
> use. Not specifying enough RAM for sort would force break what would
> be a single internal sort into multiple internal sort passes and then
> an external sort. As it is external sort is still sequential.
>
>> runtime [s] threads
>> file size [MB] 1 2 4 8
>> 1 0.06 0.04 0.03 0.04
>> 2 0.13 0.09 0.07 0.07
>> 4 0.28 0.20 0.16 0.15
>> 8 0.61 0.43 0.34 0.32
>> 16 1.34 0.94 0.74 0.72
>> 32 3.00 2.06 1.63 1.57
>> 64 6.36 4.38 3.44 3.32
>> 128 13.49 9.30 7.13 7.24
>> 256 28.62 19.49 15.17 15.18
>
> I ran these tests on a 256MB instance of the dictionary in tmpfs, on a
> 8-core machine specifying 2G of RAM.
>
> runtime [s] threads
> file size [MB] 1 2 4 8
> 256 2m41.219 1m27.357 52.53 36.429
>
>> Here's the abbreviated 'time' output for the last row:
>>
>> 26.95user 1.67system 0:28.62elapsed 100%CPU
>> 30.78user 1.98system 0:19.49elapsed 168%CPU
>> 37.41user 2.04system 0:15.17elapsed 260%CPU
>> 60.68user 2.79system 0:15.18elapsed 417%CPU
>
> I forgot to use your time format in the test above, this is from a
> seperate test run.
>
> 160.22user 1.34system 2:41.61elapsed 99%CPU
> 159.83user 1.45system 1:27.12elapsed 185%CPU
> 159.84user 1.56system 0:52.26elapsed 308%CPU
> 160.67user 1.53system 0:36.26elapsed 447%CPU
>
> This seems to be what I would expect from a good implementation.
>
>> It suggests to me that too much time is spent busy-waiting in pthread_join,
>> or that sort is computing too much (I haven't looked at the patch in detail).
>>
>> Also, I'd have expected the rate going from 1 to 2 threads to get at least
>> a bit better with bigger file size, but it remains remarkably constant,
>> around 1.45 for this setup. What am I missing?
Hi Glen,
I ran some tests (see below), and got results that look
similar to Ralf's. I used an otherwise-idle 16-core Nehalem-based
system with 8GB of RAM.
However, more importantly, while the 16- and 8-thread tests were
running, I sampled thread counts using ps and was surprised to see
the number of active threads was usually far less than the maximum.
First the data:
cat /usr/share/dict/words > k
n=$(echo '250*1024^2 / '$(wc -c < k)|bc -l|sed 's/\..*//')
n=$(expr $n + 1)
# Create a file of size ~250MB
: > in; for i in $(seq $n); do echo $i; shuf k >> in; done
$ du -sh in :
251M in
for t in 16 8 4 2 1; do
echo T=$t
for i in 1 2 3 4; do
env time --format %e sort --threads=$t --buffer-size=2G < in > /dev/null
done
done
T=16 T is number of threads
12.96 <--- This is elapsed wall-clock time in seconds.
16.43
13.76
16.60
T=8
18.44
18.53
18.30
18.19
T=4
20.82
20.86
21.17
20.72
T=2
28.15
28.25
28.33
28.16
T=1
43.36
43.43
35.09
43.42
Running it again:
T=16
16.53
13.54
13.87
16.79
T=8
16.82
18.23
18.07
18.35
T=4
22.86
22.59
22.62
20.72
T=2
28.22
28.27
28.24
28.22
T=1
43.29
35.10
43.32
35.04
========================================
Then, I reran the 16-thread tests like this in one window
$ for t in 16; do :
echo T=$t
for i in 1 2 3 4; do
env time --format %e sort --threads=$t --buffer-size=500M < in > /dev/null
done
done
while recording sort thread counts in another with this:
for i in $(seq 1600); do
n=$(env ps -C sort -o nlwp=)
date "+%s.%N $n"; sleep .05
done | tee log
The timings are much like the ones above,
T=16
15.92
14.75
16.76
16.68
but the thread counts are surprising:
(I started the thread-counter slightly before the sort commands)
1243538765.058470000
1243538765.143037000
1243538765.226945000
1243538765.310864000
1243538765.395248000
1243538765.478934000
1243538765.562841000
1243538765.650482000
1243538765.734979000
1243538765.818997000
1243538765.903010000
1243538765.986913000
1243538766.071028000
1243538766.156459000
1243538766.240510000 1
1243538766.312877000 1
1243538766.384981000 1
1243538766.456768000 1
1243538766.528718000 1
1243538766.601155000 1
1243538766.672983000 1
1243538766.745109000 1
1243538766.817065000 1
1243538766.940538000 16
1243538767.091641000 16
1243538767.235828000 16
1243538767.333629000 12
1243538767.405164000 8
1243538767.484498000 6
1243538767.571476000 4
1243538767.648512000 4
1243538767.719953000 3
1243538767.796623000 2
1243538767.881325000 2
1243538767.956114000 2
1243538768.032149000 2
1243538768.108019000 2
1243538768.179573000 2
1243538768.263249000 1
1243538768.341043000 1
1243538768.417324000 1
1243538768.493397000 1
1243538768.569287000 1
1243538768.645266000 1
1243538768.721314000 1
1243538768.797348000 1
1243538768.873369000 1
1243538768.949454000 1
1243538769.025099000 1
1243538769.101195000 1
1243538769.177205000 1
1243538769.253189000 1
1243538769.329288000 1
1243538769.405225000 1
1243538769.481172000 1
1243538769.557269000 1
1243538769.633256000 1
1243538769.704925000 1
1243538769.776947000 1
1243538769.848853000 1
1243538769.920894000 1
1243538770.001490000 13
1243538770.137652000 16
1243538770.252296000 16
1243538770.332225000 16
1243538770.429087000 13
1243538770.501546000 10
1243538770.575762000 8
1243538770.661568000 5
1243538770.739968000 4
1243538770.815859000 4
1243538770.891889000 3
1243538770.966635000 2
1243538771.040183000 2
1243538771.112079000 2
1243538771.184037000 2
1243538771.255904000 1
1243538771.328287000 1
1243538771.401391000 1
1243538771.473415000 1
1243538771.545433000 1
1243538771.617501000 1
1243538771.693027000 1
1243538771.765409000 1
1243538771.837444000 1
1243538771.909372000 1
1243538771.981375000 1
1243538772.053534000 1
1243538772.125751000 1
1243538772.201428000 1
1243538772.273399000 1
1243538772.345472000 1
1243538772.417459000 1
1243538772.489366000 1
1243538772.561388000 1
1243538772.633332000 1
1243538772.705358000 1
1243538772.781393000 1
1243538772.853367000 1
1243538772.925488000 1
1243538772.997368000 1
1243538773.069565000 1
1243538773.142100000 1
1243538773.217439000 1
1243538773.289345000 1
1243538773.361410000 1
1243538773.433517000 1
1243538773.505587000 1
1243538773.577728000 1
1243538773.649642000 1
1243538773.730255000 16
1243538773.994483000 16
1243538774.081977000 16
1243538774.165573000 14
1243538774.247914000 8
1243538774.324361000 6
1243538774.400129000 4
1243538774.476354000 4
1243538774.552069000 3
1243538774.625871000 2
1243538774.697848000 2
1243538774.769158000 2
1243538774.840369000 2
1243538774.912362000 2
1243538774.984375000 2
1243538775.056268000 1
1243538775.129348000 1
1243538775.201369000 1
1243538775.273371000 1
1243538775.345382000 1
1243538775.417359000 1
1243538775.489409000 1
1243538775.561445000 1
1243538775.633415000 1
1243538775.705423000 1
1243538775.777401000 1
1243538775.849409000 1
1243538775.921413000 1
1243538775.993428000 1
1243538776.065492000 1
1243538776.137336000 1
1243538776.209453000 1
1243538776.281370000 1
1243538776.353282000 1
1243538776.425300000 1
1243538776.497329000 1
1243538776.569319000 1
1243538776.641331000 1
1243538776.713319000 1
1243538776.785348000 1
1243538776.857371000 1
1243538776.929368000 1
1243538777.001386000 1
1243538777.073365000 1
1243538777.145576000 1
1243538777.217499000 1
1243538777.296191000 13
1243538777.376093000 16
1243538777.472967000 15
1243538777.549235000 12
1243538777.624368000 6
1243538777.709047000 4
1243538777.780475000 3
1243538777.852488000 2
1243538777.924411000 2
1243538777.996476000 2
1243538778.076647000 1
1243538778.159684000 1
1243538778.243645000 1
1243538778.326961000 1
1243538778.410996000 1
1243538778.495064000 1
1243538778.579051000 1
1243538778.662975000 1
1243538778.746855000 1
1243538778.830831000 1
1243538778.914953000 1
1243538778.998945000 1
1243538779.082912000 1
1243538779.166932000 1
1243538779.251308000 1
1243538779.335173000 1
1243538779.419070000 1
1243538779.503033000 1
1243538779.586924000 1
1243538779.671163000 1
1243538779.754925000 1
1243538779.839015000 1
1243538779.922927000 1
1243538780.007214000 1
1243538780.090986000 1
1243538780.176473000 1
1243538780.259259000 1
1243538780.343099000 1
1243538780.427083000 1
1243538780.511146000 1
1243538780.595188000 1
1243538780.679108000 1
1243538780.763296000 1
1243538780.847216000 1
1243538780.931226000 1
1243538781.015235000 1
1243538781.099220000 1
1243538781.183234000 1
1243538781.267156000 1
1243538781.351460000 1
1243538781.431482000 1
1243538781.515702000 1
1243538781.599083000 1
1243538781.683182000 1
1243538781.767198000 1
1243538781.851238000 1
1243538781.935215000 1
1243538782.019332000 1
1243538782.103630000 1
1243538782.188836000 1
1243538782.273443000 1
1243538782.345217000 1
1243538782.417181000 1
1243538782.489123000 1
1243538782.561665000 1
1243538782.633752000 1
1243538782.705643000 1
1243538782.777715000 1
1243538782.871513000 16
1243538782.956642000 16
1243538783.184162000 16
1243538783.261457000 14
1243538783.334094000 12
1243538783.411650000 7
1243538783.489796000 5
1243538783.565805000 4
1243538783.639753000 4
1243538783.717667000 3
1243538783.794005000 2
1243538783.871784000 2
1243538783.948660000 2
1243538784.024842000 2
1243538784.100676000 2
1243538784.176372000 1
1243538784.253500000 1
1243538784.329577000 1
1243538784.405677000 1
1243538784.481629000 1
1243538784.557792000 1
1243538784.633813000 1
1243538784.709724000 1
1243538784.785614000 1
1243538784.861751000 1
1243538784.937820000 1
1243538785.013855000 1
1243538785.089725000 1
1243538785.165785000 1
1243538785.241902000 1
1243538785.317304000 1
1243538785.393260000 1
1243538785.469301000 1
1243538785.545303000 1
1243538785.621397000 1
1243538785.697416000 1
1243538785.773391000 1
1243538785.849455000 1
1243538785.925435000 1
1243538786.001399000 1
1243538786.077461000 1
1243538786.153365000 1
1243538786.229532000 1
1243538786.305405000 1
1243538786.382825000 1
1243538786.458863000 1
1243538786.534934000 1
1243538786.610878000 1
1243538786.687023000 15
1243538786.774619000 16
1243538786.851056000 16
1243538786.930931000 16
1243538787.035787000 16
1243538787.121424000 16
1243538787.202669000 8
1243538787.274052000 7
1243538787.345780000 4
1243538787.417928000 4
1243538787.489390000 3
1243538787.561149000 2
1243538787.633068000 2
1243538787.705114000 2
1243538787.777100000 2
1243538787.849072000 2
1243538787.920697000 2
1243538788.003714000 2
1243538788.090353000 1
1243538788.161991000 1
1243538788.234266000 1
1243538788.306105000 1
1243538788.378024000 1
1243538788.450084000 1
1243538788.522126000 1
1243538788.594013000 1
1243538788.666069000 1
1243538788.738138000 1
1243538788.809926000 1
1243538788.881947000 1
1243538788.953942000 1
1243538789.036887000 1
1243538789.109577000 1
1243538789.181908000 1
1243538789.254002000 1
1243538789.325989000 1
1243538789.397997000 1
1243538789.470046000 1
1243538789.542080000 1
1243538789.614205000 1
1243538789.686061000 5
1243538789.818990000 16
1243538789.899136000 16
1243538790.092869000 16
1243538790.174503000 13
1243538790.246417000 10
1243538790.317791000 7
1243538790.402331000 4
1243538790.499904000 4
1243538790.578448000 4
1243538790.649058000 2
1243538790.731908000 2
1243538790.815882000 2
1243538790.896288000 2
1243538790.979965000 1
1243538791.064086000 1
1243538791.147743000 1
1243538791.231740000 1
1243538791.315963000 1
1243538791.399903000 1
1243538791.483842000 1
1243538791.567758000 1
1243538791.651883000 1
1243538791.735903000 1
1243538791.819772000 1
1243538791.903713000 1
1243538791.987635000 1
1243538792.069328000 1
1243538792.152166000 1
1243538792.236467000 1
1243538792.320548000 1
1243538792.403688000 1
1243538792.487864000 1
1243538792.591437000 16
1243538792.712185000 16
1243538792.793718000 12
1243538792.871008000 8
1243538792.946682000 4
1243538793.018055000 3
1243538793.090662000 2
1243538793.162519000 2
1243538793.233846000 2
1243538793.305783000 2
1243538793.378437000 1
1243538793.459946000 1
1243538793.540277000 1
1243538793.624072000 1
1243538793.707935000 1
1243538793.791771000 1
1243538793.875800000 1
1243538793.959830000 1
1243538794.043897000 1
1243538794.127713000 1
1243538794.212264000 1
1243538794.297318000 1
1243538794.380099000 1
1243538794.463873000 1
1243538794.547856000 1
1243538794.629323000 1
1243538794.711997000 1
1243538794.795897000 1
1243538794.879843000 1
1243538794.964206000 1
1243538795.047895000 1
1243538795.131824000 1
1243538795.215797000 1
1243538795.299929000 1
1243538795.383893000 1
1243538795.467880000 1
1243538795.551881000 1
1243538795.636012000 1
1243538795.719973000 1
1243538795.803987000 1
1243538795.887933000 1
1243538795.972112000 1
1243538796.055999000 1
1243538796.139942000 1
1243538796.223946000 1
1243538796.309371000 1
1243538796.391964000 1
1243538796.476023000 1
1243538796.560045000 1
1243538796.644022000 1
1243538796.728618000 1
1243538796.812777000 1
1243538796.897672000
1243538796.969787000 1
1243538797.054727000 1
1243538797.126217000 1
1243538797.198251000 1
1243538797.270200000 1
1243538797.342049000 1
1243538797.422008000 1
1243538797.494840000 1
1243538797.566982000 1
1243538797.639078000 1
1243538797.711373000 15
1243538797.996419000 16
1243538798.075784000 16
1243538798.156387000 16
1243538798.232955000 9
1243538798.317453000 8
1243538798.399756000 4
1243538798.478715000 4
1243538798.554713000 4
1243538798.638053000 2
1243538798.720893000 2
1243538798.804976000 2
1243538798.889372000 2
1243538798.972922000 2
1243538799.056924000 2
1243538799.140323000 1
1243538799.224321000 1
1243538799.308419000 1
1243538799.392387000 1
1243538799.476455000 1
1243538799.560384000 1
1243538799.644433000 1
1243538799.728490000 1
1243538799.812405000 1
1243538799.896376000 1
1243538799.980437000 1
1243538800.064518000 1
1243538800.148266000 1
1243538800.232304000 1
1243538800.316366000 1
1243538800.400169000 1
1243538800.484206000 1
1243538800.568324000 1
1243538800.652289000 1
1243538800.736355000 1
1243538800.820320000 1
1243538800.904286000 1
1243538800.988702000 1
1243538801.072773000 1
1243538801.156372000 1
1243538801.240321000 1
1243538801.324468000 1
1243538801.412964000 16
1243538801.678988000 16
1243538801.755326000 16
1243538801.837006000 16
1243538801.911125000 14
1243538801.987726000 8
1243538802.058747000 5
1243538802.130813000 4
1243538802.202487000 3
1243538802.273078000 3
1243538802.357107000 2
1243538802.441141000 2
1243538802.525196000 2
1243538802.609136000 2
1243538802.693098000 2
1243538802.777154000 2
1243538802.861068000 1
1243538802.945029000 1
1243538803.029111000 1
1243538803.113112000 1
1243538803.197021000 1
1243538803.281011000 1
1243538803.365049000 1
1243538803.448985000 1
1243538803.533182000 1
1243538803.617193000 1
1243538803.701150000 1
1243538803.785475000 1
1243538803.869091000 1
1243538803.952891000 1
1243538804.037095000 1
1243538804.121121000 1
1243538804.203665000 1
1243538804.282602000 1
1243538804.365206000 1
1243538804.449162000 1
1243538804.533140000 1
1243538804.616749000 1
1243538804.701142000 1
1243538804.785041000 1
1243538804.869108000 1
1243538804.953194000 1
1243538805.037244000 1
1243538805.121319000 1
1243538805.376921000 16
1243538805.601978000 15
1243538805.678903000 10
1243538805.750877000 6
1243538805.835434000 4
1243538805.918255000 4
1243538805.991048000 4
1243538806.062499000 3
1243538806.134800000 2
1243538806.206762000 2
1243538806.278795000 2
1243538806.350722000 2
1243538806.422361000 1
1243538806.505728000 1
1243538806.589366000 1
1243538806.673730000 1
1243538806.757311000 1
1243538806.841360000 1
1243538806.925324000 1
1243538807.009444000 1
1243538807.093411000 1
1243538807.177329000 1
1243538807.261304000 1
1243538807.345331000 1
1243538807.429568000 1
1243538807.513582000 1
1243538807.597267000 1
1243538807.681074000 1
1243538807.765074000 1
1243538807.849072000 1
1243538807.933099000 1
1243538808.017260000 1
1243538808.101140000 1
1243538808.185198000 1
1243538808.269057000 1
1243538808.353056000 1
1243538808.437209000 1
1243538808.521293000 1
1243538808.605416000 1
1243538808.689504000 1
1243538808.783884000 16
1243538808.869203000 16
1243538808.948045000 16
1243538809.040134000 11
1243538809.115810000 5
1243538809.187016000 4
1243538809.258650000 3
1243538809.337958000 2
1243538809.410654000 1
1243538809.484275000 1
1243538809.569830000 1
1243538809.654119000 1
1243538809.737716000 1
1243538809.823248000 1
1243538809.905672000 1
1243538809.989686000 1
1243538810.073798000 1
1243538810.157617000 1
1243538810.241701000 1
1243538810.325529000 1
1243538810.409737000 1
1243538810.493575000 1
1243538810.577529000 1
1243538810.661522000 1
1243538810.745626000 1
1243538810.830453000 1
1243538810.913339000 1
1243538810.997818000 1
1243538811.081930000 1
1243538811.165795000 1
1243538811.249618000 1
1243538811.333591000 1
1243538811.417612000 1
1243538811.501638000 1
1243538811.586080000 1
1243538811.669769000 1
1243538811.753526000 1
1243538811.838311000 1
1243538811.921772000 1
1243538812.005874000 1
1243538812.089831000 1
1243538812.173734000 1
1243538812.257797000 1
1243538812.341879000 1
1243538812.425886000 1
1243538812.509904000 1
1243538812.593696000 1
1243538812.677753000 1
1243538812.761797000 1
1243538812.845749000 1
1243538812.929834000 1
1243538813.013934000 1
1243538813.097913000 1
1243538813.182823000 1
1243538813.255465000 1
1243538813.328251000 1
1243538813.399691000 1
1243538813.471706000 1
1243538813.543710000 1
1243538813.615698000 1
1243538813.687433000 1
1243538813.770749000 1
1243538813.853092000 1
1243538813.937059000 1
1243538814.021051000 1
1243538814.105136000 1
1243538814.189165000 1
1243538814.273208000 1
1243538814.354996000 14
1243538814.449246000 16
1243538814.714718000 16
1243538814.792060000 13
1243538814.876639000 10
1243538814.965559000 8
1243538815.039289000 5
1243538815.111584000 4
1243538815.183233000 4
1243538815.254985000 3
1243538815.327491000 2
1243538815.400004000 2
1243538815.471023000 2
1243538815.543043000 2
1243538815.614962000 2
1243538815.686872000 1
1243538815.759967000 1
1243538815.842964000 1
1243538815.915864000 1
1243538815.988053000 1
1243538816.059956000 1
1243538816.132047000 1
1243538816.204040000 1
1243538816.286223000 1
1243538816.369466000 1
1243538816.453416000 1
1243538816.537475000 1
1243538816.621462000 1
1243538816.705480000 1
1243538816.789354000 1
1243538816.873197000 1
1243538816.957302000 1
1243538817.041299000 1
1243538817.125241000 1
1243538817.209280000 1
1243538817.293220000 1
1243538817.377548000 1
1243538817.461423000 1
1243538817.545271000 1
1243538817.629313000 1
1243538817.713315000 1
1243538817.797326000 1
1243538817.881401000 1
1243538817.965491000 1
1243538818.053330000 11
1243538818.221065000 16
1243538818.502928000 13
1243538818.580806000 10
1243538818.659516000 5
1243538818.735423000 4
1243538818.811379000 4
1243538818.888344000 4
1243538818.963920000 3
1243538819.040213000 2
1243538819.114834000 2
1243538819.190744000 2
1243538819.266881000 2
1243538819.342915000 1
1243538819.419885000 1
1243538819.495819000 1
1243538819.571844000 1
1243538819.647873000 1
1243538819.723938000 1
1243538819.799900000 1
1243538819.875972000 1
1243538819.951933000 1
1243538820.028000000 1
1243538820.103946000 1
1243538820.180037000 1
1243538820.256010000 1
1243538820.331962000 1
1243538820.408093000 1
1243538820.483360000 1
1243538820.559278000 1
1243538820.635330000 1
1243538820.711384000 1
1243538820.787478000 1
1243538820.863519000 1
1243538820.939521000 1
1243538821.015493000 1
1243538821.091492000 1
1243538821.167568000 1
1243538821.243540000 1
1243538821.319436000 1
1243538821.395473000 1
1243538821.471608000 1
1243538821.548976000 1
1243538821.625159000 1
1243538821.705105000 1
1243538821.781019000 1
1243538821.900668000 16
1243538822.227980000 16
1243538822.304945000 12
1243538822.377637000 9
1243538822.452444000 4
1243538822.524332000 4
1243538822.596081000 4
1243538822.666906000 4
1243538822.752152000 2
1243538822.836251000 2
1243538822.907658000 2
1243538822.979786000 2
1243538823.051708000 2
1243538823.123030000 1
1243538823.205874000 1
1243538823.289931000 1
1243538823.373961000 1
1243538823.457936000 1
1243538823.541957000 1
1243538823.625856000 1
1243538823.709891000 1
1243538823.793965000 1
1243538823.877942000 1
1243538823.961997000 1
1243538824.042351000 1
1243538824.126533000 1
1243538824.209985000 1
1243538824.293859000 1
1243538824.377872000 1
1243538824.461817000 1
1243538824.545993000 1
1243538824.629853000 1
1243538824.713852000 1
1243538824.797847000 1
1243538824.882134000 1
1243538824.965912000 1
1243538825.049841000 1
1243538825.133788000 1
1243538825.217782000 1
1243538825.301757000 1
1243538825.386031000 1
1243538825.461887000 13
1243538825.667991000 15
1243538825.744415000 10
1243538825.818094000 4
1243538825.896322000 3
1243538825.972062000 2
1243538826.047960000 2
1243538826.120744000 1
1243538826.190206000 1
1243538826.274063000 1
1243538826.358026000 1
1243538826.442004000 1
1243538826.526165000 1
1243538826.610039000 1
1243538826.694108000 1
1243538826.778003000 1
1243538826.862150000 1
1243538826.946322000 1
1243538827.030194000 1
1243538827.113931000 1
1243538827.197946000 1
1243538827.281934000 1
1243538827.365958000 1
1243538827.450289000 1
1243538827.534115000 1
1243538827.618196000 1
1243538827.702134000 1
1243538827.785774000 1
1243538827.870207000 1
1243538827.951512000 1
1243538828.034339000 1
1243538828.118037000 1
1243538828.202030000 1
1243538828.286079000 1
1243538828.369979000 1
1243538828.454085000 1
1243538828.538085000 1
1243538828.622120000 1
1243538828.706099000 1
1243538828.790087000 1
1243538828.874097000 1
1243538828.958006000 1
1243538829.042213000 1
1243538829.126144000 1
1243538829.210234000 1
1243538829.294511000 1
1243538829.378225000 1
1243538829.462237000 1
1243538829.546131000 1
1243538829.630004000 1
1243538829.714271000 1
1243538829.798245000 1
1243538829.882153000 1
1243538829.966196000 1
1243538830.050376000 1
1243538830.134275000 1
1243538830.218186000 1
1243538830.302677000 1
1243538830.386393000
1243538830.470932000
1243538830.554965000
1243538830.638890000
1243538830.722894000
1243538830.806806000
1243538830.890892000
1243538830.974906000
1243538831.058959000
1243538831.142855000
1243538831.226869000
...