[Monotone-devel] nvm.dates

[Markus Wanner]

Hi,

I'm considering nvm.dates ready to land on mainline. All it does is
using a 64 bit integer to represent dates internally, so that date_t now
provides comparison and difference operators on these timestamps. It
does not alter the textual representation in the database nor for netsync.

As there seems to be a majority voting against further checking of the
date cert information, I'm not continuing my efforts into that
direction. Instead I've turned that portion of code into an additional
"--full" helper option for "mtn db info", which now displays some
statistic analysis of those date certs, see below for examples - no
warnings, no further checking, only "db info" is affected. I've
committed that into a separate branch nvm.dates.statistics. Please
review as well, I consider that to be ready to land on mainline, too.

Regards

Markus Wanner


Here's the relevant additional part of "db info --full" for some of the
larger monotone databases. Admittedly, the seconds values are often too
large to really be human readable, but I'm not overly keen about writing
beautification code for that.


monotone
========

counts:
  ancestry edges  : 17,827

timestamp correctness between revisions
  correct dates   : 17,733 edges
  equal dates     : 1 edges
  incorrect dates : 61 edges
  based on root   : 32 edges
  missing date(s) : 0 edges

timestamp differences between revisions:
  mean            : -28,365 sec
  min             : -1,944,060 sec
  max             : 60,951,415 sec

  1st percentile  : 6 sec
  5th percentile  : 48 sec
  10th percentile : 134 sec
  25th percentile : 1,032 sec
  50th percentile : 10,026 sec
  75th percentile : 72,933 sec
  90th percentile : 263,267 sec
  95th percentile : 613,691 sec
  99th percentile : 3,904,735 sec


botan
=====

counts:
  ancestry edges  : 1,797

timestamp correctness between revisions
  correct dates   : 1,793 edges
  equal dates     : 0 edges
  incorrect dates : 3 edges
  based on root   : 1 edges
  missing date(s) : 0 edges

timestamp differences between revisions:
  mean            : 115,312 sec
  min             : -29,516 sec
  max             : 19,441,763 sec

  1st percentile  : 9 sec
  5th percentile  : 24 sec
  10th percentile : 43 sec
  25th percentile : 160 sec
  50th percentile : 736 sec
  75th percentile : 6,181 sec
  90th percentile : 96,268 sec
  95th percentile : 355,795 sec
  99th percentile : 2,287,001 sec


openembedded
============

counts:
  ancestry edges  :  32,693

timestamp correctness between revisions
  correct dates   : 32,185 edges
  equal dates     : 1 edges
  incorrect dates : 500 edges
  based on root   : 7 edges
  missing date(s) : 0 edges

timestamp differences between revisions:
  mean            : 24,547 sec
  min             : -49,295,411 sec
  max             : 49,409,456 sec

  1st percentile  : -6,927 sec
  5th percentile  : 35 sec
  10th percentile : 65 sec
  25th percentile : 263 sec
  50th percentile : 1,757 sec
  75th percentile : 8,855 sec
  90th percentile : 39,635 sec
  95th percentile : 81,190 sec
  99th percentile : 410,294 sec


pidgin
======

counts:
  ancestry edges  :  28,353

timestamp correctness between revisions
  correct dates   : 28,313 edges
  equal dates     : 11 edges
  incorrect dates : 25 edges
  based on root   : 4 edges
  missing date(s) : 0 edges

timestamp differences between revisions:
  mean            : 52,390 sec
  min             : -10,292,985 sec
  max             : 63,591,821 sec

  1st percentile  : 14 sec
  5th percentile  : 62 sec
  10th percentile : 129 sec
  25th percentile : 583 sec
  50th percentile : 3,000 sec
  75th percentile : 16,084 sec
  90th percentile : 62,077 sec
  95th percentile : 116,494 sec
  99th percentile : 675,252 sec


xaraya
======

counts:
  ancestry edges  :  23,413

timestamp correctness between revisions
  correct dates   : 17,306 edges
  equal dates     : 5,781 edges
  incorrect dates : 256 edges
  based on root   : 67 edges
  missing date(s) : 3 edges

timestamp differences between revisions:
  mean            : -88,445 sec
  min             : -80,315,704 sec
  max             : 80,488,504 sec

  1st percentile  : -345,601 sec
  5th percentile  : 0 sec
  10th percentile : 0 sec
  25th percentile : 0 sec
  50th percentile : 86,400 sec
  75th percentile : 345,601 sec
  90th percentile : 1,036,800 sec
  95th percentile : 2,335,023 sec
  99th percentile : 19,489,817 sec


This also solves the mystery of the xaraya repository: I've counted
equal dates as invalid before. Obviously they often use some sort of
automatic or scripted merge or propagation of revisions, which then end
up having equal timestamps.

The analysis confirms my assumption that developer machine's clocks are
mostly close enough to the commonly known global clock. Percentages of
invalid date cert information for the checked repositories are:

monotone: 0.34%
botan:    0.17%
oe:       1.53%
pidgin:   0.09%
xaraya:   1.09%