Re: [Bug-xorriso] generating reproducible ISOs with xorriso

[Daniel Kahn Gillmor]

On Thu 2015-06-04 19:34:32 -0400, Daniel Kahn Gillmor wrote:
> But the iso's still have bytes that differ.  here's a sample:
>
> 0 address@hidden:/tmp/cdtemp.NBVnsL$ diff -u <(hd < grub.weighted.iso ) <(hd 
> < a.weighted.iso )| head
> --- /dev/fd/63        2015-06-04 19:32:40.261987884 -0400
> +++ /dev/fd/62        2015-06-04 19:32:40.261987884 -0400
> @@ -79,7 +79,7 @@
>  00009830  00 00 00 41 6d 01 00 00  00 00 00 00 01 00 00 00  
> |...Am...........|
>  00009840  00 00 00 00 00 00 00 00  00 00 00 00 00 54 46 1a  
> |.............TF.|
>  00009850  01 0e 73 06 04 16 39 1e  00 73 06 04 16 39 1e 00  
> |..s...9..s...9..|
> -00009860  73 06 04 17 1d 04 00 43  45 1c 01 14 00 00 00 00  
> |s......CE.......|
> +00009860  73 06 04 17 1d 0b 00 43  45 1c 01 14 00 00 00 00  
> |s......CE.......|
>  00009870  00 00 14 00 00 00 00 00  00 00 00 ed 00 00 00 00  
> |................|
>  00009880  00 00 ed 00 60 00 13 00  00 00 00 00 00 13 00 08  
> |....`...........|
> 0 address@hidden:/tmp/cdtemp.NBVnsL$ 

Short version: i think this will be doable once debian gets an updated
version of libisoburn.  (see https://bugs.debian.org/787793)


Long version, if anyone is interested in the dirty details:

Every single one of these differences is this single octet, which moves
from 04 to 0b:

0 address@hidden:/tmp/cdtemp.NBVnsL$ cmp -l grub.weighted.iso a.weighted.iso |  
awk '{ print $2 " " $3 }' | sort | uniq -c
    314 4 13
0 address@hidden:/tmp/cdtemp.NBVnsL$ 

[ cmp byte value output is in octal, so:

 13 (octal) == 11 (decimal) == 0b (hex) ]

And there are a total of 307 files in the tree, which is pretty close to
314. (maybe there are 7 hidden files or metadata chunks or something
with the same label?)

after re-running "testtree /boot/grub", it's the same number of bytes
in the same locations that differ, but the bytes change to:

 314 56 65

so that's the same number offset:

   013 - 004 = 07
   065 - 056 = 07

and the cmpiso | head output is now:

--- /dev/fd/63  2015-06-05 00:34:17.660525287 -0400
+++ /dev/fd/62  2015-06-05 00:34:17.660525287 -0400
@@ -79,7 +79,7 @@
 00009830  00 00 00 41 6d 01 00 00  00 00 00 00 01 00 00 00  |...Am...........|
 00009840  00 00 00 00 00 00 00 00  00 00 00 00 00 54 46 1a  |.............TF.|
 00009850  01 0e 73 06 04 16 39 1e  00 73 06 04 16 39 1e 00  |..s...9..s...9..|
-00009860  73 06 05 04 20 2e 00 43  45 1c 01 14 00 00 00 00  |s... ..CE.......|
+00009860  73 06 05 04 20 35 00 43  45 1c 01 14 00 00 00 00  |s... 5.CE.......|
 00009870  00 00 14 00 00 00 00 00  00 00 00 ed 00 00 00 00  |................|
 00009880  00 00 ed 00 60 00 13 00  00 00 00 00 00 13 00 08  |....`...........|
 00009890  00 00 00 00 08 00 73 06  04 16 39 1e 00 02 00 00  |......s...9.....|

and yet another run gives me:

0 address@hidden:/tmp/cdtemp.NBVnsL$ cmp -l grub.weighted.iso a.weighted.iso |  
awk '{ print $2 " " $3 }' | sort | uniq -c
    314 47 50
    314 67 1
0 address@hidden:/tmp/cdtemp.NBVnsL$ diff -u <(hd < grub.weighted.iso ) <(hd < 
a.weighted.iso )| head -n 40
--- /dev/fd/63  2015-06-05 00:44:05.235205058 -0400
+++ /dev/fd/62  2015-06-05 00:44:05.235205058 -0400
@@ -79,7 +79,7 @@
 00009830  00 00 00 41 6d 01 00 00  00 00 00 00 01 00 00 00  |...Am...........|
 00009840  00 00 00 00 00 00 00 00  00 00 00 00 00 54 46 1a  |.............TF.|
 00009850  01 0e 73 06 04 16 39 1e  00 73 06 04 16 39 1e 00  |..s...9..s...9..|
-00009860  73 06 05 04 27 37 00 43  45 1c 01 14 00 00 00 00  |s...'7.CE.......|
+00009860  73 06 05 04 28 01 00 43  45 1c 01 14 00 00 00 00  |s...(..CE.......|
 00009870  00 00 14 00 00 00 00 00  00 00 00 ed 00 00 00 00  |................|
 00009880  00 00 ed 00 60 00 13 00  00 00 00 00 00 13 00 08  |....`...........|
 00009890  00 00 00 00 08 00 73 06  04 16 39 1e 00 02 00 00  |......s...9.....|

So i think this is one more timestamp, albeit one that rolls over at
around 60 in one octet, and the next value increments.

So this is definitely a timestamp, with one octet for the seconds.

Section 4.1.6 of the Rockridge extension draft spec:

   http://www.ymi.com/ymi/sites/default/files/pdf/Rockridge.pdf

Suggests that the leading TF there indicates a timestamp.

0x1a is supposed to be the length: 26 octets, right up until the next
"CE", or "Continuation area".


0x0e means this is the modification, access, and attribute access
timestamp, and the remaining bytes are speced as defined in section
9.1.5 of iso 9660:

 http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-119.pdf

that's 7 octets each, presumably one for each of the flags identified in
0x0e.

So that's:

  modification: 73 06 04 16 39 1e 00
        access: 73 06 04 16 39 1e 00
   attr access: 73 06 05 04 28 01 00

so the thing that's still being set is the attribute access time, if i'm
reading these specs correctly.

Digging around in the source, that means that we'd need to set bits 2 or
256 in the alter_date_r flag, but that's not possible in 1.3.2's version
of Xorriso_convert_datestring() in xorriso/parse_exec.c.  It was added
in http://libburnia-project.org/changeset/5190, after the (rather old)
version we have in debian.

Once we get that upgrade, we should be able to supply patches to grub's
debian/rules, which i'm tracking in https://bugs.debian.org/787795.

        --dkg