Re: xorriso format seems not to respect size argument

[Thomas Schmitt]

Hi,

carsten.kunze@arcor.de wrote:
> So this means the drive firmware has a bug and there is nothing which can be
> done by Unix libs and tools?

It is not decided yet. We would know if dvd+rw-format, too, would not be
able to format the BD-R to the desired size.

What we know is that libburn's SCSI command was not obeyed completely.
The drive did format the medium, but not to the size that was given in the
SCSI transaction which did the formatting.

If dvd+rw-format succeeds with formatting a yet unformatted BD-R to the
desired size, then it would be time for source code experiments in libburn
(e.g. in its copy that comes with the GNU xorriso tarball).
I identified two deliberate differences in what libburn does and what i
think dvd+rw-format would do. The experiments would be about letting libburn
do what dvd+rw-format does.


> Currently I live either with the much too large
> 768M spare or no spare at all, all formatting done (or none) by growisofs.

So up to now you did not format a 23610 MiB BD-R with dvd+rw-format ?

If you want to try and feel apt to make code changes, it would be worth to
let dvd+rw-format print its FORMAT UNIT transaction data before performing
the transaction.
That would be in my copy of dvd+rw-format.cpp between lines 811 and 812,
like in the patch below, which i only compiled but did not test with a
medium yet.

The line numbers might differ between our code copies. So as additional
hint: This code is the case "else if (mmc_profile==0x41) // BD-R".

=======================================================================
--- dvd+rw-format.cpp.C10108    2021-01-08 10:18:02.430120518 +0100
+++ dvd+rw-format.cpp   2021-01-08 10:22:33.082121540 +0100
@@ -809,6 +809,17 @@ printf("DEBUG:  ssa = %d\n", ssa);
        cmd[1] = 0x11;          // "FmtData"|"Format Code"
        cmd[5] = 0;
        formats[i+1] = 0x2;     // IMMED"
+
+{ int j;
+        fprintf(stderr, "BD-R FORMAT command:");
+        for (j = 0; j < 8; j++)
+          fprintf(stderr, " %2.2x", (unsigned int) cmd[j]);
+        fprintf(stderr, "\nParameter list:");
+        for (j = 0; j < 12; j++)
+          fprintf(stderr, " %2.2x", (unsigned int) formats[i + j]);
+        fprintf(stderr, "\n");
+}
+
        if ((err=cmd.transport(WRITE,formats+i,12)))
            sperror ("FORMAT UNIT",err), exit(1);
=======================================================================


> Is BD defect management (the reread and compare) done by the drive
> itself or is it implemented in libburn?

It is entirely internal to the drive. The burn program just sends data.
Checkreading and possible spare area usage are done by the drive without
asking back to the program.


> It seems to me it does not work with
> at least growisofs (burn fails unexpected with good quality Verbatim BD-R,
> the spare range seems to be left untouched).

It does not work well with any program and drive, as soon as not-so-perfect
media are involved or the drive gets slightly visually impaired.
To my experience burning fails earlier than without Defect Management.

I mostly use BD-R unformatted and BD-RE with xorriso command
  -stream_recording on
The noise of unformatted BD-R can be annoying. So a low speed like
  -speed 4
might be comfortable.

The only case where i use Defect Management is with BD-RE which shall take
many backup sessions before it is full. This causes the first 32 blocks to
be overwritten after each session was added. There is some hope that
Defect Management can compensate for the uneven wear of those first blocks.


> Is it a good idea to disable defect management for long term archiving?

I think so.
Defect Management can have benefits only at write time. At read time of
a well recorded medium, it has no work to do. With not-so-well recordable
medium it can theoretically save the burn result from being a total
failure. (But as said, i still wait to see such a case after 12 years
of BD burning.)


> I only burn ready UDF images made with other tools.

You should checkread media after burning. Given various paddings added
at the end of the medium, you need to curb the read amount of data to the
size of the written amount. E.g. with a Debian installation ISO image:

  $ bytes=$(stat -c '%s' debian-10.6.0-amd64-netinst.iso)
  $ echo $bytes
  365953024
  $ blocks=$(expr $bytes / 2048)
  $ echo $blocks
  178688
  $ dd if=/dev/sr0 bs=2048 count=$blocks | md5sum
  ...
  42c43392d108ed8957083843392c794b  -
  $ cat debian-10.6.0-amd64-netinst.iso | md5sum
  42c43392d108ed8957083843392c794b  -
  $

Write the block count and the checksum down on a paper and store it with
the archive medium so you are able to later re-confirm the correct medium
content.

------------------------------------------------------------------------

With archive use cases, where xorriso creates the fileystem, it is
helpful to have own checksums on the medium, beyond the error correction
data which the drive adds to each 64 KiB chunk.

xorriso has the command -md5 "on", which is bundled with other high-fidelity
measures in xorriso command
  -for_backup

E.g. burn a medium with MD5 sums for superblock, tree, whole session, and
each single data file:

  to_be_archived=...path.to.the.tree.you.want.to.archive...

  xorriso -for_backup \
          -outdev /dev/sr0 \
          -blank as_needed \
          -volid ARCHIVE_"$(date '+%Y_%m_%d_%H%M%S')" \
          -map "$to_be_archived" / \
          -stream_recording on \
          -commit \
          -toc \
          -check_md5 FAILURE -- \
          -eject all

(Command -blank "as_needed" is given for the case you test with a used
BD-RE. It will then overwrite a few blocks on the medium, to make it
appear empty to xorriso.
Of course, "-stream_recording on" has no effect if the medium is not
formatted. I show it here for the case you want to use one of your already
formatted BD-R media without Defect Management being active, or with a
BD-RE.)

Above run will copy the desired file tree to the ISO 9660 root directory
(and underneath) and burn the medium by command "-commit".
When burning is done, the resulting medium state is reported by "-toc"
and a checkread is done with the recorded MD5s on the medium. If this
yields mismatches or read errors, the program will complain and abort.
Else it will eject the drive tray, if the drive has a tray motor.

A test run with a BD-RE in /dev/sr4 shows here:

  GNU xorriso 1.4.6 : RockRidge filesystem manipulator, libburnia project.

  ... write run messages ...

  Writing to '/dev/sr4' completed successfully.

  xorriso : NOTE : Re-assessing -outdev '/dev/sr4'  ('/dev/sg6')
  xorriso : NOTE : Loading ISO image tree from LBA 0
  xorriso : UPDATE : 3521 nodes read in 1 seconds
  Drive current: -dev '/dev/sr4'
  Media current: BD-RE
  Media status : is written , is appendable
  Media summary: 1 session, 90912 data blocks,  178m data, 22.9g free
  Volume id    : 'ARCHIVE_2021_01_08_110246'
  Drive current: -dev '/dev/sr4'
  Drive type   : vendor 'ASUS' product 'BW-16D1HT' revision '1.01'
  Drive id     : 'K9CF92A5012 '
  Media current: BD-RE
  Media product: VERBAT/IM0/0 , Mitsubishi Kagaku Media Co.
  Media status : is written , is appendable
  Media blocks : 90944 readable , 11997376 writable , 12088320 overall
  TOC layout   : Idx ,  sbsector ,       Size , Volume Id
  ISO session  :   1 ,        32 ,     90912s , ARCHIVE_2021_01_08_110246
  Media summary: 1 session, 90912 data blocks,  178m data, 22.9g free
  Media nwa    : 90944s
  Checking loaded session by its recorded MD5.
  Session MD5 01eac019a0f0a47b45207245d40c4afa , LBA 32 , 90883 blocks
  xorriso : UPDATE : 6029312 content bytes read in 5 seconds , 0.3xB
  xorriso : UPDATE :  82688k content bytes read in 10 seconds , 3.5xB
  xorriso : UPDATE :  156.3m content bytes read in 15 seconds , 3.5xB
  xorriso : UPDATE :  177.5m content bytes read in 16 seconds = 2.5xB
  Ok, session data match recorded md5.

... and the tray came out finally.

If you later want to re-confirm such a medium, do:

  xorriso -for_backup -indev /dev/sr0 -check_media --

which will report like

  ...
  Volume id    : 'ARCHIVE_2021_01_08_110246'
  xorriso : UPDATE :      32 blocks read in 3 seconds , 0.0xB
  xorriso : UPDATE : Found matching MD5 superblock tag: start=32 size=18
  xorriso : UPDATE : Found matching MD5 tree tag: start=32 size=310
  xorriso : UPDATE :    1760 blocks read in 4 seconds , 0.8xB
  xorriso : UPDATE :    7584 blocks read in 5 seconds , 2.6xB
  ...
  xorriso : UPDATE :   76704 blocks read in 16 seconds , 3.5xB
  xorriso : UPDATE :   84480 blocks read in 17 seconds , 3.5xB
  xorriso : UPDATE : Found matching MD5 session tag: start=32 size=90911
  xorriso : UPDATE :   90944 blocks read in 18 seconds = 2.3xB
  Media checks :        lba ,       size , quality
  Media region :          0 ,      90944 , + good
  Media region :      90944 ,   11997376 , 0 untested
  MD5 checks   :        lba ,       size , result
  MD5 tag range:         32 ,      90911 , + md5_match

Even if there are no MD5 sums recorded, such a run can be re-assuring if
all media ranges are "+ good" or "0 <some_excuse>".
Command -check_media can also be used to retrieve the remaining good blocks
from a partly bad medium. (The classic dd*rescue programs have not the means
means to talk directly to the drive. They have to talk to the kernel's
block layer, which might be suboptimal with damaged media.)


Have a nice day :)

Thomas