Re: Re-write of documentation tree build process

[Han-Wen Nienhuys]

John Mandereau escreveu:
> Hello all,
> 
> Here's a (long-awaited) patch against the stable/2.10 branch.
> 
> 'make web' can now generate two documentation trees in
> <src-dir>/out-www:
> 
> - one in online-root/, where .png and .html extensions are stripped in
> HTML files, for hosting on a web site with content negociation.
> 
> - the other in /offline-root, for browsing the docs on a hard disk.
> 
> Only one of these trees or both can be generated, depending on the
> FINAL_TARGETS variable:
> 
> make FINAL_TARGETS=offline web  # same as 'make web'
> make FINAL_TARGETS=online web
> 
> only builds one tree, and
> 
> make FINAL_TARGETS="offline online" web
> 
> builds both.
> 
> If this can be applied and a possible link to the French tutorial in
> index.html.in is added, Jan and Han-Wen may want to release 2.10.3 or
> 2.10.2-2, as the current docball is broken because of .html stripping
> (maybe the 2.11 docball is broken too).

Hello John,

thanks for your patch. Before it's applied, I think it should be
cleaned up a bit first.

First of all: do you think you could shove the entire 

  local-WWW-post

target into a python script? In that case, we don't need any awkward
quoting, and can junk most of the command line decoding, as we could
do

  import mirrortree
  mirrortree.miror_tree ('out-www/offline-webroot', {'ignore-dirs': 
'foo.*(ext|') , 'lose-dirs': 'more-regexes'})

(or use keyword arguments to pass options.)    


> diff --git a/stepmake/bin/add-html-footer.py b/stepmake/bin/add-html-footer.py
> index 08ff5ff..9a28237 100644
> --- a/stepmake/bin/add-html-footer.py
> +++ b/stepmake/bin/add-html-footer.py
> @@ -13,7 +13,7 @@ import getopt
>  index_url=''
>  top_url=''
>  changelog_file=''
> -content_negotiation = False
> +targets = []

I think that this script could be cleaned up a lot if didn't try to do
two targets in one go. Can you look into doing something like

  find  online-root -name '*.html' | xargs $(footify)
  find  offline-root -name '*.html' | xargs $(footify)

if that's unpractical, it's probably better to do one target a time,
so you can remove the 

  for t in targets:  
    if t == 'online'
    elif t == 'offline'

logic.


> +def help ():
> +    sys.stdout.write (r"""Usage: build-trees [OPTIONS]... TARGETS
> +Build trees for different targets by hardlinking input trees.
> +
> +Options:
> + --input-root=DIR          use DIR as input tree root (default is current 
> directory,
> +                             multiple roots may be specified with a comma 
> separator)
> + --target-pattern=STRING   use STRING as target root directory name pattern
> + --lose-dir=PATTERN        lose directories whose name matches PATTERN in 
> copies
> +                             (write its content to parent)

is this to remove out-www from the paths?  I'd call it
--strip-component or similar.  The difference between --exclude-dir is
not clear.

> + --process-dir=PATTERN     only process files in directories whose name 
> matches PATTERN

"only process directories matching PATTERN"

> + --common-files=PATTERN    filters files commmon to all trees
> +                             (they are hardlinked instead of being copied).
> + --specific-files=PATTERN  filters files specific to different trees
> +                             (regular files are only hardlinked to first 
> target).
> + --exclude-dir=PATTERN     don't recurse into directories whose name matches 
> PATTERN
> + --exclude-files=PATTERN   exclude files whose name matches PATTERN
> + --dump-sfl=FILE           dump specific files list to FILE

"sfl " -> no abbreviations please. 


> + -h, --help                print this help
> +
> +PATTERN should be a Python regular expression.
> +Common and specific files which are symlinks are always copied to all 
> targets.
> +""")
> +    sys.exit (0)
> +
> +(options, targets) = getopt.getopt(sys.argv[1:], 'h', [
> +    'input-root=', 'lose-dir=', 'common-files=', 'help', 'specific-files=',
> +    'exclude-dir=', 'exclude-files=', 'dump-sfl=', 'process-dir=', 
> 'target-pattern=']) 

Nowadays, optparse is the standard module for handling option
parsing. Can you change to that?  See lilypond-book.py
(get_option_parser) for an example how to use.


> +input_roots = []
> +target_pattern = ''
> +
> +common_f = '.*'
> +specific_f = '.*'
> +excluded_d = ''
> +excluded_f = ''
> +process_d = '.*'
> +lost_d_names = ''
> +sfl_dump = ''
> +
> +#file_list_re = re.compile (r'(?:^| )(?:[^ ]*|".*?")(?: |$)')
> +
> +for opt in options:
> +    o = opt[0]
> +    a = opt[1]
> +    if o == '--input-root':
> +        input_roots = re.split (',', a)
> +    elif o == '--lose-dir':
> +        lost_d_names = a
> +    elif o == '--common-files':
> +        common_f = a
> +    elif o == '--specific-files':
> +        specific_f = a
> +    elif o == '-h' or o == '--help':
> +        help ()
> +    elif o == '--exclude-dir':
> +        excluded_d = a
> +    elif o == '--exclude-files':
> +        excluded_f = a
> +    elif o == '--process-dir':
> +        process_d = a
> +    elif o == '--target-pattern':
> +        target_pattern = a
> +    elif o == '--dump-sfl':
> +        sfl_dump = a
> +    else:
> +        raise 'unknown opt ', o
> +
> +if input_roots == []:
> +    input_roots =  ['.']
> +
> +for s in ['common_f', 'specific_f', 'excluded_d',
> +          'excluded_f', 'process_d', 'lost_d_names']:
> +    exec s + '_re = re.compile (' + s + ')'


Don't use exec. If you must, try


  locals()[s + "_re"] = ..

what's _f and _d?  Write _dir or _file if you mean that. 

> +strip_lost_d_names_re = re.compile ('/(?:' + lost_d_names + ')')
> +slash_re = re.compile ('/')
> +
> +
> +if '%s' in target_pattern:
> +    target_dirs = map (lambda s: target_pattern % s, targets)

usually, list comprehensions  are shorter and clearer,

  [p % s for p in targets]

> +for dir in target_dirs:
> +    os.mkdir (dir)


map(os.mkdir, target_dirs)

if you like short.


> +path_strip = lambda bigpath, root: bigpath[:len (root)]

what's this?

> +
> +def new_link_path (link, dir, r):
> +    l = slash_re.split (link)
> +    d = slash_re.split (dir)
> +    i = 0
> +    while l[i] == '..':
> +        if r.match (d[i]):
> +            del l[i]
> +        else:
> +            i += 1
> +    return reduce (lambda s1, s2: s1 + '/' + s2,
> +                   filter (lambda x: not r.match (x), l))


If you want to do it functionally, use zip to fold (l,d) together,
then filter on the d component, and unzip the resulting list.

> +    sfl = open (sfl_dump, 'w')
> +
> +for d in input_roots:
> +    for in_dir, dirs, files in os.walk(d):
> +        i = 0
> +        while i < len(dirs):
> +            if excluded_d_re.search (dirs[i]):
> +                del dirs[i]
> +            else:
> +                i += 1
> +        out_dir = strip_lost_d_names_re.sub ('', in_dir)
> +        if not lost_d_names_re.match (os.path.basename (in_dir)):
> +            for t in target_dirs:
> +                os.mkdir (os.path.join (t, out_dir))
> +        if not process_d_re.search (in_dir):
> +            continue
> +        for f in filter (lambda s: not excluded_f_re.match (s),
> files):

preferred:

  for f in files:
     if excluded_f_re.match (f):
       continue

> +            in_f = os.path.join (in_dir, f)
> +            if os.path.islink (in_f): # all symlinks are assumed to be 
> relative and to point to files in the input trees
> +                link_path = new_link_path (os.readlink (in_f), in_dir, 
> lost_d_names_re)
> +                for t in target_dirs:
> +                    os.symlink (link_path, os.path.join (t, out_dir, f))
> +            elif specific_f_re.match (f):
> +                os.link (in_f, os.path.join (target_dirs[0], out_dir, f)) # 
> only hardlink specific file in first target
> +                if sfl_dump:
> +                    sfl.write (os.path.join (out_dir, f) + '\n')
> +            elif common_f_re.match (f):
> +                for t in target_dirs:
> +                    os.link (in_f, os.path.join (t, out_dir, f))


> +if sfl_dump:
> +    sfl.close()

should be unnecessary.


-- 

Han-Wen Nienhuys - address@hidden - http://www.xs4all.nl/~hanwen

LilyPond Software Design
 -- Code for Music Notation
http://www.lilypond-design.com