[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [bug #55243] Request for a way to indicate that the same recipe exec

From: David Boyce
Subject: Re: [bug #55243] Request for a way to indicate that the same recipe execution produces several targets
Date: Sat, 29 Dec 2018 13:31:52 -0800

I've only had time to skim this so apologies if my reply is (even more) off topic but the use of strace implies that you're doing what I call file auditing, tracking file usage via open system calls and the like. In this context I could mention a couple of my old OSS projects:

1. Poor Man's File Audit: https://github.com/boyski/pmaudit

This is a very lightweight way of autodiscovering file usage by monitoring metadata (atime, mtime). I won't say more here since it would just repeat the README.

2. Audited Objects: https://github.com/boyski/audited-objects

This is a more involved C program for Linux use which works by intercepting system calls in user space which I had to abandon some years ago due to day-job pressures. Roughly analogous to strace which at this point may be a better choice unless you get into advanced usage. You definitely don't want to touch the server side of this, which was being rewritten when I had to abandon it, but the client worked pretty well when I dropped it. Close to dead but mentioned for completeness.

David Boyce

On Fri, Dec 28, 2018 at 5:36 PM Brian Vandenberg <address@hidden> wrote:
> blah.h : blah.cc
>> no real recipe was provided, just adding this note for emphasis; this
> last part is how I suggest you solve it

Perfect but this does not work in the situations I am interested in,
that is where one single atomic command produces several files. Example:
pdflatex produces several files and there are cases where you want to
track some of them, e.g. document.pdf and document.aux.

I may be mistaken, but this sounds like a problem I tried tackling: force PDF's to re-build if any of its dependencies changed.  I started with pdftex's "-recorder" flag, but it doesn't allow you to control output filename / directory and there were other problems that made it completely un-suited to generating a dependency list.  After looking at all the docs, what others on the internet tried, etc, I was ready to throw in the towel.

The following ugly hack came about as I was thinking about how nice it'd be if I had something "magic" (eg, tup's use of fuse) to 'discover' what files were accessed; this is roughly what I threw together:

$ cat blah.sh
set -euo pipefail
strace -y -f -o blah.dep.tmp pdflatex -halt-on-error -interaction=batchmode "address@hidden"
perl -ni -e 'while( m{<([^>]+)>}g ) { my $tmp = $1; $tmp =~ m{'^$(pwd)'} && print "$tmp\n"}' blah.dep.tmp
echo "address@hidden : $(sort -u blah.dep.tmp)" > blah.dep

$ cat makefile
-include $(shell find docs -name '*.dep')
# note lack of dependencies listed; they come from the .dep files
%.pdf :
> blah.sh address@hidden

* First time the build runs .dep files don't exist and it'll just assume everything is ok; subsequent builds will have all necessary dependencies
* The "-y" flag in strace causes it to emit paths or descriptive text (eg, socket:12344321) in <>s for file descriptor arguments to functions:
fstat(3</path/to/some/file>, {st_mode=S_IFREG|0644, st_size=170187, ...}) = 0
* The paths within <> are always fully qualified; the perl one-liner extracts each path and if it is a descendent of $(pwd) it gets printed; obviously the paths need to be filtered further to distinguish between output -vs- input, etc; the above is just a rough sketch

* It's stupid; clever, but stupid
* Platforms without strace would require a different solution
* The "-y" flag is a relatively recent addition to strace; without it, extracting valid paths is more error prone.

dtrace (solaris/bsd) and system tap (linux) scripts are also an option; in fact if I weren't looking at moving to tup I'd probably go that route.  The only hurdle there is that execution of the script would either require the user to have special permissions or the script needs to be owned by a user with those special permissions & marked setuid so normal users can execute it.


Bug-make mailing list

reply via email to

[Prev in Thread] Current Thread [Next in Thread]