[POKOLOGY] Writing binary utilities with GNU poke

[Jose E. Marchesi]

[This article is also available online at the Applied Pokology blog at
 http://www.jemarch.net/pokology]

Writing binary utilities with GNU poke
======================================

GNU poke is, first and foremost, intended to be used as an interactive
editor, either directly on the command line or using a graphical user
interface built on it.  However, since its conception poke was intended
to provide a suitable and useful foundation on which other programs, the
so-called binary utilities, could be written.  At last, the development
of poke has progressed to a point where we can start writing such
utilities, and the purpose of this article is to show a small, albeit
working and useful example of what can be achieved by writing a few
lines of Poke.

elfextractor
------------

We will be hacking a very simple utility called elfextractor, that
extracts the contents of the sections of an ELF file, whose name is
provided as an argument in the command line, into several output files.
This is the synopsis of the program:

   elfextractor FILE [SECTION_NAME]

Where FILE is the name of the ELF file from which to extract sections,
and an optional SECTION_NAME specifies the name of the section to
extract.

Say we have a file foo.o and we would like to extract its string table.
We would use elfextractor like:

   $ elfextractor foo.o .text

Provided foo.o indeed has a section named .text, the utility will create
a file foo.o.text with the section's contents.  Note how the names of
the output files are derived concatenating the name of the input ELF
file and the name of the extracted section.

If no section name is specified, then all sections are extracted.  For
example:

   $ elfextractor foo.o
   $ ls foo.o*
   foo.o         foo.o.eh_frame       foo.o.shstrtab  foo.o.symtab
   foo.o.comment foo.o.rela.eh_frame  foo.o.strtab    foo.o.text

Before writing elfextractor, however, we must first learn a few things
about writing Poke scripts.

Poke scripts
------------

In interactive usage, there are two main ways to execute Poke code: at
the interactive prompt (or REPL), and loading pickles.

Executing Poke code at the REPL is as easy as introducing a statement or
expression:

   (poke) print "Hello\n"
   Hello

Executing Poke code in a pickle is performed by loading the file
containing the code:

   (poke) .load say-hello.pk
   Hello

Where say-hello.pk contains simply:

   print "Hello\n";

However, we would like to have Poke scripts, i.e. to be able to execute
Poke programs as standalone programs, from the shell.  In other words,
we want to use GNU poke as an interpreter.  This is achieved by using a
shebang, which should appear at the top of the script file.  The poke
shebang looks like this:

   #!/usr/bin/poke -L
   !#

The -L command line option tells poke that it is being used as an
interpreter.  Additional arguments for poke can be specified before -L
(but not after).  The #! ... !# is an alternative syntax for multi-line
comments, which allows to have the shebang at the top of a Poke program
without causing a syntax error.  This nice trick has been borrowed from
guile.

Therefore, we could write say-hello as a Poke script like this:

   #!/usr/bin/poke -L
   !#
    
    print "Hello\n";

And then execute it like any other program or script:

    $ ./say-hello
    Hello

Handling command-line arguments
-------------------------------

When a Poke script is executed, the command line arguments passed to the
script are available in the array argv.  Example:

   #!/usr/bin/poke -L
   !#
   
   for (arg in argv)
     print "Argument: " + arg + "\n";

Executing this script results in:

   $ ./printargs foo bar 'baz quux'
   Argument: foo
   Argument: bar
   Argument: baz quux
    
Note how it is not needed to have an argc variable, since the number of
elements stored in a Poke array can be queried using an attribute:
argv'length.

Note also that argv is only defined when poke runs as an interpreter:

$ poke
[...]
(poke) argv
<stdin>:1:1: error: undefined variable 'argv'
argv;
^~~~

Exiting from scripts
--------------------

By default a Poke script will communicate a successful status to the
environment, upon exiting:

   $ cat hello
   #!/usr/bin/poke -L
   !#
   
   print "hello\n";
   $ ./hello && echo $?
   0

In order to exit with an explicitly provided status code, most typically
to signal an erroneous situation, the Pokeish way is to raise an E_exit
exception with the desired exit status code:

   raise Exception { code = EC_exit, exit_status = 1 };

This can be a bit cumbersome to write, so poke provides a more
conventional syntax in the form of an `exit' function:

   defun exit = (int<32> exit_code = 0) void:
   {
     raise Exception { code = EC_exit, exit_status = exit_code };
   }

So the above raise statement becomes the much simpler:

   exit (1);

Loading pickles as modules
--------------------------

elfextractor deals with ELF object files.  Extracting sections requires
dealing with several data structures encoded in the ELF file, such as
the header, the section header table, the string table (that contains
the names of the sections) and so on.  It would be of course possible to
define Poke types for these structures in the script itself but, as it
happens, GNU poke provides an already written pickle that describes the
ELF structures.  It is called elf.pk.

A script needing to mess with ELF data structures can just make use of
elf.pk using the load construction:

   load elf;

This will look for a file called elf.pk in a set of directories, which
are predefined by poke, and load it.  The list of directories where poke
looks for pickles is stored in the load_path variable as a colon
separated list of directory names, and can be customized:

   $ poke
   [...]
   (poke) load_path
   "/home/jemarch/.poke.d:.:/home/jemarch/.local/share/poke:..."

The default value of load_path contains both user-specific directories
and system-wide directories.  This assures that all the pickles
installed by poke are available, and that the user can load her own
pickles in her scripts.

Once a pickle is loaded in a script the types, functions and variables
defined in it (either directly or indirectly by loading its own pickles)
become available.
    
Back to elfextractor
--------------------

All right, now that we know more about writing Poke scripts, let's go
back to our original task: to write elfextractor.  This is an
implementation:

   #!/home/jemarch/gnu/hacks/poke/build/poke/poke -L
   !#
   
   /* elfextractor - Extract sections from ELF64 files. */
   
   load elf;
   
   if (!(argv'length in [1,2]))
     {
       print "Usage: elfextractor FILE [SECTION_NAME]\n";
       exit (1);
     }
   
   defvar file_name = argv[0];
   defvar section_name = (argv'length > 1) ? argv[1] : "";
   
   try
     {
       defvar fd = open (file_name, IOS_M_RDONLY);
       defvar elf = Elf64_File @ fd : 0#B;
   
       for (shdr in elf.shdr where shdr.sh_type != 0x0)
         {
           defvar sname = elf.get_string (shdr.sh_name);

           if (section_name == "" || sname == section_name)
             save :ios elf'ios :file file_name + sname
                 :from shdr.sh_offset :size shdr.sh_size;
         }

       close (fd);
     }
   catch (Exception e)
     {
       if (e == E_constraint)
         printf ("error: `%s' is not a valid ELF64 file\n", file_name);
       else if (e == E_io)
         printf ("error: couldn't open file `%s'\n", file_name);
       else
         raise e;

       exit (1);
     }

First the command line arguments are handled.  The script checks whether
the right number of arguments have been passed (either 1 or 2) exiting
with an error code otherwise.  The file name and the section name are
then extracted from the argv array.

Once we have the file name and the optional desired section name, it is
time to do the real work.  The code is enclosed in a try-catch block
statement, because some of the operations may result in exceptions being
raised.
    
First, the ELF file whose name is specified in the command line is
opened for reading:

   defvar fd = open (file_name, IOS_M_RDONLY);

The `open' built-in function returns a file descriptor that can be
subsequently used in mapping operations.  If the provided file name
doesn't identify a file, or if the file can't be read for whatever
reason, an E_io exception is raised.  Note how the exception is handled
in the `catch' block, emitting an appropriate diagnostic message and
exiting with an error status.

Once the ELF file is open for reading, we map an Elf64_File on it, at
the expected offset (zero bytes from the beginning of the file):

   defvar elf = Elf64_File @ fd : 0#B;

If the file doesn't contain valid ELF data, this map will fail and raise
an E_constraint exception.  Again, the `catch' block handles this
situation.

At this point the variable `elf' contains an ELF file.  Since we want to
extract the sections contained in the file, we need to somehow iterate
on them.  The section header table is available in elf.shdr.  A
for-in-where loop is used to iterate on all the headers, skipping the
"null" ELF sections which are always empty, and are characterized by a
shdr.sh_type of 0.  An inner conditional filters out sections whose name
do not match the provided name in the command line, if it was specified
at all.

For each "matching" section we then save its contents in a file named
after the input ELF file, by calling a function `save', which is
provided by poke:

   save :ios elf'ios :file file_name + sname
        :from shdr.sh_offset :size shdr.sh_size;

This is exactly what we would have written at the poke REPL! (modulus
trailing semicolon).  How is this supposed to work?  Thing is, GNU poke
commands are implemented as Poke functions.  Let's consider `save', for
example.  It is defined as a function having the following prototype:

   defun save = (int ios = get_ios,
                 string file = "",
                 off64 from = 0#B,
                 off64 size = 0#B,
                 int append = 0,
                 int verbose = 0) void:
   { ... }

Once a Poke function is defined in the environment, it becomes available
as such.  Therefore, in a poke session we could call it like:

   (poke) save (get_ios, "filename", 0#B, 12#B, 0, 1)
    
However, this is cumbersome and error prone.  To begin with, we should
remember the name, position and nature of each argument accepted by the
command.  What is even more annoying, we are forced to provide explicit
values for them, like in the example above we have to pass the current
IOS (the default), and 0 for append (the default) just to being able to
set `verbose'.

To ease commanding poke, the Poke language supports an alternative
syntax to call functions, in which the function arguments are referred
by name, can be given in any order, and can be omitted.  The command
above can be thus written like:

   (poke) save :from 0#B :size 12#B :verbose 1

This syntax is mostly intended to be used interactively, but nothing
prevents to use it in Poke programs and scripts whenever it is deemed
appropriate, like we did in elfextractor.  We could of course have used
the more conventional syntax:

   if (section_name == "" || sname == section_name)
     save (elf'ios, file_name + sname,
           shdr.sh_offset, shdr.sh_size, 0, 0);

What style to use is certainly a matter of taste.

Anyhow, once the sections have been written out, the file descriptor is
closed and the program exits with the default status, which is success.
Should the `save' function find any problem saving the data, such as a
full disk, not enough permissions or the like, exceptions will be
raised, caught and maybe handled by our `catch' block.

And this is it!  The complete program is 44 lines long.  This is a good
example that shows how, given a pickle providing a reasonable
description of some binary-oriented format (ELF in this case) poke can
be leveraged to achieve a lot in a very concise way, free from the many
details involved in the encoding, reading and writing of binary data.

Happy poking! :)