Re: G expressions

[Amirouche Boubekki]

2014-04-28 9:43 GMT+02:00 Ludovic Courtès <address@hidden>:
# Hello!

# 1. The Problem
# ===============

Héllo!

I was going to learn a bit of emacs lisp then... a duck got in my way.

# There’s a recurrent problem that we need to communicate the file name of
# store items to Scheme code that is going to live in another process:
# expressions to build a derivation, Scheme files that are to be loaded by
# other processes, etc.

This seems to be a referential transparency problem. Probably the
Funarg problem.

Quoting http://en.wikipedia.org/wiki/Funarg_problem:

:w: The difficulty only arises if the body of a [[nested function]]
refers directly
:w: (i.e., not via argument passing) to identifiers defined in the environment
:w: in which the function is defined, but not in the environment of the function
:w: call.
:w:   <ref>[http://portal.acm.org/citation.cfm?id=1093411
:w:   ''The function of FUNCTION in LISP or why the FUNARG problem should be
:w:   called the environment problem'',
:w:   </ref>
:w:   by Joel Moses, in: ACM SIGSAM Bulletin 17 (July 1970), pp. 13-27]</ref>
:w: [Two] standard resolutions are to either forbid such references or to
:w: create [[closure (computer science)|closure]]s.
:w:   <ref>[http://portal.acm.org/citation.cfm?id=1093420.1093422
:w:   ''A proposed solution to the FUNARG problem'',
:w:   by Erik Sandewall, in: ACM SIGSAM Bulletin 17 (Jan. 1971), pp. 29-42]
:w:   </ref>

It's a scoping issue due to the context of where the call is really done. In
Guile arguments are passed as value. So, the procedure loose the context of
the value. It's just a "thunk" or log.

# This had been partly addressed by having ‘build-expression->derivation’
# install two global variables in the build code: ‘%build-inputs’ and
# ‘%outputs’.

I don't like globals but I have no good reasons, except that it "pollutes"
the environment. IRL, it's a problem when you want to move to "higher
level"s.

# However, for generated Scheme files (as in (gnu system) and related
# modules), there’s no such mechanism.  Thus, either we use something
# like:

#   (mlet %store-monad ((ls (package-file coreutils "bin/ls")))
#     (text-file "foo" (object->string `(system* ,ls))))

cf. https://www.gnu.org/software/guix/manual/guix.html#The-Store-Monad

# but then the problem is that the resulting text file doesn’t hold a
# reference to Coreutils, which is wrong.  Or, we do something like:

#   (text-file* "foo" "(system* \"" coreutils "/bin/ls\")")

# The resulting file does have a reference to Coreutils, but the approach
# obviously sucks.

# Besides, ‘%build-inputs’ is not particularly elegant either, and it’s
# error-prone (if you refer to an input by the wrong name, you only notice
# when building, with an obscure wrong-type-arg error.)  That’s been OK as
# it’s only used occasionally in package recipes.

I can not make justice to the style (it's ugly!). I don't have enough experience
in scheme/fp/guile to distinguish between a good solution and workaround
-- something. AFAIK, both process don't share the same memory, so it's even
more difficult to share data, in the broad sens. In this case, writring sexp
directly allows to keep some of the information, but not all of it, because
the values are evaluated.

This evaluation thing troubles me a lot. That's one of the reasons I started
scheme: macros. I did not write, consciously, a single macro, yet.

.. python:: https://github.com/lihaoyi/macropy

In python, there is a *run* *time*. That is all. Some time you take care of
*import* *time*. But then you forget about it. So far, my understanding
of scheme led me to think _that anything anywhere could be something else in
reality_. In particular, it's possible to provide procedures to the parser
called macros that are evaluated when sexps are parsed.

In the particular case of guix why not send to just data? So that
context can not be
/overriden/.

Like I said, I don't understand fully macros nor monads and I stumbled upon
http://adit.io/posts/2013-04-17-functors,_applicatives,_and_monads_in_pictures.html

I understand that the article explains what does it mean that Haskell is typed.
I understand that haskell has "higher order" procedures, that are not readily
available in imperative languages plus some extras for being self aware of that
fact.

In fact, what /functors/, /applicatives/ and /monads/ do, is possible in all
languages. If they are not used often so far, it's because people were missing
something... Those /higher level/ procedures that can be found in Haskell
are *forbidden* in imperative languages: GOTO, monkey-patching, metaclasses or
grossly ineficient like callbacks for dealing with asynchronous behaviors and
otherwise.

.. seealso:: (url [Callbacks as our Generations' Go To Statement]
                   http://tirania.org/blog/archive/2013/Aug-15.html)

IMO the article explains different *patterns* that dynamically change
the runtime.
It's like a bestiary of some sort, see
http://en.wikibooks.org/wiki/Haskell/Understanding_arrows
In the end it's just a procedure call, a combinatory logic, to make something
that looked like http://community.schemewiki.org/?amb, more discrete
(and more human?).
I don't say there is no interest in this approach(!).

.. seealso:: (url [Combinatorial Generation Using Coroutines With
Examples in Python]
                   https://news.ycombinator.com/item?id=7655488)

At least the above, lead me to (quote gexp).

# 2. G Expressions
# =================

# “G-expressions”, or gexps, are meant to address these two issues
# mentioned above:

Passing parameters by *intent* instead of by value?

  # 1. Replacing a reference to a package or derivation by its output file
  #    name;

cf. https://www.gnu.org/software/guix/manual/guix.html#Derivations

  # 2. Keeping track of the derivations it refers to.

I interpret this as:

   «It must be possible to (back)trace anywhere, anytime... anything»

Out of the context of guix, that could be better or worse misunderstanding :)

# In addition, the substitution in (1) must be done lazily, so that you
# get the output file name for the ‘%current-system’ value when the gexp
# is used.

# The ‘gexp’ form is essentially like ‘quasiquote’, with ‘ungexp’ as the
# counterpart of ‘unquote’:

#   (gexp (system* (string-append (ungexp coreutils) "/bin/ls")))

What about the following:

;;  (quote (system* (string-append (coreutils "/bin/ls"))))

I don't understand the difference with classic macro. Is it a problem of
nesting? Something like that isn't possible:

;;  (quote (system* (string-append (coreutils
(:somekind-of-sexp-parser-hook-eval: program-path))))

# That gexp can then be passed to ‘gexp->file’, which returns a derivation
# that builds a file containing:

It is passed to the server ie. the guix builder?

#   (system* (string-append "/gnu/store/…" "/bin/ls"))

# And it can also be used to describe derivation builds:

#   (gexp->derivation "foo"
#                     (gexp
#                       (begin
#                         (mkdir (ungexp output))
#                         (chdir (ungexp output))
#                         (symlink (ungexp coreutils) "cu"))))

# Note that we don’t need #:inputs here, because the gexp embeds that
# info.  So eventually, we could even get rid of the label in the ‘inputs’
# field of packages (not a priority, though.)

# We could use some sugar to make it more concise (suggestions welcome):

  # (gexp->derivation "foo"
  #                   #~(begin
  #                       (mkdir #$output)
  #                       (chdir #$output)
  #                       (symlink #$coreutils "cu")))

- It's possible to parse variable names in macros?
- I don't like #~ or #$ I prefer an infix notation with *words* as
  it is more readable:

  ;; (gexp (gexp.begin (make.directory ungexp.output)
                       (change.directory ungexp.output)
                       (symlink ungexp.coreutils "cu")))

I added some fioritures that you will probably find pointless. IMHO
it's significant.

# 3. Conclusion
# ==============

# I think gexps can be helpful to serialize Scheme code that refers to
# packages/derivations.  Preliminary work is in ‘wip-gexp’.
# What do you think?

It's seems to me that it's yet another /upward & downard funarg/.

# 4. Related Work :-)
# ====================

# HOP has a quotation mechanism to introduce client-side code in server
# code and vice-versa:

#   (<BUTTON>
#     :onclick ~(with-hop ($foo)))

# Nix uses string interpolation to keep achieve what we do:

#   script = ''${coreutils}/bin/ls'';

# Here ${coreutils} is replaced by the output file name, and the resulting
# string internally has additional info about the reference to that
# derivation.

- This can be the job of guile reader?
- It exists also in native javascript framework like react
  http://facebook.github.io/react/#helloExample

  .. seealso:: http://www.quora.com/What-does-Facebook-use-OCaml-for

Quoting your last message:

# I’ve been experimenting with gexps in all the contexts with it seemed
# “obviously” useful and in some other places under gnu/{system,services}
# (see the commits that say “Rewrite using gexps” in wip-gexp.)

cf. (url [Add (guix gexp).]
         
http://git.savannah.gnu.org/cgit/guix.git/commit/?id=20ed77c71e41b41911cf152f8feae7e9ac5f9f6b)

- I need to learn macros. The decyphering route is langweileich.
- I did not read all the code, it seems like ``ungexp`` is
``:sexp-parser-hook-eval:``.

=> if that's it, that was the exact problem I have (had?). What I'd like to do
is "describe an intent" more precisly an intent that is bound to an unknown
context, a lot of /Maybe/ in you prefer. This reflexion grew out of:

- an api to traverse graphs in Python based on Gremlin DSL
https://bitbucket.org/amirouche/omak-unmaintained
- declarative programming in general (cf. Prolog)
- the reactive programming culprit (cf. reactconf)

guile-2d does reactive programming, I'm quoting David Thompson the creator:

// Functional reactive programming (FRP) provides a way to describe
// time-varying values and their relationships using a functional
// and declarative programming style. To understand what this means,
// let's investigate a typical variable definition in Scheme [...]

Later there is an example of guile-2d FRP ie. Functional Reactive Programming:

// ;;
// ;; (define-signal position
// ;;   (signal-fold v+ (vector2 320 240)
// ;;                (signal-map (lambda (v)
// ;;                            (vscale v 4))
// ;;                           (signal-sample game-agenda 1 key-arrows))))
// ;;

In the project I'm working on, using guilen I would have written in as:

;; (voxel player position signal shift with (* 4 (voxel agenda sample
(voxel keyboard arrows)) 1))

or with somekind of macro

;; (player.position.signal.shift.with (4 * (agenda.sample keyboard.arrows 1)))

I'm calling the thing voxel in reference to "scalar field"s used in some 3D
engines like the one described in [1] and [2]. Current s/code/draft is
attached to
this mail.

# that carries all the necessary dependency info.

Regarding the commentary in guix/gexp.scm, It can be improved, by describing
what it does in general and then explain the handy procedure useful only in the
context of (guix). I commenting the commentary below:

# ;;; This module implements "G-expressions", or "gexps".  Gexps are like
# ;;; S-expressions (sexps), with two differences:
# ;;;
# ;;;   1. References (un-quotations) to derivations or packages in a gexp are
# ;;;      replaced by the corresponding output file name;

In the point of view of the caller, the derivation will refer to itself,
the locally bound object.

# ;;;
# ;;;   2. Gexps embed information about the derivations they refer to.
# ;;;

In "derivation" word, there the notion of palpable future (cf.
https://en.wiktionary.org/wiki/palpable),
things that will happen. Indeed the derivation declares the expected outcome
of the build. (ungexp) kind of fights this global movement induced by the
Guile/Scheme environment...

# ;;; Gexps make it easy to write to files Scheme code that refers to store
# ;;; items, or to write Scheme code to build derivations.

Outside the context of guix, I wouldn't understand.

;;;
;;; Code:

If I'm not clear (gexp) is very useful to me, I will use it. I am curious to
know how it relates to http://wingolog.org/archives/2014/03/17/stack-overflow

[1] Jump to AI http://www.voxelquest.com/1/post/2014/03/march-mega-update.html
    (via https://hn.algolia.com/#!/story/sort_by_date/0/voxel)

[2] 
http://www.simonsfoundation.org/quanta/20140416-times-arrow-traced-to-quantum-source/

voxels-and-the-arrow-of-time.scm
Description: Text Data