Re: [PATCH] gnu: Add dlib.

[Marius Bakke]

Ben Woodcroft <address@hidden> writes:

> On 16/08/16 08:28, Leo Famulari wrote:
>> On Mon, Aug 15, 2016 at 09:29:11PM +0100, Marius Bakke wrote:
>>> Leo Famulari <address@hidden> writes:
>>>> Running test_empirical_kernel_map / phase `check' failed after 2043.7 
>>>> seconds
>>> Strange, I've built this dozens of times now with no test failures
>>> (GuixSD on amd64). Are you memory constrained by any chance? Any OOM or
>>> segfaults? I don't think the tests have taken more than ~15 minutes to
>>> run either, once they are built.
>> I don't think I'm particularly resource constrained. I have ~7 GiB RAM
>> available to build with, plenty of disk space, and the processor is this
>> thing:
>>
>> http://ark.intel.com/products/64896/Intel-Core-i5-3320M-Processor-3M-Cache-up-to-3_30-GHz
>>
>> It appears to exit cleanly (no segfault or OOM).
>>
>> It fails the same test on this machine as well as a slower x86_64
>> (Thinkpad x200s). One machine is Guix / Debian, the other is GuixSD.
> I built this patch on a x86_64 server with 40 cores and 256G of memory 
> twice, here's what I got:
>
> Running test_one_vs_all_trainer \
>
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!! TEST FAILED: test_one_vs_all_trainer 
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
> Failure message from test:
>
> Error occurred at line 141.
> Error occurred in file 
> /tmp/nix-build-dlib-19.1.drv-0/dlib-19.1/dlib/test/one_vs_all_trainer.cpp.
> Failing expression was ans == res.
> res:
> 60  0  0
>   0 69  1
>   0  0 80
>
>
>
>
> Testing Finished
> Total number of individual testing statements executed: 471078543
> Number of failed tests: 1
> Number of passed tests: 133

I can reproduce Leos error on a foreign distro (Ubuntu 16.04, 32GiB RAM)
and see this in dmesg:

[497259.758549] dtest[27102]: segfault at b50 ip 00007ffff68270e0 sp 
00007fffffffa978 error 4 in libopenblasp-r0.2.15.so[7ffff5715000+18c6000]

That memory address is 128 TiB. Though it consistently builds fine on a
64GiB GuixSD machine as verified with `guix build --check`.

I initially made this package on a foreign distro without "lapack" in
inputs and have verified that dropping LAPACK makes the tests pass.

I also found some other optional dependencies after digging around the
source, as well as a recommendation to disable/enable asserts:

http://dlib.net/dlib/config.h.html

Ben, Leo: Can you try the following patch and see if that works for you?

>From 7cc190763e1312536fe3ac9cd91ea3756cf66b80 Mon Sep 17 00:00:00 2001
From: Marius Bakke <address@hidden>
Date: Sat, 13 Aug 2016 11:26:10 +0100
Subject: [PATCH] gnu: Add dlib.

* gnu/packages/machine-learning.scm (dlib): New variable.
---
 gnu/packages/machine-learning.scm | 63 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 62 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/machine-learning.scm 
b/gnu/packages/machine-learning.scm
index f96672c..ebf3307 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015, 2016 Ricardo Wurmus <address@hidden>
 ;;; Copyright © 2016 Efraim Flashner <address@hidden>
+;;; Copyright © 2016 Marius Bakke <address@hidden>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -27,18 +28,21 @@
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system r)
   #:use-module (gnu packages)
+  #:use-module (gnu packages algebra)
   #:use-module (gnu packages autotools)
   #:use-module (gnu packages boost)
   #:use-module (gnu packages compression)
   #:use-module (gnu packages dejagnu)
   #:use-module (gnu packages gcc)
+  #:use-module (gnu packages image)
   #:use-module (gnu packages maths)
   #:use-module (gnu packages perl)
   #:use-module (gnu packages pkg-config)
   #:use-module (gnu packages python)
   #:use-module (gnu packages statistics)
   #:use-module (gnu packages swig)
-  #:use-module (gnu packages xml))
+  #:use-module (gnu packages xml)
+  #:use-module (gnu packages xorg))
 
 (define-public libsvm
   (package
@@ -467,3 +471,60 @@ geometric models.")
      "This package provides functions for feed-forward neural networks with a
 single hidden layer, and for multinomial log-linear models.")
     (license (list license:gpl2+ license:gpl3+))))
+
+(define-public dlib
+  (package
+    (name "dlib")
+    (version "19.1")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append
+                    "http://dlib.net/files/dlib-"; version ".tar.bz2"))
+              (sha256
+               (base32
+                "0p2pvcdalc6jhb6r99ybvjd9x74sclr0ngswdg9j2xl5pj7knbr4"))
+              (modules '((guix build utils)))
+              (snippet
+               '(begin
+                  ;; Delete ~13MB of bundled dependencies.
+                  (delete-file-recursively "dlib/external")
+                  (delete-file-recursively "docs/dlib/external")))))
+    (build-system cmake-build-system)
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'disable-asserts
+           (lambda _
+             ;; config.h recommends explicitly enabling or disabling asserts
+             ;; when building as a shared library. By default neither is set.
+             (substitute* "dlib/config.h"
+               (("^//#define DLIB_DISABLE_ASSERTS") "#define 
DLIB_DISABLE_ASSERTS"))))
+         (replace 'check
+           (lambda _
+             ;; No test target, so we build and run the unit tests here.
+             (let ((test-dir (string-append "../dlib-" ,version 
"/dlib/test/build")))
+               (mkdir-p test-dir)
+               (with-directory-excursion test-dir
+                 (and (zero? (system* "cmake" ".."))
+                      (zero? (system* "cmake" "--build" "." "--config" 
"Release"))
+                      (zero? (system* "./dtest" "--runall"))))))))))
+    (native-inputs
+     `(("pkg-config" ,pkg-config)))
+    (inputs
+     `(("fftw" ,fftw)
+       ("giflib" ,giflib)
+       ;("lapack" ,lapack) XXX lapack here causes test failures in some setups.
+       ("libjpeg" ,libjpeg)
+       ("libpng" ,libpng)
+       ("libx11" ,libx11)
+       ("openblas" ,openblas)
+       ("zlib" ,zlib)))
+    (synopsis
+     "Toolkit for making machine learning and data analysis applications in 
C++")
+    (description
+     "Dlib is a modern C++ toolkit containing machine learning algorithms and
+tools.  It is used in both industry and academia in a wide range of domains
+including robotics, embedded devices, mobile phones, and large high performance
+computing environments.")
+    (home-page "http://dlib.net";)
+    (license license:boost1.0)))
-- 
2.9.2