parallel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

calling python script from exported bash function, module import fails


From: Benjamin Leutner
Subject: calling python script from exported bash function, module import fails
Date: Tue, 20 Dec 2016 19:16:20 +0100
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1

Hi everyone,

I'm not sure whether GNU-parallel or python (or me) is at fault here, hence I decided to start with this list.

I am trying to run a bash function both locally and on remote machines.
The function calls a python script, which then fails to load additional modules. Python finds the right modules but fails to load them with "IOError: [Errno 10] No child processes"

In order to run it on the remote machines I export the function and pass it on with --env. However, this breaks running my program locally. Without --env it runs fine locally, but fails of course on remote.

The issue seems to arise from the nestedness level, because this is the only difference in the environment between the two (except for PID and tmp). Without --env, $SHLVL is 2, while with passing it it is 3

Here is a small example how exporting --env breaks it on the local machine (running on python 2.7):

pip2 install tables

function FUN {
echo -e "\n----- Try number: ${1}  -----"
echo "Host: $(hostname), Level:  $SHLVL"
python -c "
from tables import *
print 'import successful'"
}
export -f FUN

# without env
parallel 'FUN {}' ::: 1

# with env for function
parallel --env FUN 'FUN {}' ::: 2

# with env for everything
parallel --env _ 'FUN {}' ::: 3

For the first try 1) I get:
----- Try number: 1  -----
Host: wfhn080, Level:  2
import successful

While for 2) and 3) I get
----- Try number: 2  -----
Host: myMachine, Level:  3
Traceback (most recent call last):
  File "<string>", line 2, in <module>
File "/usr/local/lib/python2.7/dist-packages/tables/__init__.py", line 128, in <module>
    from .file import File, open_file, copy_file
File "/usr/local/lib/python2.7/dist-packages/tables/file.py", line 32, in <module>
    import numexpr
File "/usr/local/lib/python2.7/dist-packages/numexpr/__init__.py", line 42, in <module>
    from numexpr.tests import test, print_versions
File "/usr/local/lib/python2.7/dist-packages/numexpr/tests/__init__.py", line 11, in <module>
    from numexpr.tests.test_numexpr import test, print_versions
File "/usr/local/lib/python2.7/dist-packages/numexpr/tests/test_numexpr.py", line 293, in <module>
    class test_evaluate(TestCase):
File "/usr/local/lib/python2.7/dist-packages/numexpr/tests/test_numexpr.py", line 450, in test_evaluate
    if 'sparc' not in platform.machine():
  File "/usr/lib/python2.7/platform.py", line 1323, in machine
    return uname()[4]
  File "/usr/lib/python2.7/platform.py", line 1253, in uname
    processor = _syscmd_uname('-p','')
  File "/usr/lib/python2.7/platform.py", line 988, in _syscmd_uname
    rc = f.close()
IOError: [Errno 10] No child processes


What am I doing wrong here?
Best regards,
Benjamin




reply via email to

[Prev in Thread] Current Thread [Next in Thread]