Re: [Bug-apl] strange behavior of --

[Xiao-Yong Jin]

In terms of how interpreters should do with multiple arguments in the #! line, 
perl's behavior is useful.
The following is from 'perldoc perlrun'

    The "#!" line is always examined for switches as the line is being parsed.
    Thus, if you're on a machine that allows only one argument with the "#!"
    line, or worse, doesn't even recognize the "#!" line, you still can get
    consistent switch behaviour regardless of how Perl was invoked, even if
    -x was used to find the beginning of the program.

You may also want to read the rest of the document.

> On Feb 6, 2017, at 2:06 PM, Juergen Sauermann <address@hidden> wrote:
> 
> Hi Xiao-Yong,
> 
> thanks a lot for this explanation!
> 
> /// Jürgen
> 
> 
> On 02/06/2017 08:25 PM, Xiao-Yong Jin wrote:
>> I would guess he is using zsh instead of bash.
>> zsh allows interpreter after #! without an absolute path, irrelevant of the 
>> OS kernel.
>> 
>> Try running the following script under zsh:
>> 
>> ---- SCRIPT BEGIN ----
>> #!bash
>> echo 'this will fail'
>> ---- SCRIPT END ----
>> 
>> You will get
>> 
>> ---- OUTPUT BEGIN ----
>> bash: echo 'this will fail'
>> : No such file or directory
>> ---- OUTPUT END ----
>> 
>> Zsh under both Linux and OSX should give this error, because zsh is making 
>> the syscall with its own interpretation of the shebang line.
>> 
>> On the other hand, with the absolute path, you get the standard OSX kernel 
>> handling the syscall.
>> Google gives a detailed list of behaviors on a webpage
>> http://www.in-ulm.de/~mascheck/various/shebang/
>> Your code supporting argv[1] with a concatenated string like '-l 37 --script 
>> --' may not be the best practice, IMHO, and should be avoided.
>> 
>> Putting more than 1 argument on the shebang line is calling for troubles if 
>> you want cross OS/kernel support.
>> 
>>> On Feb 6, 2017, at 12:43 PM, Juergen Sauermann <address@hidden> wrote:
>>> 
>>> Hi Alexey,
>>> 
>>> very odd indeed. It very much looks like OSX is starting apl but then not 
>>> piping
>>> the subsequent lines of the script into APL. As if they are opening the 
>>> script with
>>> popen() instead of execve(). Its probably more a problem of the shell than 
>>> of
>>> the entire OS.
>>> 
>>> I would also believe that after starting the script you are in immediate 
>>> execution
>>> mode in GNU APL but with input echo off (apl was started with --script).
>>> 
>>> I suppose if you replace '--' by '-f' in your first script line (without 
>>> mentioning the
>>> script name as I proposed earlier), then it should work.
>>> 
>>> BTW a non-absolute path does not work in GNU/APL, which shows how different 
>>> the
>>> platforms are in this area.
>>> 
>>> /// Jürgen
>>> 
>>> 
>>> On 02/06/2017 06:45 PM, Alexey Veretennikov wrote:
>>>> Hi!
>>>> 
>>>> Finally I've down to something.
>>>> So the difference is whether I specify full path to apl in a file
>>>> header.
>>>> Consider:
>>>> head -1 aaa.apl
>>>> #!apl -l 37 --script --
>>>> 
>>>> sizeof(Svar_record) is    328
>>>> sizeof(Svar_partner) is   28
>>>> increasing rlimit RLIMIT_NPROC from 709 to infinity
>>>> 
>>>> initializing paths from argv[0] = apl
>>>> initializing paths from  $PATH = 
>>>> /Users/alexeyv/Applications:/Users/alexeyv/Development/stm32tools:/Users/alexeyv/Development/arm-eabi-toolchain/arm-cs-tools-2012.03-56-e3f4013-20130413/bin:/Users/alexeyv/Development/gapl:/Users/alexeyv/.cabal/bin:/Users/alexeyv/Development/gradle-2.3/bin:/Users/alexeyv/Development/groovy-2.4.3/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin:/Library/TeX/texbin
>>>> APL_bin_path is: /Users/alexeyv/Development/gapl
>>>> APL_bin_name is: apl
>>>> Reading config file 
>>>> /Users/alexeyv/Development/gapl/etc/gnu-apl.d/preferences ...
>>>> Reading config file /Users/alexeyv/.gnu-apl/preferences ...
>>>> Reading config file 
>>>> /Users/alexeyv/Development/gapl/etc/gnu-apl.d/parallel_thresholds ...
>>>> Not reading config file /Users/alexeyv/.config/gnu-apl/parallel_thresholds 
>>>> (not found/readable)
>>>> 0 input files:
>>>> using ANSI terminal output ESC sequences (or those configured in your 
>>>> preferences file(s))
>>>> using ANSI terminal input ESC sequences(or those configured in your 
>>>> preferences file(s))
>>>> Using TCP socket towards APserver...
>>>> connecting to 127.0.0.1 TCP port 16366 failed.
>>>>     (the first ::connect() to APserver is expected to fail, unless
>>>>      APserver was started manually)
>>>> Starting a new APserver listening on 127.0.0.1 TCP port 16366
>>>> Found /Users/alexeyv/Development/gapl/APserver
>>>> Starting /Users/alexeyv/Development/gapl/APserver --port 16366 --auto...
>>>> 
>>>> connecting to 127.0.0.1 TCP port 16366 failed.
>>>>     (::connect() should succeed eventually. This was attempt 1 of 5)
>>>> connecting to 127.0.0.1 TCP port 16366 failed.
>>>>     (::connect() should succeed eventually. This was attempt 2 of 5)
>>>> connecting to 127.0.0.1 TCP port 16366 failed.
>>>>     (::connect() should succeed eventually. This was attempt 3 of 5)
>>>> connecting to 127.0.0.1 TCP port 16366 failed.
>>>> ::connect() to supposedly existing APserver failed: Invalid argument
>>>> PID is 14426
>>>> argc: 3
>>>>   argv[0]: 'apl'
>>>>   argv[1]: '-l 37 --script --'
>>>>   argv[2]: './aaa.apl'
>>>> stdin is: OPEN
>>>> uprefs.user_do_svars:   1
>>>> uprefs.system_do_svars: 1
>>>> uprefs.requested_id:    0
>>>> uprefs.requested_par:   0
>>>> Svar_DB not connected in Svar_DB::is_registered_id()
>>>> id.proc: 1001 at ProcessorID.cc:77
>>>> Processor ID was completely initialized: 1001:0:0
>>>> system_do_svars is: 1
>>>> ┌→──────────────────────────────────────────┐
>>>> │┌→──┐ ┌→─┐ ┌→─┐ ┌→───────┐ ┌→─┐ ┌→────────┐│
>>>> ││apl│ │-l│ │37│ │--script│ │--│ │./aaa.apl││
>>>> │└───┘ └──┘ └──┘ └────────┘ └──┘ └─────────┘│
>>>> └∊──────────────────────────────────────────┘
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> However:
>>>> 
>>>> 
>>>> head -1 bbb.apl
>>>> #!/Users/alexeyv/Development/gapl/apl -l 37 --script --
>>>> 
>>>> 
>>>> 
>>>> ./bbb.apl
>>>> sizeof(Svar_record) is    328
>>>> sizeof(Svar_partner) is   28
>>>> increasing rlimit RLIMIT_NPROC from 709 to infinity
>>>> 
>>>> initializing paths from argv[0] = /Users/alexeyv/Development/gapl/apl
>>>> initializing paths from  $PWD = /Users/alexeyv/Sources/apl
>>>> APL_bin_path is: /Users/alexeyv/Development/gapl
>>>> APL_bin_name is: apl
>>>> Reading config file 
>>>> /Users/alexeyv/Development/gapl/etc/gnu-apl.d/preferences ...
>>>> Reading config file /Users/alexeyv/.gnu-apl/preferences ...
>>>> Reading config file 
>>>> /Users/alexeyv/Development/gapl/etc/gnu-apl.d/parallel_thresholds ...
>>>> Not reading config file /Users/alexeyv/.config/gnu-apl/parallel_thresholds 
>>>> (not found/readable)
>>>> 0 input files:
>>>> using ANSI terminal output ESC sequences (or those configured in your 
>>>> preferences file(s))
>>>> using ANSI terminal input ESC sequences(or those configured in your 
>>>> preferences file(s))
>>>> Using TCP socket towards APserver...
>>>> connected to APserver, socket is 4
>>>> using Svar_DB on APserver!
>>>> PID is 14455
>>>> argc: 6
>>>>   argv[0]: '/Users/alexeyv/Development/gapl/apl'
>>>>   argv[1]: '-l'
>>>>   argv[2]: '37'
>>>>   argv[3]: '--script'
>>>>   argv[4]: '--'
>>>>   argv[5]: './bbb.apl'
>>>> stdin is: OPEN
>>>> uprefs.user_do_svars:   1
>>>> uprefs.system_do_svars: 1
>>>> uprefs.requested_id:    0
>>>> uprefs.requested_par:   0
>>>> id.proc: 1001 at ProcessorID.cc:77
>>>> Processor ID was completely initialized: 1001:0:0
>>>> system_do_svars is: 1
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> - Here it hangs.
>>>> Here you see what depending on whether we have a full path in the first
>>>> line command line arguments interpreted differently. Very odd.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> Juergen Sauermann
>>>> <address@hidden>
>>>>  writes:
>>>> 
>>>> 
>>>>> Hi Alexey,
>>>>> 
>>>>> but then everything is just fine, isn't it? I believe in an
>>>>> earlier post you said that in OSX you don't see any output and have to 
>>>>> type )OFF
>>>>> blindly (which suggest that you didn't have a working stdin)?
>>>>> 
>>>>> /// Jürgen
>>>>> 
>>>>> On 02/06/2017 05:53 PM, Alexey Veretennikov wrote:
>>>>> 
>>>>>  Hi,
>>>>> 
>>>>> Here are the results:
>>>>> 
>>>>> sizeof(Svar_record) is    328
>>>>> sizeof(Svar_partner) is   28
>>>>> increasing rlimit RLIMIT_NPROC from 709 to infinity
>>>>> 
>>>>> initializing paths from argv[0] = apl
>>>>> initializing paths from  $PATH = 
>>>>> /Users/alexeyv/Applications:/Users/alexeyv/Development/stm32tools:/Users/alexeyv/Development/arm-eabi-toolchain/arm-cs-tools-2012.03-56-e3f4013-20130413/bin:/Users/alexeyv/Development/gapl:/Users/alexeyv/.cabal/bin:/Users/alexeyv/Development/gradle-2.3/bin:/Users/alexeyv/Development/groovy-2.4.3/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin:/Library/TeX/texbin
>>>>> APL_bin_path is: /Users/alexeyv/Development/gapl
>>>>> APL_bin_name is: apl
>>>>> Reading config file 
>>>>> /Users/alexeyv/Development/gapl/etc/gnu-apl.d/preferences ...
>>>>> Reading config file /Users/alexeyv/.gnu-apl/preferences ...
>>>>> Reading config file 
>>>>> /Users/alexeyv/Development/gapl/etc/gnu-apl.d/parallel_thresholds ...
>>>>> Not reading config file 
>>>>> /Users/alexeyv/.config/gnu-apl/parallel_thresholds (not found/readable)
>>>>> 0 input files:
>>>>> using ANSI terminal output ESC sequences (or those configured in your 
>>>>> preferences file(s))
>>>>> using ANSI terminal input ESC sequences(or those configured in your 
>>>>> preferences file(s))
>>>>> Using TCP socket towards APserver...
>>>>> connected to APserver, socket is 4
>>>>> using Svar_DB on APserver!
>>>>> PID is 13743
>>>>> argc: 3
>>>>>   argv[0]: 'apl'
>>>>>   argv[1]: '-l 37 --script --'
>>>>>   argv[2]: './aaa.apl'
>>>>> stdin is: OPEN
>>>>> uprefs.user_do_svars:   1
>>>>> uprefs.system_do_svars: 1
>>>>> uprefs.requested_id:    0
>>>>> uprefs.requested_par:   0
>>>>> id.proc: 1001 at ProcessorID.cc:77
>>>>> Processor ID was completely initialized: 1001:0:0
>>>>> system_do_svars is: 1
>>>>> ┌→──────────────────────────────────────────┐
>>>>> │┌→──┐ ┌→─┐ ┌→─┐ ┌→───────┐ ┌→─┐ ┌→────────┐│
>>>>> ││apl│ │-l│ │37│ │--script│ │--│ │./aaa.apl││
>>>>> │└───┘ └──┘ └──┘ └────────┘ └──┘ └─────────┘│
>>>>> └∊──────────────────────────────────────────┘
>>>>> 
>>>>> 
>>>>> Juergen Sauermann
>>>>> <address@hidden>
>>>>>  writes:
>>>>> 
>>>>>  Hi,
>>>>> 
>>>>> i have added a check if stdin is open when GNU APL starts, SVN 881.
>>>>> 
>>>>> If you start the following script:
>>>>> 
>>>>> #!/usr/local/bin/apl -l 37 --script --
>>>>> 
>>>>> ]BOXING ¯8
>>>>> ⎕ARG
>>>>> )off
>>>>> 
>>>>> Then we can see if stdin is open on OSX and how apl is being called in 
>>>>> OSX.
>>>>> 
>>>>> /// Jürgen
>>>>> 
>>>>> On 02/06/2017 11:21 AM, Juergen Sauermann wrote:
>>>>> 
>>>>>  Hi Alexey,
>>>>> 
>>>>>  yes. I changed it recently to fix the '--' issue.
>>>>> 
>>>>>  A GNU APL script assumes that it was called by execve(). The 
>>>>> expand_argv() function
>>>>>  "undoes" the behavior of execve(), which lumps together all arguments on 
>>>>> the first script line
>>>>>  into argv[1] of GNU APL's main(argc, argv) functions. In that process 
>>>>> the filename of the script
>>>>>  gets lost (or may not even exist, e.g. in a pipe) so -f could be used to 
>>>>> tell the script where to
>>>>>  fetch its input. This is because execve() had already closed the file 
>>>>> descriptor before it called
>>>>>  apl, so apl has no stdin when it starts. With -f you point apl to 
>>>>> re-open the input file again.
>>>>> 
>>>>>  /// Jürgen
>>>>> 
>>>>>  On 02/05/2017 09:24 PM, Alexey Veretennikov wrote:
>>>>> 
>>>>>  Ok, as I understand I need to take a look at
>>>>> UserPreferences::expand_argv
>>>>> and
>>>>> UserPreferences::is_APL_script
>>>>> 
>>>>> correct?
>>>>> 
>>>>> Juergen Sauermann
>>>>> <address@hidden>
>>>>>  writes:
>>>>> 
>>>>>  Hi,
>>>>> 
>>>>> yes, most of this trouble is caused by how execve() works, which is quite 
>>>>> different
>>>>> on different platforms. And it happens before apl is being called so I 
>>>>> cant do much
>>>>> about it.
>>>>> 
>>>>> Sometimes it helps to start apl with -f so that the interpreter knows 
>>>>> where to fetch
>>>>> its input, like:
>>>>> 
>>>>> #!/usr/local/bin/apl -f /home/eedjsa/tmp/script --script --
>>>>> 
>>>>> /// Jürgen
>>>>> 
>>>>> On 02/04/2017 02:00 PM, Alexey Veretennikov wrote:
>>>>> 
>>>>>  Hi Juergen,
>>>>> 
>>>>> Something is apparently strange on OSX(?). With the latest version
>>>>> when I run the same script below I get the silent input without
>>>>> echoing, no output like below, and I have to blindly type )off manually
>>>>> to exit interpreter.
>>>>> 
>>>>> 
>>>>> Juergen Sauermann
>>>>> <address@hidden>
>>>>>  writes:
>>>>> 
>>>>>  Hi Alexey,
>>>>> 
>>>>> I have changed the handling of command line options, SVN 877.
>>>>> 
>>>>> It now works like this:
>>>>> 
>>>>> script:
>>>>> 
>>>>> #!/usr/local/bin/apl --script --
>>>>> 
>>>>> )copy 5 FILE_IO FIO∆errno
>>>>> 8⎕CR ⎕ARG
>>>>> )off
>>>>> 
>>>>> output:
>>>>> 
>>>>> address@hidden:~/tmp$ ./script scriptarg
>>>>> DUMPED 2017-01-28 22:57:44 (GMT+1)
>>>>> ┌→──────────────────────────────────────────────────────────┐
>>>>> 
>>>>> │┌→─────────────────┐ ┌→───────┐ ┌→─┐ ┌→───────┐
>>>>> ┌→────────┐│
>>>>> ││/usr/local/bin/apl│ │--script│ │--│ │./script│ │scriptarg││
>>>>> │└──────────────────┘ └────────┘ └──┘ └────────┘
>>>>> └─────────┘│
>>>>> └∊──────────────────────────────────────────────────────────┘
>>>>> 
>>>>> 
>>>>> /// Jürgen
>>>>> 
>>>>> On 02/03/2017 11:06 PM, Alexey Veretennikov wrote:
>>>>> 
>>>>>  Hi,
>>>>> 
>>>>> Yes ./script -- myarg works.
>>>>> 
>>>>> The problem is why I have to repeat -- in command line since I've
>>>>> already specified it in the first line of the script, as it is specified
>>>>> in documentiation.
>>>>> 
>>>>> Basically I would like to pass my arguments to the script without
>>>>> mentioning "--" in the command line.
>>>>> 
>>>>> Juergen Sauermann
>>>>> <address@hidden>
>>>>>  writes:
>>>>> 
>>>>>  Hi Alexey,
>>>>> 
>>>>> how about this:
>>>>> 
>>>>> address@hidden:~/tmp$ ./script -- arg**
>>>>> **DUMPED 2017-01-28 22:57:44 (GMT+1)**
>>>>> **┌→────────────────────────────────────────────────────┐**
>>>>> **│┌→─────────────────┐ ┌→───────┐ ┌→───────┐ ┌→─┐ ┌→──┐│**
>>>>> **││/usr/local/bin/apl│ │--script│ │./script│ │--│ │arg││**
>>>>> **│└──────────────────┘ └────────┘ └────────┘ └──┘ └───┘│**
>>>>> **└∊────────────────────────────────────────────────────┘*
>>>>> 
>>>>> There is no point (and it does not work) to put the arguments in the 
>>>>> first line
>>>>> of the script,
>>>>> because if the script itself knows them then the rest of the script can 
>>>>> use them
>>>>> as well.
>>>>> 
>>>>> *⎕ARG *is what is passed to the script, not what is passed to the 
>>>>> interpreter
>>>>> mentioned in the script.
>>>>> See also *man execve*.
>>>>> 
>>>>> /// Jürgen
>>>>> 
>>>>> 
>>>>> On 02/03/2017 08:29 PM, Alexey Veretennikov wrote:
>>>>> 
>>>>>  Given the following script:
>>>>> ------------------------------------------
>>>>> #!apl --script --
>>>>> )copy 5 FILE_IO FIO∆errno
>>>>> 8⎕CR ⎕ARG
>>>>> )off
>>>>> ------------------------------------------
>>>>> when I try to run it like
>>>>> 
>>>>> ./script.apl myarg
>>>>> 
>>>>> I get the error:
>>>>> 
>>>>> /script.apl myarg
>>>>> unknown option 'myarg'
>>>>> ...
>>>>> 
>>>>> This happens on 833 and 1.5 and on both linux and osx.
>>>>> 
>>>>> In INFO file it explicitely states:
>>>>> 
>>>>> "Using ’—-’ as last argument on the first line of the script file
>>>>> prevents any of the options given to the script to be interpreted as APL
>>>>> options; all such options are passed to the application via ⎕ARG."
>>>>> 
>>>>> But it is not happening.
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>> 
>