SecurityManager troubles

[Guilhem Lavaux]

Hi,

I have stumbled across a bug probably shared by all VMs using GNU 
Classpath. In Apache Ant a SecurityManager is installed to be able to 
execute java application without forking the VM. Thanks to the SM 
exiting a VM is forbidden and so ant is protected from the applications 
using System.exit(). However it causes us some troubles for the cases 
where we have not pre-loaded some core classes such as 
java.security.Permission/java.security.Security (and probably some 
other). I have attached with this email a testcase which summarises 
quite well the situation.

1) If you try it without the '.class' invokations to preload the 
specified classes you will end up with a NoClassDefFoundError in the 
best case while invoking checkPermission. This is due to an hidden 
infinite loop recursion while the class loader invokes checkPermission 
for loading/defining a class. It happens that java/security/Permission 
is not yet loaded at the time checkPermission is invoked. So the VM 
tries to resolve/load/link it at method invokation. It's really bad 
because to be able to do it the VM invokes again checkPermission. In 
that case either the VM must check for circularity and throw a 
NoClassDefFoundError (or circularity I am not really sure here as I 
haven't checked carefully the JVM spec).

2) With the ".class" invokations the classes are preloaded by the VM 
before the SM is installed and so no problems occur because the VM does 
not invoke the class loader to retrieve the class.

3) One solution of the problem is to load some core classes. But it will 
appear quite soon that some other classes may also be loaded for really 
wicked applications. It is a limitative solution and I would not support it.

4) Another solution is to make a special case for selected core 
packages. In that case the class loader will bypass security 
verifications. This is a variant of 3 but less limitative and it does 
not force us to preload classes.

5) I don't know if we cannot go into other troubles if we apply (4).
At the moment the JDK has passed variants of the attached tests (and 
obviously without the ".class" preloading).

Any other idea ?

Regards,

Guilhem.

import java.net.*;
import java.security.*;

public class testSM
{
        static class mytest
        {
                mytest()
                {
                }
        }

        static class MySM extends SecurityManager
        {
                public void checkPermission(java.security.Permission perm)
                {
                        if (perm.getName().equals("aaaaaa"))
                        {
                                throw new SecurityException("no exit !");
                        } else if (!perm.getName().equals("charsetProvider"))
                        {
                        System.err.println("perm.getName()=" + perm.getName());
                        }
                }
        }

        static public void main(String args[]) throws Exception
        {
                Class sc = SecurityManager.class;
                Class sc2 = Security.class;
                Class sc3 = java.security.Permission.class;
                Class sc4 = java.lang.StringBuffer.class;
                Class sc5 = java.io.PrintStream.class;

                System.setSecurityManager(new MySM());
                
                URLClassLoader cl = 
(URLClassLoader)testSM.class.getClassLoader();
                URLClassLoader cl2 = new URLClassLoader(cl.getURLs());
                Class c = Class.forName("testSM$mytest", true, cl2);

                c.newInstance();
        }
}