path: root/user/jkoenig/gsoc2011_proposal.mdwn

# Java for Hurd (and vice vera)

*[Draft] I'll finish this later today and send an email to bug-hurd when I'm done*

Contact information:

  * Full name: Jérémie Koenig
  * Email: jk@jk.fr.eu.org
  * IRC: jkoenig on Freenode and OFTC

## Introductions

I am a first year M.Sc. student
in Computer Science at University of Strasbourg (France).
My interests include capability-based security,
programming languages and formal methods (in particular, proof-carrying code).

### Previous involvement

Although I have known the Hurd for some time,
I did not contribute until last summer,
during which I participated to Google Summer of Code
as a student for the Debian project.
I worked on porting Debian-Installer to Hurd.
This project was mostly a success,
although we still have to use a special mirror for installation
with a few modified packages
and tweaked priorities
to work around some uninstallable packages
with Priority: standard.

Shortly afterwards,
I rewrote the procfs translator
to fix some issues with memory leaks,
make it more reliable,
and improve comptibility with Linux-based tools
such as `procps` or `htop`.

Although I have not had as much time
as I would have liked to dedicate to the Hurd
since that time,
I have continued to maintain the mirror in question,
and I have started to work
on implementing POSIX threads signal semantics in glibc.

### Project-related skills and interests

I have used Java mostly for university assignements.
This includes non-trivial projects
using threads and distributed programming frameworks
such as Java RMI or CORBA.
I have also used it to experiment with
Google App Engine
(web applications)
and Google Web Toolkit
(a compiler from Java to Javascript which helps with AJAX code),
and I have some limited experience with JNI
(the Java Native Interface, to link Java with C code).

My knowledge of the Hurd and Debian GNU/Hurd is reasonable,
as the Debian-Installer and procfs projects
gave me the opportunity to fiddle with many parts of the system.


## Improve Java support

### Justification

Java is a popular language and platform used by many desktop and web
applications (mostly on the server side). As a consequence, competitive Java
support is important for any general-purpose operating system.

### Current situation

Java is currently supported on Hurd with the GNU Java suite:

   * [GCJ](http://gcc.gnu.org/java/),
     the GNU Compiler for Java, is part of GCC and can compile Java
     source code to Java bytecode, and both source code and bytecode to
     native code;
   * libgcj is the implementation of the Java runtime which GCJ uses.
     It is based on [GNU Classpath](http://www.gnu.org/software/classpath/).
     It includes a bytecode interpreter which enables
     Java applications compiled to native code to dynamically load and execute
     Java bytecode from class files.
   * The gij command is a wrapper around the above-mentioned virtual machine
     functionality of libgcj and can be used as a replacement for the java
     command.

However, GCJ does not work flawlessly on Hurd.r
For instance, some parts of libgcj relies on
the POSIX threads signal semantics, which are not yet implemented.
In particular, this makes ant hang waiting for child processes,
which makes some packages fail to build on Hurd
(“ant” is the “make” of the Java world).

### Tasks

   * **Finish implementing POSIX thread semantics** in glibc (high priority).
     According to POSIX, signal dispositions should be global to a process,
     while signal blocking masks should be thread-specific. Signals sent to the
     process as a whole are to be delivered to any thread which does not block
     them. By contrast, Hurd has per-thread signal dispositions and signals
     sent to a process are delivered to the main thread only. I have been
     working on refactoring the glibc signal code and implementing the POSIX
     semantics as a per-thread option. However, due to lack of time I have not
     yet been able to test and debug my code properly. Finishing this work
     would be my first task.
   * **Fix further problems with GCJ on Hurd** (high priority). While I’m not
     aware of any other problems with GCJ at the moment, I suspect some might
     turn up as I progress with the other tasks. Fixing these problems would
     also be a high-priority task.
   * **Port OpenJDK 6** (medium priority). While GCJ is fine, it is not yet
     100% complete. It is also slower than OpenJDK on architectures where a
     just-in-time compiler is available. Porting OpenJDK would therefore
     improve Java support on Hurd in scope and quality. Besides, it would also
     be a good way to test GCJ, which is used for bootstrapping by the Debian
     OpenJDK packages. Also note that OpenJDK 6 is now the default Java
     Runtime Environment on all released Linux-based Debian architectures;
     bringing Hurd in line with this would probably be a good thing.
   * **Port Eclipse and other Java applications** (low priority). Eclipse is a
     popular, state-of-the-art IDE and tool suite used for Java and other
     languages. It is a dependency of the Joe-E verifier (see part 3 of this
     proposal). Porting Eclipse would be a good opportunity to test GCJ and
     OpenJDK.


## Create Java bindings for the Hurd interfaces

### Justification

Java is a popular language, used for many applications and often taught to
introduce object-oriented programming. The fact that Java is a
garbage-collected language makes it easier to use, especially for the less
experienced programmers. Besides, the object-oriented nature of Java is a
natural fit for the capability-based design of Hurd.

Advantages over other garbage-collected, object-oriented languages include
performance, type safety and the possibility to compile a Java translator to
native code and
[link it statically](http://gcc.gnu.org/wiki/Statically_linking_libgcj)
using GCJ, should anyone want to use a
translator written in Java for booting. Note that Java is
[being](http://oss.readytalk.com/avian/)
[used](http://www.linuxjournal.com/article/8757)
in this manner for embedded development.

Java bindings would lower the bar for newcomers
to begin experimenting with what makes Hurd unique
without being faced right away with the complexity of
low-level systems programming.

### Approach

One approach used previously to interface programming languages with the Hurd
has been to create bindings for helper libraries such as libtrivfs. Instead,
for Java I would like to take a lower-level approach by providing access to
Mach primitives and extending MIG to generate Java code from the interface
description files.

This approach would be initially more involved, and would introduces several
issues related to overcoming the "impedence mismatch" between Java and Mach.
However, once an initial implementation is done it would be easier to maintain
in the long run and we would be able to provide Java bindings for a large
percentage of the Hurd’s interfaces.

### Design goals

FIXME: a completer

  * Give maximum flexibility to the Java code while maintaining the memory
    safety of Java, and using a minimum amount of C code.
  * Provide higher-level interfaces as well, ...
  * Hurd objects would map to Java objects.
  * Local objects can be accessed directly,
    remote objects can be accessed transparently over IPC.

### Bindings for Mach system calls

In this low-level approach, my intention is to enable Java code to use Mach
system calls (in particular, mach_msg) more or less directly. This would
ensure full access to the system from Java code, but it raises a number of
issues:

  * the Java code must be able to manipulate Mach-level entities, such as port
    rights or page-aligned buffers mapped outside of the garbage-collected
    heap (for out-of-line transfers);
  * putting together IPC messages requires control of the low-level
    representation of data.

In order to address these concerns, classes would be encapsulating these
low-level entities so that they can be referenced through normal, safe objects
from standard Java code. Bindings for Mach system calls can then be provided
in terms of these classes. Their implementation would use C code through the
Java Native Interface (JNI).

More specifically, this functionality would be provided by the `org.gnu.mach`
package, which would contain at least the following classes:

  * `MachPort` would encapsulate a `mach_port_t`. (Some of) its constructors
    would act as an interface for the `mach_port_allocate()` system call.
    `MachPort` objects would also be instanciated from other parts of the JNI
    C code to represent port rights received through IPC. The `deallocate()`
    mehod would call `mach_port_deallocate()` and replace the encapsulated
    port name with `MACH_PORT_DEAD`. We would recommend that users call it
    when a port is no longer used, but the finalizer would also deallocate the
    port when the `MachPort` object is garbage collected.
  * `Buffer` would represent a page-aligned buffer allocated outside of the
    Java heap, to be transferred (or having been received) as out-of-line
    memory. The JNI code would would provide methods to read and write data at
    an arbitrary offset (but within bounds) and would use `vm_allocate()` and
    `vm_deallocate()` in the same spirit as for `MachPort` objects.
  * `Message` would allow Java code to put together Mach messages. The
    constructor would allocate a `byte[]` member array of a given size.
    Additional methods would be provided to fill in or query the information
    in the message header and additional data items, including `MachPort` and
    `Buffer` objects which would be translated to the correponding port names
    and out-of-line pointers.
    A global map from port names to the corresponding `MachPort` object
    would probably be needed to ensure that there is a one-to-one
    correspondance.
  * `Syscall` would provide static JNI methods for performing system calls not
    covered by the above classes, such as `mach_msg()` or
    `mach_thread_self()`.  These methods would accept or return `MachPort`,
    `Buffer` and `Message` objects when appropriate. The associated C code
    would access the contents of such objects directly in order to perform the
    required unsafe operations, such as constructing `MachPort` and `Buffer`
    objects directly from port names and C pointers.

Note that careful consideration should be given to the interfaces of these
classes to avoid “safety leaks” which would compromise the safety guarantees
provided by Java. Potential problematic scenarios include the following
examples:

  * It must not be possible to write an integer at some position in a
    `Message` object, and to read it back as a `MachPort` or `Buffer` object,
    since this would allow unsafe access to arbitrary memory addresses and
    mach port names.
  * Providing the `mach_task_self()` system call would also provide acces to
    arbitrary addresses and ports by using the `vm_*` family of RPC operations
    with the returned `MachPort` object. This means that the relevant task
    operations should be provided by the `Syscall` class instead.

Finally, access should be provided to the initial ports and file descriptors
in `_hurd_ports` and `FIXME`, for instance through static methods such as
`getCRDir()`, `getCWDir()`, `getProc()`, ... in a dedicated class such as
`org.gnu.hurd.InitPorts`.

A realistic example of code based on such interfaces would be:

    import org.gnu.mach.MsgType;
    import org.gnu.mach.MachPort;
    import org.gnu.mach.Buffer;
    import org.gnu.mach.Message;
    import org.gnu.mach.Syscall;
    import org.gnu.hurd.InitPorts;

    public class Hello
    {   
       public static main(String argv[])
           /* Parent class for all Mach-related exceptions */
           throws org.gnu.mach.MachException
       {   
           /* Allocate a reply port */
           MachPort reply = new MachPort();

           /* Allocate an out-of-line buffer */
           Buffer data = new Buffer(MsgType.CHAR, 13);
           data.writeString(0, "Hello, World!");

           /* Craft an io_write message */
           Message msg = new Message(1024);
           msg.setRemotePort(InitPorts.getdport(1));
           msg.setLocalPort(reply, Message.Type.MAKE_SEND_ONCE);
           msg.setId(21000);
           msg.addBuffer(data);

           /* Make the call, MACH_MSG_SEND | MACH_MSG_RECEIVE */
           Syscall.machMsg(msg, true, true, reply);

           /* Extract the returned value */
           msg.assertId(21100);
           int retCode = msg.readInt(0);
           int amount = msg.readInt(1);
       }
    }

Should this paradigm prove insufficient,
more ideas could be borrowed from the
[`org.vmmagic`](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.151.5253&rep=rep1&type=pdf)
package used by [Jikes RVM](http://jikesrvm.org/),
a research Java virtual machine itself written in Java.

### Generating Java stubs with MIG

Once the basic machinery is in place to interface with Mach, Java programs
have more or less equal access to the system functionality without resorting
to more JNI code. However, as illustrated above, this access is far from
convenient.

As a solution I would modify MIG to add the option to output Java code. MIG
would emit a Java interface, a client class able to implement the interface
given a Mach port send right, an a server class which would be able to handle
incoming messages. The class diagram below, although it is by no means
complete or exempt of any problem, illustrates the general idea:

[[gsoc2011_classes.png]]

This structure is somewhat reminiscent of
[Java RMI](http://en.wikipedia.org/wiki/Java_remote_method_invocation)
or similar systems,
which aim to provide more or less transparent access to remote objects.
The exact way the Java code would be generated still needs to be determined,
but basically:

  * An interface, corresponding to the header files generated by MIG, would
    enumerate the operations listed in a given .defs files. Method names would
    be transformed to adhere to Java conventions (for instance,
    `some_random_identifier` would become `someRandomIdentifier`).
  * A user class, corresponding to the `*User.c` files,
    would implement this interface by doing RPC over a given MachPort object.
  * A server class, corresponding to `*Server.c`, would be able to handle
    incoming messages using a user-provided implementation of the interface.
    (Possibly, a skelton class providing methods which would raise
    `NotImplementedException`s would be provided as well.
    Users would derive from this class and override the relevant methods.
    This would allow them not to implement some operations,
    and would avoid pre-existing code from breaking when new operations are
    introduced.)

In order to help with the implementation of servers, some kind of library
would be needed to associate Mach receive rights with server objects and to
handle incoming messages on dedicated threads, in the spirit of libports.
This would probably require support for port sets at the level of the Mach
primitives described in the previous section.

When possible, operations involving the transmission of send rights
of some kind would be expressed in terms of the MIG-generated interfaces
intead of `MachPort` objects.
Upon reception of a send right,
a `FooUser` object would be created
and associated with the corresponding `MachPort` object.
If the received send right corresponds to a local port
to which a server object has been associated,
this object would be used instead.
This way,
subsequent operations on the received send right
would be handled as direct method calls
instead of going through RPC mechanisms.

Some issues will still need to be solved regarding how MIG will convert
interface description files to Java interfaces. For instance:

  * `.defs` files are not explicitely associated with a type. For instance in
    the example above, MIG would have to somehow infer that io_t corresponds
    to `this` in the `Io` interface.
  * More generally, a correspondance between MIG and Java types would have
    to be determined. Ideally this would be automated and not hardcoded
    too much.
  * Initially, reply port parameters would be ignored. However they may be
    needed for some applications.

So the details would need to be flushed out during the community bonding
period and as the implementation progresses. However I’m confident that a
satisfactory solution can be designed.

Using these new features, the example above could be rewritten as:

    import org.gnu.hurd.InitPorts;
    import org.gnu.hurd.Io;
    import org.gnu.hurd.IoUser;

    class Hello {
       static void main(String argv[]) throws ...
       {
           Io stdout = new IoUser(InitPorts.getdport(1));
           String hello = “Hello, World!\n”;

           int amount = stdout.write(hello.getBytes(), -1);

           /* (A retCode corresponding to an error
               would be signalled as an exception.) */
       }
    }

An example of server implementation would be:

    import org.gnu.hurd.Io;
    import java.util.Arrays;

    class HelloIo implements Io {
       final byte[] contents = “Hello, World!\n”.getBytes();

       int write(byte[] data, int offset) {
           return SOME_ERROR_CODE;
       }

       byte[] read(int offset, int amount) {
           return Arrays.copyOfRange(contents, offset,
                                     offset + amount - 1);
       }

       /* ... */
    }

A new server object could then be created with `new IoServer(new HelloIo())`,
and associated with some receive right at the level of the ports management
library.

### Base classes for common types of translators

Once MIG can target Java code, and a libports equivalent is available,
creating new translators in Java would be greatly facilitated. However,
we would probably want to introduce basic implementations of filesystem
translators in the spirit of libtrivfs or libnetfs. They could take the form
of base classes implementing the relevant MIG-generated interfaces which
would then be derived by users,
or could define a simpler interface
which would then be used by adapter classes
to implement the required ones.

I would draw inspiration from libtrivfs and libnetfs
to design and implement similar solutions for Java.

### Packaging and long-term maintenance

The Java libraries resulting from this work
(including any MIG support classes),
as well as the class files built from the MIG-generated code
for the Mach and Hurd interface definition files,
would be provided as single `hurd-java` package for
Debian GNU/Hurd.
This package would be separate from both Hurd and Mach,
so as not to impose unreasonable build dependencies on them.

I expect I would be able to act as its maintainer in the forseeable future, 
either as an individual or as a part of the Hurd team.
Hopefully,
my code would be claimed by the Hurd project as their own,
and consequently the modifications to MIG
(which would at least conceptually depend on the Mach Java package)
could be integrated upstream.

Since by design,
the Java code would use only a small number of stable interfaces,
it would not be subject to excessive amounts of bitrot.
Consequenty,
maintenance would primarily consist in
fixing bugs as they are reported,
and adding new features as they are requested.
A large number of such requests
would mean the package is useful,
so I expect that the overall amount of work
would be correlated with the willingness of more people
to help with maintenance
should I become overwhelmed or get hit by a bus.


## Timeline

The dates listed are deadlines for the associated tasks.

  * *Community bonding period.*
    Discuss, refine and complete the design of the Java bindings
    (in particular the MIG and "libports" parts)
  * *May 23.*
    Coding starts.
  * *May 30.*
    Finish implementing pthread signal semantics.
  * *June 5.*
    Port OpenJDK
  * *June 19 (two weeks).* 
    Fix the remaining problems with GCJ and/or OpenJDK,
    possibly port Eclipse or other big Java packages.
  * *June 26.*
    Create the bindings for Mach.
  * *July 3.*
    Work on some kind of basic Java libports
    to handle receive rights.
  * *July 10.*
    Test, write some documentation and examples.
  * *July 24 (two weeks).*
    Add the Java target to MIG.
  * *July 31.*
    Test, write some documentation and examples.
  * *August 7.*
    Try to write a basic but non-trivial translator
    to evaluate the performance and ease of use of the result,
    rectify any rough edges this would uncover.
  * *Last two weeks.*
    Polish the code,
    do the packaging.


## Conclusion




## Appendix: potential applications of object-capability languages

The work discussed is this last part would have
fewer immediate benefits for the Hurd project
and has more of a research orientation.
It is also unlikely that there would be any time remaining 
to work on it at the end of the summer.
(Though it could work as some kind of reward
if I somehow managed to do a prefect job of all the rest
within the allocated time :-) ).
As a consequence,
I don't really consider this a part of my application.

This being said,
to some extent the project discussed here
will informed the way I design the Java bindings,
since it depends on them
and I intend to work on this at some point in the future.
I also believe it touches on some interesting ideas,
and a Summer of Code application is probably
as good an occasion as any to discuss them.

### Justification

The primary advantage of multi-server operating systems is the ability to
break what used to be the kernel into small pieces which can be isolated
from each others. This makes sense from an engineering perspective, as
smaller components can be swapped with different implementations and reduce
the impact of bugs.
A capability-based approach also ensures that the
authority wielded by components is clear and reduced to the minimum required
for them to function.
These properties are crucial to the Hurd's agenda of user freedom,
since in order to allow them to plug their own code into the system
[FIXME: développer]

However, this flexibility has a cost. In a system where the isolation of
components relies on running them inside different address spaces,
communication between them must be done through IPC calls.
This introduces a trade-off between the size of the modules
and performance as well as practicality,
which imposes a limit to the granularity with which the system
can be decomposed and the principle of least authority applied
(to the code within a given process, a Mach port is ambient authority).

Another issue is that of the threading structure of the system as a whole.
In systems based on a monolithic kernel, user threads execute the kernel
code themselves, which is then intrinsically concurrent. By contrast, in a
system based on a “client-server” paradigm, each server must be explicitly
multi-threaded if it is to serve requests concurrently. 

### Object-capability languages

An object-capability language is an object-oriented language which is
restricted enough so that object references are themselves capabilities.

One such language is Joe-E (FIXME: lien),
which is an object-capability subset of Java:
global state and static methods are mostly forbidden;
careful white-listing of the objects and methods
provided by the Java standard library
ensures that compliant code cannot not access ambient autority.
Ways in which object references can be transferred
are restricted to the most obvious ones
(for instance, exceptions are carefully restricted).

As a result, untrusted Joe-E code can be executed without any further
isolation and its autority can be controlled by carefully limiting the
object references which are passed to it.
This would allow to load and execute translators written in Joe-E
in a single address space.

### Bundling translators into a single process

[mechanisme pour transmettre le code Joe-E
et les port initiaux au serveur]
[émulation des différentes tâches]

### Challenges and further work

[proof-carrying code / typed assembly,
resource accounting (passer en revue la conception de Viengoos?)]