diff options
| -rw-r--r-- | community/meetings.mdwn | 13 | ||||
| -rw-r--r-- | community/meetings/ghm2010.mdwn | 4 | ||||
| -rw-r--r-- | community/meetings/ghm2011.mdwn | 13 | ||||
| -rw-r--r-- | contributing/web_pages/news/moth_next.mdwn | 3 | ||||
| -rw-r--r-- | hurd/running/qemu/discussion.mdwn | 92 | ||||
| -rw-r--r-- | hurd/translator/ext2fs.mdwn | 16 | ||||
| -rw-r--r-- | hurd/translator/ext2fs/filetype.mdwn | 33 | ||||
| -rw-r--r-- | ikiwiki.setup | 4 | ||||
| -rw-r--r-- | irc.mdwn | 65 | ||||
| -rw-r--r-- | microkernel/mach/gnumach/ports/xen.mdwn | 24 | ||||
| -rw-r--r-- | open_issues/git-core-2.mdwn | 92 | ||||
| -rw-r--r-- | open_issues/git_duplicated_content.mdwn | 131 | ||||
| -rw-r--r-- | open_issues/git_nfs_mmap.mdwn | 48 | ||||
| -rw-r--r-- | open_issues/nightly_builds.mdwn | 2 | ||||
| -rw-r--r-- | toolchain.mdwn | 8 | ||||
| -rw-r--r-- | toolchain/elfosabi_hurd.mdwn | 83 | ||||
| -rw-r--r-- | user/jkoenig/java.mdwn | 849 | ||||
| -rw-r--r-- | user/jkoenig/java/discussion.mdwn | 108 | ||||
| -rw-r--r-- | user/jkoenig/java/proposal.mdwn | 628 |
19 files changed, 1420 insertions, 796 deletions
diff --git a/community/meetings.mdwn b/community/meetings.mdwn index 939d5269..6c15d144 100644 --- a/community/meetings.mdwn +++ b/community/meetings.mdwn @@ -1,5 +1,5 @@ -[[!meta copyright="Copyright © 2007, 2008, 2009, 2010 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2007, 2008, 2009, 2010, 2011 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -13,13 +13,18 @@ License|/fdl]]."]]"""]] # Upcoming - * [[FOSDEM_2011]] + * [[GNU Hackers Meeting, 2011, Paris|ghm2011]] + +## In the Future + * [[Self-organised]] + # Past + * [[FOSDEM_2011]] * [[DebConf10]] - * [[GNU Hackers Meeting in the Hague 2010|ghm2010]] + * [[GNU Hackers Meeting, 2010, Den Haag|ghm2010]] * [[FOSDEM_2010]] * [[EuroSys_2009]] * [[FOSDEM_2008]] diff --git a/community/meetings/ghm2010.mdwn b/community/meetings/ghm2010.mdwn index b7d7e880..c8a9d8c3 100644 --- a/community/meetings/ghm2010.mdwn +++ b/community/meetings/ghm2010.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -8,7 +8,7 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -[[!meta title="GNU Hackers Meeting in the Hague 2010"]] +[[!meta title="GNU Hackers Meeting, 2010, Den Haag"]] <http://www.gnu.org/ghm/2010/denhaag/> diff --git a/community/meetings/ghm2011.mdwn b/community/meetings/ghm2011.mdwn new file mode 100644 index 00000000..7a2df8a0 --- /dev/null +++ b/community/meetings/ghm2011.mdwn @@ -0,0 +1,13 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="GNU Hackers Meeting, 2011, Paris"]] + +<http://www.gnu.org/ghm/2011/paris/> diff --git a/contributing/web_pages/news/moth_next.mdwn b/contributing/web_pages/news/moth_next.mdwn index 6dc16443..07cd5960 100644 --- a/contributing/web_pages/news/moth_next.mdwn +++ b/contributing/web_pages/news/moth_next.mdwn @@ -39,6 +39,9 @@ else=" * Plan for Debian GNU/Hurd: Get about 85% of packages ported and become a full Debian project for Wheezy (probably planed for something like between summer 2012 and winter 2013). So, if you always wanted to contribute to the Hurd and did not know how to help most effectively: [[Porting|contributing]] is a great way to help us! +* [[community/meetings/ghm2011]] + + This month [hurd hacker] [item] Also … diff --git a/hurd/running/qemu/discussion.mdwn b/hurd/running/qemu/discussion.mdwn index fd0df4c5..1ce14b01 100644 --- a/hurd/running/qemu/discussion.mdwn +++ b/hurd/running/qemu/discussion.mdwn @@ -50,3 +50,95 @@ The problem is actually that the linux block cache doesn't make any consistency between /dev/hda and /dev/hda6, so if you give /dev/hda to qemu, qemu writings won't be consistent with mounting /dev/hda6 in linux. You can give /dev/hda6 directly to qemu and it will be fine. + + +# Host-side Writeback Caching + +IRC, freenode, #hurd, 2011-06-07 + + <braunr> hm, i guess i should have used cache=writeback with kvm before + starting the debian installer :/ + <braunr> ah yes, much better + <braunr> this shows how poor the state of our I/O drivers and subsystem is + :/ + <antrik> indeed... still no clustered pageout :-( + <braunr> and no I/O scheduler either + <braunr> although an I/O scheduler has limited value without clustered + pageouts + <braunr> since one of its goals is to pack related I/O requests together eh + <braunr> i wonder if the wiki mentions using cache=writeback to speed up + qemu performances + <braunr> it would help those unable to use kvm a lot + <braunr> and even those running kvm too + <braunr> kvm -m $RAM \ -monitor stdio \ -drive + cache=writeback,index=0,media=disk,file=hd0.img \ + <braunr> etc.. + <braunr> the idea is that qemu doesn't open its disk file synchronously + <braunr> changes are queued in the host page cache before being flushed to + the disk image + <braunr> but if you brutally close your qemu instance, you're likely to + loose file system consistency + <braunr> ext2fs will think it has committed its metadata to the disk, but + the disk image won't be updated synchronously + <braunr> on my machine (which is quite fast), my kvm has installed debian + like 10 times faster than without the option + <antrik> braunr: I don't think killing qemu should hurt in this + case... probably only matters when the host machine dies + <braunr> antrik: ah yes, right + <braunr> it really makes everything faster, even downloading, since I/O + requests aren't interleaved between networking RPCs + <antrik> regarding I/O sheduler... this discussion came up before, but I + don't remember the outcome -- doesn't the glued Linux driver actually + come with one? + <braunr> i don't remember either + <antrik> braunr: err... I don't think interleaving has anything to do with + it... I guess it's simply the fact that downloading writes the result to + disk, which suffers from lacking clustered pageout like everything else + <antrik> (my internet connection is too slow though to notice :-) ) + <braunr> well, if there is no I/O during downloading, downloading is faster + :) + +IRC, freenode, #hurd, 2011-06-08 + + <braunr> youpi: does xen provide disk caching options ? + <youpi> through a blktap, probably + <braunr> ok + +([[microkernel/mach/gnumach/ports/Xen]], *Host-side Writeback Caching*.) + + <braunr> we should find the pages mentioning qemu on the wiki and add the + options to enable disk image caching + <braunr> it really makes the hurd run a lot faster + <braunr> as a workaround for emulators until I/O is reworked, ofc + +IRC, freenode, #hurd, 2011-06-09 + + <gnu_srs> braunr recommends to use writeback caching with kvm. Is this + really recommended with the frequent crashes I experience? + <youpi> provided that you terminate your kvm normaly (i.e. quitting it, not + killing it), there should be no difference + <jkoenig> I think the host's stability is what matters + <jkoenig> the data presumably sits in linux's cache even if qemu dies + violently + <gnu_srs> But the freezes I see force me to kill kvm :-( + <youpi> maybe kvm doesn't even do caching indeed, I don't know + <youpi> gnu_srs: you can quit even when frozen + <youpi> use the console + <youpi> (the kvm console) + <jkoenig> gnu_srs, "Writeback caching will report data writes as completed + as soon as the data is present in the host page cache. This is safe as + long as you trust your host. If your host crashes or loses power, then + the guest may experience data corruption." (from the qemu manpage) + +IRC, freenode, #hurd, 2011-06-11 + + <gnu_srs> braunr: If you are online. For me setting the parameters -drive + cache=writeback,index=0,media=disk,file=hd0.img does not show any speed + improvement at all compared to the default. + <braunr> gnu_srs: what's your complete qemu command line ? + <gnu_srs> kvm -m 1024 -net nic,model=rtl8139 -net + user,hostfwd=tcp::5556-:22 -drive + cache=writeback,index=0,media=disk,file=hd0.img -cdrom netinst.iso + <braunr> what qemu version ? + <gnu_srs> qemu-kvm 0.14.1+dfsg-1: Sorry, I cannot be online until + tomorrow again. diff --git a/hurd/translator/ext2fs.mdwn b/hurd/translator/ext2fs.mdwn index 305576b8..fff2e74b 100644 --- a/hurd/translator/ext2fs.mdwn +++ b/hurd/translator/ext2fs.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2007, 2008, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2007, 2008, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -9,14 +9,20 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -# Large Stores +# Implementation + + * [[filetype]] option + + +## Large Stores The `ext2fs` translator from the upstream Hurd code base can only handle file systems with sizes of less than roughly 2 GiB. [[!tag open_issue_hurd]] -## Ognyan's Work + +### Ognyan's Work * Ognyan Kulev, [[*Supporting Large ext2 File Systems in the Hurd*|ogi-fosdem2005.mgp]], 2005, at FOSDEM @@ -34,4 +40,6 @@ small backend stores, like floppy devices. # Documentation -<http://www.nongnu.org/ext2-doc/> + * <http://e2fsprogs.sourceforge.net/ext2.html> + + * <http://www.nongnu.org/ext2-doc/> diff --git a/hurd/translator/ext2fs/filetype.mdwn b/hurd/translator/ext2fs/filetype.mdwn new file mode 100644 index 00000000..5d85bac9 --- /dev/null +++ b/hurd/translator/ext2fs/filetype.mdwn @@ -0,0 +1,33 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +The *ext2fs* translator doesn't support the ext2 format's *filetype* option. + +According to *mke2fs(8)*: + +> **filetype**: Store file type information in directory entries. + +By setting directory listings' informational `d_type` field (`readdir`, etc.), +this may avoid the need for subsequent `stat` calls. + +Not all file systems can support this option. + +In `[hurd]/ext2fs/dir.c` the `EXT2_FEATURE_INCOMPAT_FILETYPE` is generally +masked out (is not even considered) when adding a node to a directory in +`diskfs_direnter_hard` and when reading in `diskfs_get_directs`. The Hurd's +ext2fs unconditionally sets this field to 0 (`EXT2_FT_UNKNOWN`). + + +# `e2fsck` + +Running `e2fsck` on a file system with the *filetype* option, will correct the +*filetype* for a lot of files (all `EXT2_FT_UNKNOWN`?) to either 1 (regular +file, `EXT2_FT_REG_FILE`), or 2 (directory, `EXT2_FT_DIR`), and likely others. +The Hurd's ext2fs will again ignore these fields, of course. diff --git a/ikiwiki.setup b/ikiwiki.setup index 3ef1bb73..9ae1f669 100644 --- a/ikiwiki.setup +++ b/ikiwiki.setup @@ -39,8 +39,8 @@ IkiWiki::Setup::Standard->import({ # users who are wiki admins adminuser => [qw{tschwinge}], # users who are banned from the wiki - banned_users => [qw{AlbertF bernhart ColetCris flamberian jasclaine - NateNash + banned_users => [qw{AlbertF bernhart ColetCris flamberian Jack_Dark + jasclaine NateNash http://calvinyoung.myopenid.com/ http://heaton.myopenid.com/ http://hilarybunton.myopenid.com/}], @@ -11,41 +11,22 @@ License|/fdl]]."]]"""]] [[!meta title="IRC"]] -While all official development takes place on the mailing lists and the Savannah trackers, -a lot of discussions are had on IRC as well. Everybody is welcome to join and follow these channels, but please -respect the below guidelines if you want to participate. - -# Asking Questions - -[[!inline pages=asking_questions raw=yes feeds=no]] - -# Staying On-Topic +There are some IRC channels where Hurd topics are discussed. -Please try to stay on topic. +Everybody is welcome to join and participate in the discussions, but please +respect the below guidelines if you want to participate. -* emacs vs. vi **is not** on topic -* If it is appropriate for a **slashdot comment**, it's **not appropriate** - here -* why GNU sucks is **off-topic** -* when the next release of the Hurd will be **is inappropriate** -* you should not advocate your favorite **GNU/Linux** ditribution +Please try to stay on topic. We're not interested in Emacs vs. vi discussions, +we don't want to hear why you think that the GNU project sucks, or what your +favorite GNU/Linux distribution is. -# Pasting Logs +[[!inline pages=asking_questions raw=yes feeds=no]] Sometimes providing a log or some other excerpt of text can -help solve a problem or answer a question. **Do not** paste -the log in the channel itself. Instead use a +help solve a problem or answer a question. Do not paste +the log in the channel itself if it's more than a few lines. Instead use a [paste bin](http://paste.debian.net). -# Rich Text - -Don't use it. Don't use colors. Don't use bold. Don't use emphasis. - -# Greeting - -If you never contribute to the discussion, there is no need -to always greet the channel when you enter and before leave. - <a name="regular_meetings"></a> # Regular Meetings @@ -54,33 +35,33 @@ Starting in early 2008, there have been regular IRC meetings held between the (now former) [[Google Summer of Code|community/gsoc]] students and their mentors. These meetings turned out to considerably help student-mentor interactions, and other developers regularely took part, too. -<!-- For this reason, we decided to continue having these meetings, even if it's not currently Google Summer of Code time. ---> Currently, the meetings take place in the **`#hurd` channel every -Tuesday at 19:00 UTC** and are open to any interested party. So, +Wednesday at 19:00 UTC** and are open to any interested party. So, everyone, take your chance to chat with GNU Hurd developers! # Channels -All Hurd IRC channels are hosted on [Freenode.net](http://freenode.net/). +## Freenode + +`irc.freenode.net`, <http://freenode.net/> + + * `#hurd`, the official GNU Hurd IRC channel. Some of the Hurd developers + and users hang out there. [Logs](http://richtlijn.be/~larstiq/hurd/). + + * `#hurdfr`, the French chapter. -* #hurd - The official Hurd IRC channel. Some of the Hurd developers and users hang out there, and discussions about GNU Hurd, GNU/Hurd and -Debian GNU/Hurd are had there. + * `#archhurd`, [[hurd/running/Arch_Hurd]]. + [Logs](http://files.archhurd.org/irclogs). -Local user channels include: -* #hurd-it - Italian -* #hurd-es - Spanish -* #hurdfr - French -* #hurd-de - German -* #hurd.in - Indian regional languages(but primarily English) +## OFTC -# Channel logs +`irc.oftc.net`, <http://www.oftc.net/> -* [#hurd logs](http://richtlijn.be/~larstiq/hurd/) + * `#debian-hurd`, [[Debian GNU/Hurd|hurd/running/debian]]. # Related diff --git a/microkernel/mach/gnumach/ports/xen.mdwn b/microkernel/mach/gnumach/ports/xen.mdwn index bf26410a..af431c92 100644 --- a/microkernel/mach/gnumach/ports/xen.mdwn +++ b/microkernel/mach/gnumach/ports/xen.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2007, 2008, 2009 Free Software Foundation, +[[!meta copyright="Copyright © 2007, 2008, 2009, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -6,11 +6,12 @@ id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!toc]] + # Xen dom0, hypervisor /!\ Now that GNU Mach handles PAE you can use a PAE-enabled hypervisor. @@ -30,6 +31,7 @@ If you have a free partition, you can fdisk to type 0x83, create a filesystem us Replace /dev/sda4 with your partition. Install and use crosshurd to setup a GNU/Hurd system on this partition. + # /etc/xen/hurd configuration Here is a sample /etc/xen/hurd configuration @@ -49,6 +51,7 @@ Suggestions about [[networking_configuration]] are available. If you need stable MAC addresses, use a syntax like `vif = [ 'mac=00:16:3e:XX:XX:XX, bridge=br0' ]`. + # Running Hurd with Xen To run Hurd with Xen, use: @@ -63,6 +66,7 @@ and gnumach should get started. Proceed with native-install. - If `xm` complains about networking (`vif could not be connected`), it's Xen scripts' fault, see Xen documentation for how to configure the network. The simplest way is network-bridge with fixed IPs (note that you need the bridge-utils package for this). You can also just disable networking by commenting the vif line in the config. - If `xm` complains `Error: (2, 'Invalid kernel', 'xc_dom_compat_check: guest type xen-3.0-x86_32 not supported by xen kernel, sorry\n')`, you most probably have a PAE-enabled hypervisor and a non-PAE gnumach. Either install and boot non-PAE hypervisor and kernel, or rebuilt gnumach in PAE mode. + # Building from sources If you want to generate these images, first get the `gnumach-1-branch-Xen-branch` branch from gnumach CVS. @@ -75,7 +79,6 @@ Then use The current `hurd-modules` was built from the debian packages `hurd 20070606-2` and `libc0.3 2.6.1-1`. /!\ This means that when using this image, your GNU/Hurd system also needs to be a glibc version 2.6 or later-based one! ---- # Miscellaneous @@ -83,7 +86,6 @@ The current `hurd-modules` was built from the debian packages `hurd 20070606-2` [[!GNU_Savannah_task 5468]], [[!GNU_Savannah_task 6584]]. ---- # `pv-grub` @@ -91,3 +93,15 @@ From Xen 4.0 on you'll be able to run the GNU Hurd directly using `pv-grub`, without the need to [prepare a special bootstrap image](http://youpibouh.thefreecat.org/hurd-xen/build_hurd-modules) (like an initrd). + + +# Host-side Writeback Caching + +Optimization possible as it is with [[QEMU|hurd/running/qemu/discussion]], +*Host-side Writeback Caching*? + +IRC, freenode, #hurd, 2011-06-08 + + <braunr> youpi: does xen provide disk caching options ? + <youpi> through a blktap, probably + <braunr> ok diff --git a/open_issues/git-core-2.mdwn b/open_issues/git-core-2.mdwn index 2cac56f2..2d8ad96b 100644 --- a/open_issues/git-core-2.mdwn +++ b/open_issues/git-core-2.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -13,6 +13,13 @@ License|/fdl]]."]]"""]] [[!tag open_issue_porting]] +[[!toc]] + + +# Log + +December, 2008. + On the otherwise-idle flubber: $ git clone git://sources.redhat.com/git/glibc.git @@ -98,3 +105,86 @@ differences to HEAD. fatal: cannot pread pack file: Interrupted system call fatal: index-pack failed error: Could not fetch savannah + +--- + +2011-06-10, coulomb.SCHWINGE, checking out [[binutils]]' master branch, +starting from an empty working directory (after an external `git push`): + + $ git checkout -f + fatal: cannot create directory at 'gas/testsuite/gas/bfin': Interrupted system call + $ git checkout -f + error: unable to create file gas/testsuite/gas/i386/ilp32/x86-64-sse4_1-intel.d (File exists) + warning: unable to unlink gas/testsuite/gas/m68k-coff: Operation not permitted + fatal: cannot create directory at 'gas/testsuite/gas/m68k-coff': Operation not permitted + $ git checkout -f + error: unable to create file gas/testsuite/gas/h8300/h8300.exp (File exists) + error: unable to create file gas/testsuite/gas/i386/x86-64-addr32-intel.d (File exists) + error: unable to create file gas/testsuite/gas/ia64/secname.d (File exists) + error: unable to create file gas/testsuite/gas/m68k/pr11676.s (File exists) + Checking out files: 100% (12315/12315), done. + $ git status + # On branch master + # Changes not staged for commit: + # (use "git add <file>..." to update what will be committed) + # (use "git checkout -- <file>..." to discard changes in working directory) + # + # modified: gas/testsuite/gas/h8300/h8300.exp + # modified: gas/testsuite/gas/i386/x86-64-addr32-intel.d + # modified: gas/testsuite/gas/ia64/secname.d + # modified: gas/testsuite/gas/m68k/pr11676.s + # + no changes added to commit (use "git add" and/or "git commit -a") + $ rm gas/testsuite/gas/h8300/h8300.exp gas/testsuite/gas/i386/x86-64-addr32-intel.d gas/testsuite/gas/ia64/secname.d gas/testsuite/gas/m68k/pr11676.s + $ git checkout -f + $ git status + # On branch master + nothing to commit (working directory clean) + + +# Analysis + +2011-06-13 + +Running `git checkout -f` under GDB: + + error: git checkout-index: unable to create file gas/testsuite/gas/cris/string-1.s (File exists) + error: git checkout-index: unable to create file gas/testsuite/gas/i386/x86-64-sse-check.d (File exists) + error: git checkout-index: unable to create file gas/testsuite/gas/i386/x86-64-sse4_1.d (File exists) + error: git checkout-index: unable to create file gas/testsuite/gas/ppc/astest.d (File exists) + error: git checkout-index: unable to create file gas/testsuite/gas/tic6x/reloc-bad-4.s (File exists) + warning: unable to unlink include/cgen: Operation not permitted + fatal: cannot create directory at 'include/cgen': Operation not permitted + +Again: + + error: git checkout-index: unable to create file gas/config/te-vxworks.h (File exists) + error: git checkout-index: unable to create file gas/testsuite/gas/cris/string-1.s (File exists) + error: git checkout-index: unable to create file gas/testsuite/gas/d10v/warning-019.s (File exists) + error: git checkout-index: unable to create file gas/testsuite/gas/i860/dual03.s (File exists) + error: git checkout-index: unable to create file ld/testsuite/ld-mmix/sec-7a.s (File exists) + warning: unable to unlink ld/testsuite/ld-powerpc: Operation not permitted + fatal: cannot create directory at 'ld/testsuite/ld-powerpc': Operation not permitted + +And: [[git_duplicated_content]]. + +All these (very likely) have the same root cause: `SA_RESTART` restarting too +much. + +With `git checkout`, Git uses in progress.c a SIGALRM handler (`SA_RESTART`; +invoked every second via `setitimer(ITIMER_REAL)`) to display status messages: +*x % already checked out*. + +To avoid the status update signals every second, in +`[git]/progress.c:start_progress_delay` we can just return `NULL` (manually in +GDB, for example), then both the *error: git checkout-index* and the +[[duplicated content|git_duplicated_content]] issues go away. + +I'm guessing that when returning from a `SA_RESTART` signal handler, too much +of the \`\`syscall''s is being restarted. For example, if a file has already +been created, the restarted creation attempt would fail: *File exists*. If +data has been written, it might get written again (duplication issue). Then, +there are cases where `unlink` apparently returns EINTR, which is not kosher +either. Etc. + +Do we have problems with `SA_RESTART` vs. the atomicity of our syscall-alikes? diff --git a/open_issues/git_duplicated_content.mdwn b/open_issues/git_duplicated_content.mdwn new file mode 100644 index 00000000..cbc171a7 --- /dev/null +++ b/open_issues/git_duplicated_content.mdwn @@ -0,0 +1,131 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_porting]] + + $ git-new-workdir ~/tmp/binutils/git /media/hd1s1/tmp/master master + error: unable to create file gas/testsuite/gas/arm/attr-mfpu-vfpv3-d16.d (Interrupted system call) + Checking out files: 100% (12315/12315), done. + Already on 'master' + $ cd /media/hd1s1/tmp/master + $ git status + # On branch master + # Changes not staged for commit: + # (use "git add <file>..." to update what will be committed) + # (use "git checkout -- <file>..." to discard changes in working directory) + # + # modified: gas/testsuite/gas/arm/attr-mfpu-vfpv3-d16.d + # + no changes added to commit (use "git add" and/or "git commit -a") + $ git checkout -f + $ git status + # On branch master + nothing to commit (working directory clean) + +([[Git issue|git-core-2]] is known.) + + $ git-new-workdir ~/tmp/binutils/git /media/hd1s2/tmp/master master + error: unable to create file bfd/elf32-dlx.c (Interrupted system call) + error: unable to create file bfd/sunos.c (Interrupted system call) + error: unable to create file gas/testsuite/gas/arm/attr-mfpu-vfpv3-d16.d (Interrupted system call) + error: unable to create file gas/testsuite/gas/mmix/regx-op.d (Interrupted system call) + error: unable to create file gas/testsuite/gas/tic6x/reloc-bad-4.s (Interrupted system call) + error: unable to create file gold/testsuite/script_test_2.t (Interrupted system call) + error: unable to create file ld/testsuite/ld-mmix/loc7m.d (Interrupted system call) + error: unable to create file ld/testsuite/ld-powerpc/tlsexe.g (Interrupted system call) + Checking out files: 100% (12315/12315), done. + Already on 'master' + $ cd /media/hd1s2/tmp/master + $ git status + # On branch master + # Changes not staged for commit: + # (use "git add <file>..." to update what will be committed) + # (use "git checkout -- <file>..." to discard changes in working directory) + # + # modified: bfd/elf32-dlx.c + # modified: bfd/sunos.c + # modified: gas/testsuite/gas/arm/attr-mfpu-vfpv3-d16.d + # modified: gas/testsuite/gas/mmix/regx-op.d + # modified: gas/testsuite/gas/tic6x/reloc-bad-4.s + # modified: gold/testsuite/script_test_2.t + # modified: ld/testsuite/ld-mmix/loc7m.d + # modified: ld/testsuite/ld-powerpc/tlsexe.g + # + no changes added to commit (use "git add" and/or "git commit -a") + $ git checkout -f + $ git status + # On branch master + nothing to commit (working directory clean) + +Now you'd expect these directories to have identical content, but: + + $ diff -x .git -ru /media/hd1s{1,2}/tmp/master/ > /tmp/diff + $ ls -l /tmp/diff + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:12 /tmp/diff + $ grep '^[^ @+-]' < /tmp/diff + diff -x .git -ru /media/hd1s1/tmp/master//ld/configure /media/hd1s2/tmp/master//ld/configure + +(Note that this isn't a file that Git had issues with.) + +Try again: + + $ diff -x .git -ru /media/hd1s{1,2}/tmp/master/ > /tmp/diff_ + $ ls -l /tmp/diff* + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:12 /tmp/diff + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:17 /tmp/diff_ + $ cmp /tmp/diff{,_}; echo $? + 0 + +At least it's consistent. Force a reload: + + # settrans -ag /media/hd1s1 + # settrans -ag /media/hd1s2 + +Try again: + + $ diff -x .git -ru /media/hd1s{1,2}/tmp/master/ > /tmp/diff__ + $ ls -l /tmp/diff* + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:12 /tmp/diff + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:17 /tmp/diff_ + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:30 /tmp/diff__ + $ cmp /tmp/diff{,__}; echo $? + 0 + +Consistent; thus very likely corrupt on-disk. + +After a few tries, the pattern generally is that for the files where there are +differences, once the file regularely ends, its content appears once more. +That is, the files' content appears once (regularely), and then the same again. + +Some more copying: + + $ (cd /media/hd1s1/tmp/ && cp -a master master_) + $ (cd /media/hd1s2/tmp/ && cp -a master master_) + $ diff -x .git -ru /media/hd1s1/tmp/master{,_}/ > /tmp/diff1 + $ diff -x .git -ru /media/hd1s2/tmp/master{,_}/ > /tmp/diff2 + $ ls -l /tmp/diff{1,2} + -rw-r--r-- 1 thomas thomas 0 10. Jun 19:46 /tmp/diff1 + -rw-r--r-- 1 thomas thomas 0 10. Jun 19:46 /tmp/diff2 + +No further difference. + +--- + + $ git-new-workdir git master master + $ diff -x .git -ur tar_master/ master/ > master.diff + + $ rm -rf ar_master* && (cd git/ && git archive master) | (mkdir ar_master && cd ar_master/ && tar -x) && diff -x .git -ru tar_master/ ar_master/ > ar_master.diff; ls -l ar_master.diff + $ (cd git/ && git archive master) | md5sum + +--- + +2011-06-13 + +-> [[git-core-2]] diff --git a/open_issues/git_nfs_mmap.mdwn b/open_issues/git_nfs_mmap.mdwn new file mode 100644 index 00000000..b7e86491 --- /dev/null +++ b/open_issues/git_nfs_mmap.mdwn @@ -0,0 +1,48 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd open_issue_porting]] + + $ git-new-workdir /media/kepler-data/home/thomas/tmp/source/binutils/git master master + fatal: Out of memory? mmap failed: No such device + $ echo $? + 128 + $ showtrans /media/kepler-data + /hurd/nfs kepler.schwinge.homeip.net:/media/data + +With `sh -x`: + + [...] + + ln -s /media/kepler-data/home/thomas/tmp/source/binutils/git/.git/remotes master/.git/remotes + + ln -s /media/kepler-data/home/thomas/tmp/source/binutils/git/.git/rr-cache master/.git/rr-cache + + ln -s /media/kepler-data/home/thomas/tmp/source/binutils/git/.git/svn master/.git/svn + + cd master + + cp /media/kepler-data/home/thomas/tmp/source/binutils/git/.git/HEAD .git/HEAD + + git checkout -f master + fatal: Out of memory? mmap failed: No such device + +As one can easily guess (and confirm with [[hurd/debugging/rpctrace]]), `git` +tries to [[glibc/mmap]] a file via the [[hurd/translator/nfs]] translator, this +fails, and it isn't prepared to cope with that: + + [...] + 88->dir_lookup (".git/objects/pack/pack-37ca560e7877fa0cc6e5ddcd556aa73e5a3e3f40.idx" 2049 0) = 0 3 "/media/kepler-data/home/thomas/tmp/source/binutils/git/.git/objects/pack/pack-37" (null) + 62->dir_lookup ("media/kepler-data/home/thomas/tmp/source/binutils/git/.git/objects/pack/pack-37c" 2049 0) = 0 1 "/home/thomas/tmp/source/binutils/git/.git/objects/pack/pack-37ca560e7877fa0cc6e5" 61 + 61->dir_lookup ("home/thomas/tmp/source/binutils/git/.git/objects/pack/pack-37ca560e7877fa0cc6e5d" 2049 0) = 0 1 "" 84 + task3741-> 3206 (pn{ 33}) = 0 + 84->term_getctty () = 0xfffffed1 ((ipc/mig) bad request message ID) + 84->io_stat_request () = 0 {1 704 0 36308992 0 0 -1 33060 1 1000 1000 4712 0 1307711395 0 1307657003 0 1307657003 0 4096 16 0 1000 0 0 100663296 1836017780 29537 0 0 0 0} + 84->io_map_request () = 0x4000002d (Operation not supported) + 84->io_map_request () = 0x4000002d (Operation not supported) + 76->io_write_request ("fatal: Out of memory? mmap failed: No such device + " -1) = 0 50 + 64->proc_mark_exit_request (32768 0) = 0 + task3741-> 2008 () = 0 + Child 3741 exited with 128 diff --git a/open_issues/nightly_builds.mdwn b/open_issues/nightly_builds.mdwn index 5d1257fb..6eef7d19 100644 --- a/open_issues/nightly_builds.mdwn +++ b/open_issues/nightly_builds.mdwn @@ -20,7 +20,7 @@ Resources: * As reported in the [[news/2010-05-31]] news, there's Hydra doing nightly builds / Nix packages. - * <http://hudson-ci.org/> + * <http://hudson-ci.org/>, <http://jenkins-ci.org/> * <http://buildbot.net/> diff --git a/toolchain.mdwn b/toolchain.mdwn index b08dba95..0059317b 100644 --- a/toolchain.mdwn +++ b/toolchain.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2007, 2008, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2007, 2008, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -24,4 +24,10 @@ for example, how things are done differently for GNU/Hurd than they are done for GNU/Linux. +--- + + * [[ELFOSABI_HURD]] + +--- + * [[cross-gnu]] diff --git a/toolchain/elfosabi_hurd.mdwn b/toolchain/elfosabi_hurd.mdwn new file mode 100644 index 00000000..4d60f761 --- /dev/null +++ b/toolchain/elfosabi_hurd.mdwn @@ -0,0 +1,83 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="ELFOSABI_HURD"]] + +[[!tag open_issue_binutils open_issue_glibc]] + + +# [[!debbug 630180]] + + +# [[open_issues/binutils]] commit 51b2f560ad035dffad3371093f8e5c80608d196c + +Usage of `ELFOSABI_LINUX`/`STB_GNU_UNIQUE`. Has also been wrong before already +with respect to `STT_GNU_IFUNC`? + + +# IRC + +IRC, freenode, #debian-hurd, 2011-06-11 + + <pinotree> youpi: not that there would be any hope in that, but id you try + asking doko about the gcc miscompiling (wrong elf format) issue? + <youpi> I didn't + <youpi> I'm still investigating + <youpi> maybe it's a binutils change actually + <pinotree> youpi: hm, are you sure it could be binutils? after all, even + some .o files are produced with format gnu/linux, so there's no binutils + involved up to that point of thecompilation yet? + <youpi> as is + <youpi> "as", I mean + <pinotree> i see + <youpi> since it's so unclear, I really prefer to investigate before + bothering doko + <pinotree> youpi: maybe i could be wrong, + <pinotree> in binutils, bfd/elf.c, around lines 9580 + <youpi> the faulty thing seems to be gnu_unique_object in the source .s + file produced by g++ + <pinotree> that's what that comment (which changed wrt binutils from eg + march) says + <youpi> seems to concur with my comment above :) + <pinotree> http://paste.debian.net/119542/ ¡û extract of diff + <youpi> ok, that really seems the culprit + <youpi> starting reportbug + <pinotree> who's the fault then? + <youpi> binutils + <youpi> it shouldn't hardcode LINUX + <pinotree> g++ emitting those symbols, or binutil considering them "linux"? + <youpi> it's a GNU thing, not a Linux thing + <pinotree> ah ok + <youpi> it's the same dynamic linker actually + <pinotree> youpi: http://sourceware.org/bugzilla/show_bug.cgi?id=10549 + <pinotree> see the reporter :) + <youpi> heh + <pinotree> youpi: see also gas/config/obj-elf.c:1725 + <pinotree> (another change related to that bug, it seems) + +IRC, freenode, #hurd, 2011-06-15 + + <jkoenig> also I still get an "ELF file OS ABI invalid" error with binutils + 2.21.52.20110606-2+hurd.1, is that expected? + <youpi> tschwinge: oops, the OS ABI invalid is actually due to the file + being marked GNU/Hurd + <youpi> I guess the linker is simply not aware that it should accept + GNU/Hurd + <tschwinge> youpi: So we got to work on glibc'S ld.so to teach it aboput + the Hurd OS ABI? (Or probably simply make that equivalent to the Linux + one?) + <youpi> probably simply an equivalent + <youpi> ELFOSABI_HURD is missing from elf/elf.h, for a start... + <youpi> linux' glibc has tests in lsdodefs.h + <youpi> the VALID_ELF_OSABI macro + <youpi> it's thus apparently a matter of providing an ldsodefs.h file with + VALID_ELF_HEADER, VALID_ELF_OSABI and VALID_ELF_ABIVERSION definitions + <youpi> (and include_next the generic one) + <youpi> I've prepared a patch for ldsodefs.h, I'll test it diff --git a/user/jkoenig/java.mdwn b/user/jkoenig/java.mdwn index 4052f455..8e1fb097 100644 --- a/user/jkoenig/java.mdwn +++ b/user/jkoenig/java.mdwn @@ -1,628 +1,225 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] -# Java for Hurd (and vice versa) - -Contact information: - - * Full name: Jérémie Koenig - * Email: jk@jk.fr.eu.org - * IRC: jkoenig on Freenode and OFTC - -## Introductions - -I am a first year M.Sc. student -in Computer Science at University of Strasbourg (France). -My interests include capability-based security, -programming languages and formal methods -(in particular, object-capability languages and proof-carrying code). - -### Proposal summary - -This project would consist in improving Java support on Hurd. -The first part would consist in -fixing bugs and porting Java-related packages. -The second part would consist in -creating low-level Java bindings for the Hurd interfaces, -as well as libraries to make translator development easier. - -### Previous involvement - -I started contributing to Hurd last summer, -during which I participated to Google Summer of Code -as a student for the Debian project. -I worked on porting Debian-Installer to Hurd. -This project was mostly a success, -although we still have to use a special mirror for installation -with a few modified packages -and tweaked priorities -to work around some uninstallable packages -with Priority: standard. - -Shortly afterwards, -I rewrote the procfs translator -to fix some issues with memory leaks, -make it more reliable, -and improve compatibility with Linux-based tools -such as `procps` or `htop`. - -Although I have not had as much time -as I would have liked to dedicate to the Hurd -since that time, -I have continued to maintain the mirror in question, -and I have started to work -on implementing POSIX threads signal semantics in glibc. - -### Project-related skills and interests - -I have used Java mostly for university assignments. -This includes non-trivial projects -using threads and distributed programming frameworks -such as Java RMI or CORBA. -I have also used it to experiment with -Google App Engine -(web applications) -and Google Web Toolkit -(a compiler from Java to Javascript which helps with AJAX code), -and I have some limited experience with JNI -(the Java Native Interface, to link Java with C code). - -My knowledge of the Hurd and Debian GNU/Hurd is reasonable, -as the Debian-Installer and procfs projects -gave me the opportunity to fiddle with many parts of the system. - -Initially, -I started working on this project because I wanted to use -[Joe-E](http://code.google.com/p/joe-e/) -(a subset of Java) -to investigate the potential -[[applications of object-capability languages|objcap]] -in a Hurd context. -I also believe that improving Java support on Hurd -would be an important milestone. - -### Organisational matters - -I am subscribed to bug-hurd@g.o and -I do have a permanent internet connexion. - -I would be able to attend the regular IRC meetings, -and otherwise communicate with my mentor -through any means they would prefer -(though I expect email and IRC would be the most practical). -Since I'm already familiar with the Hurd, -I don't expect I would require too much time from them. - -My exams end on May 20 so I would be able to start coding -right at the beginning of the GSoC period. -Next year's term would probably begin around September 15, -so that would not be an issue either. -I expect I would work around 40 hours per week, -and my waking hours would be flexible. - -I don't have any other plans for the summer -and would not make any if my project were to be accepted. - -Full disclosure: -I also submitted a proposal to the Jikes RVM project -(which is a research-oriented Java Virtual Machine, -itself written in Java) -for implementing a new garbage collector into the MMTk subsystem. - -## Improve Java support - -### Justification - -Java is a popular language and platform used by many desktop and web -applications (mostly on the server side). As a consequence, competitive Java -support is important for any general-purpose operating system. -Better Java support would also be a prerequisite -for the second part of my proposal. - -### Current situation - -Java is currently supported on Hurd with the GNU Java suite: - - * [GCJ](http://gcc.gnu.org/java/), - the GNU Compiler for Java, is part of GCC and can compile Java - source code to Java bytecode, and both source code and bytecode to - native code; - * libgcj is the implementation of the Java runtime which GCJ uses. - It is based on [GNU Classpath](http://www.gnu.org/software/classpath/). - It includes a bytecode interpreter which enables - Java applications compiled to native code to dynamically load and execute - Java bytecode from class files. - * The gij command is a wrapper around the above-mentioned virtual machine - functionality of libgcj and can be used as a replacement for the java - command. - -However, GCJ does not work flawlessly on Hurd.r -For instance, some parts of libgcj relies on -the POSIX threads signal semantics, which are not yet implemented. -In particular, this makes ant hang waiting for child processes, -which makes some packages fail to build on Hurd -(“ant” is the “make” of the Java world). - -### Tasks - - * **Finish implementing POSIX thread semantics** in glibc (high priority). - According to POSIX, signal dispositions should be global to a process, - while signal blocking masks should be thread-specific. Signals sent to the - process as a whole are to be delivered to any thread which does not block - them. By contrast, Hurd has per-thread signal dispositions and signals - sent to a process are delivered to the main thread only. I have been - working on refactoring the glibc signal code and implementing the POSIX - semantics as a per-thread option. However, due to lack of time I have not - yet been able to test and debug my code properly. Finishing this work - would be my first task. - * **Fix further problems with GCJ on Hurd** (high priority). While I’m not - aware of any other problems with GCJ at the moment, I suspect some might - turn up as I progress with the other tasks. Fixing these problems would - also be a high-priority task. - * **Port OpenJDK 6** (medium priority). While GCJ is fine, it is not yet - 100% complete. It is also slower than OpenJDK on architectures where a - just-in-time compiler is available. Porting OpenJDK would therefore - improve Java support on Hurd in scope and quality. Besides, it would also - be a good way to test GCJ, which is used for bootstrapping by the Debian - OpenJDK packages. Also note that OpenJDK 6 is now the default Java - Runtime Environment on all released Linux-based Debian architectures; - bringing Hurd in line with this would probably be a good thing. - * **Port Eclipse and other Java applications** (low priority). Eclipse is a - popular, state-of-the-art IDE and tool suite used for Java and other - languages. It is a dependency of the Joe-E verifier (see part 3 of this - proposal). Porting Eclipse would be a good opportunity to test GCJ and - OpenJDK. - -### Deliverables - - * The glibc pthreads patch and any other fixes on the Hurd side - would be submitted upstream - * Patches against Debian source packages - required to make them build on Hurd would be submitted - to the [Debian bug tracking system](http://bugs.debian.org/). - - -## Create Java bindings for the Hurd interfaces - -### Justification - -Java is used for many applications and often taught to -introduce object-oriented programming. The fact that Java is a -garbage-collected language makes it easier to use, especially for the less -experienced programmers. Besides, its object-oriented nature is a -natural fit for the capability-based design of Hurd. -The JVM is also used as a target for many other languages, -all of which would benefit from the access provided by these bindings. - -Advantages over other garbage-collected, object-oriented languages include -performance, type safety and the possibility to compile a Java translator to -native code and -[link it statically](http://gcc.gnu.org/wiki/Statically_linking_libgcj) -using GCJ, should anyone want to use a -translator written in Java for booting. -Note that Java is -[being](http://www.linuxjournal.com/article/8757) -[used](http://oss.readytalk.com/avian/) -in this manner for embedded development. -Since GCJ can take bytecode as its input, -this expect this possibility would apply to any JVM-based language. - -Java bindings would lower the bar for newcomers -to begin experimenting with what makes Hurd unique -without being faced right away with the complexity of -low-level systems programming. - -### Tasks summary - - * Implement Java bindings for Mach - * Implement a libports-like library for Java - * Modify MIG to output Java code - * Implement libfoofs-like Java libraries - -### Design principles - -The principles I would use to guide the design -of these Java bindings would be the following ones: - - * The system should be hooked into at a low level, - to ensure that Java is a "first class citizen" - as far as the access to the Hurd's interfaces is concerned. - * At the same time, the memory safety of Java should be maintained - and extended to Mach primitives such as port names and - out-of-line memory regions. - * Higher-level interfaces should be provided as well - in order to make translator development - as easy as possible. - * A minimum amount of JNI code (ie. C code) should be used. - Most of the system should be built using Java itself - on top of a few low-level primitives. - * Hurd objects would map to Java objects. - * Using the same interfaces, - objects corresponding to local ports would be accessed directly, - and remote objects would be accessed over IPC. - -One approach used previously to interface programming languages with the Hurd -has been to create bindings for helper libraries such as libtrivfs. Instead, -for Java I would like to take a lower-level approach by providing access to -Mach primitives and extending MIG to generate Java code from the interface -description files. - -This approach would be initially more involved, and would introduces several -issues related to overcoming the "impedance mismatch" between Java and Mach. -However, once an initial implementation is done it would be easier to maintain -in the long run and we would be able to provide Java bindings for a large -percentage of the Hurd’s interfaces. - -### Bindings for Mach system calls - -In this low-level approach, my intention is to enable Java code to use Mach -system calls (in particular, mach_msg) more or less directly. This would -ensure full access to the system from Java code, but it raises a number of -issues: - - * the Java code must be able to manipulate Mach-level entities, such as port - rights or page-aligned buffers mapped outside of the garbage-collected - heap (for out-of-line transfers); - * putting together IPC messages requires control of the low-level - representation of data. - -In order to address these concerns, classes would be encapsulating these -low-level entities so that they can be referenced through normal, safe objects -from standard Java code. Bindings for Mach system calls can then be provided -in terms of these classes. Their implementation would use C code through the -Java Native Interface (JNI). - -More specifically, this functionality would be provided by the `org.gnu.mach` -package, which would contain at least the following classes: - - * `MachPort` would encapsulate a `mach_port_t`. (Some of) its constructors - would act as an interface for the `mach_port_allocate()` system call. - `MachPort` objects would also be instantiated from other parts of the JNI - C code to represent port rights received through IPC. The `deallocate()` - method would call `mach_port_deallocate()` and replace the encapsulated - port name with `MACH_PORT_DEAD`. We would recommend that users call it - when a port is no longer used, but the finalizer would also deallocate the - port when the `MachPort` object is garbage collected. - * `Buffer` would represent a page-aligned buffer allocated outside of the - Java heap, to be transferred (or having been received) as out-of-line - memory. The JNI code would would provide methods to read and write data at - an arbitrary offset (but within bounds) and would use `vm_allocate()` and - `vm_deallocate()` in the same spirit as for `MachPort` objects. - * `Message` would allow Java code to put together Mach messages. The - constructor would allocate a `byte[]` member array of a given size. - Additional methods would be provided to fill in or query the information - in the message header and additional data items, including `MachPort` and - `Buffer` objects which would be translated to the corresponding port names - and out-of-line pointers. - A global map from port names to the corresponding `MachPort` object - would probably be needed to ensure that there is a one-to-one - correspondence. - * `Syscall` would provide static JNI methods for performing system calls not - covered by the above classes, such as `mach_msg()` or - `mach_thread_self()`. These methods would accept or return `MachPort`, - `Buffer` and `Message` objects when appropriate. The associated C code - would access the contents of such objects directly in order to perform the - required unsafe operations, such as constructing `MachPort` and `Buffer` - objects directly from port names and C pointers. - -Note that careful consideration should be given to the interfaces of these -classes to avoid “safety leaks” which would compromise the safety guarantees -provided by Java. Potential problematic scenarios include the following -examples: - - * It must not be possible to write an integer at some position in a - `Message` object, and to read it back as a `MachPort` or `Buffer` object, - since this would allow unsafe access to arbitrary memory addresses and - mach port names. - * Providing the `mach_task_self()` system call would also provide access to - arbitrary addresses and ports by using the `vm_*` family of RPC operations - with the returned `MachPort` object. This means that the relevant task - operations should be provided by the `Syscall` class instead. - -Finally, access should be provided to the initial ports and file descriptors -in `_hurd_ports` and provided by the `getdport()` function, -for instance through static methods such as -`getCRDir()`, `getCWDir()`, `getProc()`, ... in a dedicated class such as -`org.gnu.hurd.InitPorts`. - -A realistic example of code based on such interfaces would be: - - import org.gnu.mach.MsgType; - import org.gnu.mach.MachPort; - import org.gnu.mach.Buffer; - import org.gnu.mach.Message; - import org.gnu.mach.Syscall; - import org.gnu.hurd.InitPorts; - - public class Hello - { - public static main(String argv[]) - /* Parent class for all Mach-related exceptions */ - throws org.gnu.mach.MachException - { - /* Allocate a reply port */ - MachPort reply = new MachPort(); - - /* Allocate an out-of-line buffer */ - Buffer data = new Buffer(MsgType.CHAR, 13); - data.writeString(0, "Hello, World!"); - - /* Craft an io_write message */ - Message msg = new Message(1024); - msg.setRemotePort(InitPorts.getdport(1)); - msg.setLocalPort(reply, Message.Type.MAKE_SEND_ONCE); - msg.setId(21000); - msg.addBuffer(data); - - /* Make the call, MACH_MSG_SEND | MACH_MSG_RECEIVE */ - Syscall.machMsg(msg, true, true, reply); - - /* Extract the returned value */ - msg.assertId(21100); - int retCode = msg.readInt(0); - int amount = msg.readInt(1); - } - } - -Should this paradigm prove insufficient, -more ideas could be borrowed from the -[`org.vmmagic`](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.151.5253&rep=rep1&type=pdf) -package used by [Jikes RVM](http://jikesrvm.org/), -a research Java virtual machine itself written in Java. - -### Generating Java stubs with MIG - -Once the basic machinery is in place to interface with Mach, Java programs -have more or less equal access to the system functionality without resorting -to more JNI code. However, as illustrated above, this access is far from -convenient. - -As a solution I would modify MIG to add the option to output Java code. MIG -would emit a Java interface, a client class able to implement the interface -given a Mach port send right, an a server class which would be able to handle -incoming messages. The class diagram below, although it is by no means -complete or exempt of any problem, illustrates the general idea: - -[[gsoc2011_classes.png]] - -This structure is somewhat reminiscent of -[Java RMI](http://en.wikipedia.org/wiki/Java_remote_method_invocation) -or similar systems, -which aim to provide more or less transparent access to remote objects. -The exact way the Java code would be generated still needs to be determined, -but basically: - - * An interface, corresponding to the header files generated by MIG, would - enumerate the operations listed in a given .defs files. Method names would - be transformed to adhere to Java conventions (for instance, - `some_random_identifier` would become `someRandomIdentifier`). - * A user class, corresponding to the `*User.c` files, - would implement this interface by doing RPC over a given MachPort object. - * A server class, corresponding to `*Server.c`, would be able to handle - incoming messages using a user-provided implementation of the interface. - (Possibly, a skeleton class providing methods which would raise - `NotImplementedException`s would be provided as well. - Users would derive from this class and override the relevant methods. - This would allow them not to implement some operations, - and would avoid pre-existing code from breaking when new operations are - introduced.) - -In order to help with the implementation of servers, some kind of library -would be needed to associate Mach receive rights with server objects and to -handle incoming messages on dedicated threads, in the spirit of libports. -This would probably require support for port sets at the level of the Mach -primitives described in the previous section. - -When possible, operations involving the transmission of send rights -of some kind would be expressed in terms of the MIG-generated interfaces -instead of `MachPort` objects. -Upon reception of a send right, -a `FooUser` object would be created -and associated with the corresponding `MachPort` object. -If the received send right corresponds to a local port -to which a server object has been associated, -this object would be used instead. -This way, -subsequent operations on the received send right -would be handled as direct method calls -instead of going through RPC mechanisms. - -Some issues will still need to be solved regarding how MIG will convert -interface description files to Java interfaces. For instance: - - * `.defs` files are not explicitly associated with a type. For instance in - the example above, MIG would have to somehow infer that io_t corresponds - to `this` in the `Io` interface. - * More generally, a correspondence between MIG and Java types would have - to be determined. Ideally this would be automated and not hardcoded - too much. - * Initially, reply port parameters would be ignored. However they may be - needed for some applications. - -So the details would need to be flushed out during the community bonding -period and as the implementation progresses. However I’m confident that a -satisfactory solution can be designed. - -Using these new features, the example above could be rewritten as: - - import org.gnu.hurd.InitPorts; - import org.gnu.hurd.Io; - import org.gnu.hurd.IoUser; - - class Hello { - static void main(String argv[]) throws ... - { - Io stdout = new IoUser(InitPorts.getdport(1)); - String hello = “Hello, World!\n”; - - int amount = stdout.write(hello.getBytes(), -1); - - /* (A retCode corresponding to an error - would be signalled as an exception.) */ - } - } - -An example of server implementation would be: - - import org.gnu.hurd.Io; - import java.util.Arrays; - - class HelloIo implements Io { - final byte[] contents = “Hello, World!\n”.getBytes(); - - int write(byte[] data, int offset) { - return SOME_ERROR_CODE; - } - - byte[] read(int offset, int amount) { - return Arrays.copyOfRange(contents, offset, - offset + amount - 1); - } - - /* ... */ - } - -A new server object could then be created with `new IoServer(new HelloIo())`, -and associated with some receive right at the level of the ports management -library. - -### Base classes for common types of translators - -Once MIG can target Java code, and a libports equivalent is available, -creating new translators in Java would be greatly facilitated. However, -we would probably want to introduce basic implementations of file system -translators in the spirit of libtrivfs or libnetfs. They could take the form -of base classes implementing the relevant MIG-generated interfaces which -would then be derived by users, -or could define a simpler interface -which would then be used by adapter classes -to implement the required ones. - -I would draw inspiration from libtrivfs and libnetfs -to design and implement similar solutions for Java. - -### Deliverables - - * A hurd-java package would contain the Java code developed - in the context of this project. - * The Java code would be documented using javadoc - and a tutorial for writing translators would be written as well. - * Modifications to MIG would be submitted upstream, - or a patched MIG package would be made available. - -The Java libraries resulting from this work, -including any MIG support classes -as well as the class files built from the MIG-generated code -for the Mach and Hurd interface definition files, -would be provided as single `hurd-java` package for -Debian GNU/Hurd. -This package would be separate from both Hurd and Mach, -so as not to impose unreasonable build dependencies on them. - -I expect I would be able to act as its maintainer in the foreseeable future, -either as an individual or as a part of the Hurd team. -Hopefully, -my code would be claimed by the Hurd project as their own, -and consequently the modifications to MIG -(which would at least conceptually depend on the Mach Java package) -could be integrated upstream. - -Since by design, -the Java code would use only a small number of stable interfaces, -it would not be subject to excessive amounts of bitrot. -Consequently, -maintenance would primarily consist in -fixing bugs as they are reported, -and adding new features as they are requested. -A large number of such requests -would mean the package is useful, -so I expect that the overall amount of work -would be correlated with the willingness of more people -to help with maintenance -should I become overwhelmed or get hit by a bus. - - -## Timeline - -The dates listed are deadlines for the associated tasks. - - * *Community bonding period.* - Discuss, refine and complete the design of the Java bindings - (in particular the MIG and "libports" parts) - * *May 23.* - Coding starts. - * *May 30.* - Finish implementing pthread signal semantics. - * *June 5.* - Port OpenJDK - * *June 12.* - Fix the remaining problems with GCJ and/or OpenJDK, - possibly port Eclipse or other big Java packages. - * *June 19.* - Create the bindings for Mach. - * *June 26.* - Work on some kind of basic Java libports - to handle receive rights. - * *July 3.* - Test, write some documentation and examples. - * *July 17 (two weeks).* - Add the Java target to MIG. - * *July 24.* - Test, write some documentation and examples. - * *August 7 (two weeks).* - Implement a modular libfoofs to help with translator development. - Try to write a basic but non-trivial translator - to evaluate the performance and ease of use of the result, - rectify any rough edges this would uncover. - * *August 22. (last two weeks)* - Polish the code and packaging, - finish writing the documentation. - - -## Conclusion - -This project is arguably ambitious. -However, I have been thinking about it for some time now -and I'm confident I would be able to accomplish most of it. - -In the event multiple language bindings projects -would be accepted, -some work could probably be done in common. -In particular, -[ArneBab](http://www.bddebian.com/~hurd-web/community/weblogs/ArneBab/2011-04-06-application-pyhurd/) -seems to favor a low-level approach for his Python bindings as I do for Java, -and I would be happy to discuss API design and coordinate MIG changes with him. -I would also have an extra month after the end of the GSoC period -before I go back to school, -which I would be able to use to finish the project -if there is some remaining work. -(Last year's rewrite of procfs was done during this period.) - -As for the project's benefits, -I believe that good support for Java -is a must-have for the Hurd. -Java bindings would also further the Hurd's agenda -of user freedom by extending this freedom to more people: -I expect the set of developers -who would be able to write Java code against a well-written libfoofs -is much larger than -those who master the intricacies of low-level systems C programming. -From a more strategic point of view, -this would also help recruit new contributors -by providing an easier path to learning the inner workings of the Hurd. - -Further developments -which would build on the results of this project -include my planned [[experiment with Joe-E|objcap]] -(which I would possibly take on as a university project next year). -Another possibility would be to reimplement some parts -of the Java standard library -directly in terms of the Hurd interfaces -instead of using the POSIX ones through glibc. -This would possibly improve the performance -of some Java applications (though probably not by much), -and would otherwise be a good project -for someone trying to get acquainted with Hurd. - -Overall, I believe this project would be fun, interesting and useful. -I hope that you will share this sentiment -and give me the opportunity to spend another summer working on Hurd. +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] +# Improve Java on Hurd (GSoC 2011) + + +## Description + +The project consists in improving Java support on Hurd. +This includes porting OpenJDK, +creating low-level Java bindings for Mach and Hurd, +as well as creating Java libraries to help with translator development. + +For details, see my original [[proposal]]. + + +## Current status + + +### Apt repository + +Modified Debian packages are available in this repository: + + deb http://jk.fr.eu.org/debian experimental/ + deb-src http://jk.fr.eu.org/debian experimental/ + + +### Glibc signal code improvements + +I have submitted +[preliminary patches](http://lists.gnu.org/archive/html/bug-hurd/2011-05/msg00182.html) +for global signal dispositions, +which I'm currently testing. +I have since fixed a few things and implemented `SA_SIGINFO` +(which is required by OpenJDK.) +My latest code is available on +[github](http://github.com/jeremie-koenig/glibc/commits/master-beware-rebase), +and modified Debian packages +are available in my apt repository. + +One question is how the new symbols introduced by my patches +should be handled. +Weak symbols turned out to be impractical, +so I'm currently considering using a Debian-specific +symbol version in the interim period (`GLIBC_2.13_DEBIAN_7` so far). +The ultimate symbol version to be used will depend on +the time at which the patches get integrated upstream, +at which point we will alias the interim version +to the new one in debian packages. + +I have modified libc0.3 to include a `deb-symbols(5)` file +(alternatively see <http://wiki.debian.org/Projects/ImprovedDpkgShlibdeps>) +so that we get an accurate libc dependency in `hurd` and other packages +when the symbols in question are pulled in. + +[[hurd/libthreads]] (cthreads library) will not be changed. There's no reason +why its behavior should change, whereas for [[libpthread]] it's needed for +conformance. Patches posted on 2011-05-25, but there's a more recent one in +the modified hurd package (adds `_hurd_sigstate_delete` and removes the weak +symbols). + +Another issue which came up with OpenJDK is the expansion +by the dynamic linker of `$ORIGIN` in the `RPATH` header, +see below. + +#### Plans + +I will submit revised series for review later this week, +as well as matching Debian patches. +I expect only the last patch (implement global dispositions) will change, +and new ones will be added on top of it. + + +##### Open Items + + * Test patches: in progress, [[jkoenig]], Svante. More volunteers welcome, + of course. + + * If [[jkoenig]] thinks it's mature enough: should ask + [[Samuel|samuelthibault]] to test these patches on the buildds. + + * Get patches reviewed (Roland?), and integrated into official sources: [!] + [[tschwinge]]. + + * Documentations bits (from [[proposal]] and elsewhere) should probably be + moved either into the appropriate glibc or Hurd documentation + files/reference manuals, or to [[glibc/signal]]. + + * `SA_SIGINFO` patche is based on [[Samuel|samuelthibault]]'s earlier work. + Thus, have him review the new patch? + + * Perhaps have a look at `SA_NOCLDWAIT`. + + * Which numeric values to use for `SA_SIGINFO` (and `SA_NOCLDWAIT`)? + + +### Port OpenJDK + +As suggested by [[tschwinge]], I have targeted OpenJDK 7 at first. +I don't expect it will be too hard to backport my patches to OpenJDK 6. +I have succeeded in building a working JIT-less ("zero") version, +although the dynamic linker issue must be worked around. +Porting Hotspot (the original just-in-time compiler of OpenJDK) +should not be too hard. +If that fails we can fall back on Shark +(a portable alternative JIT which uses LLVM). + +Complexity of porting HotSpot: probably low. The complex things should be +arch- rather than OS-specific. Not many Linux-specific interfaces used. +Garbage collection/memory management, etc. and/or most of other Linux-specific +interfaces are already dealt with for the zero build. + +The dynamic linker issue is as follows. +An executable-specific search path can be provided in the ELF RPATH header. +RPATH directories can include the special string `$ORIGIN`, +which is to be expanded to the directory the executable was loaded from. +OpenJDK's `java` command uses this feature to locate +the right `libjli.so` at runtime. +However, +on Hurd this information is not available to the dynamic linker +and as a consequence RPATH components which include `$ORIGIN` +are silently discarded. + +This can be worked around by defining +the `LD_ORIGIN_PATH` environment variable. +(which have I used to build and test OpenJDK so far.) + +#### Plans + +I intend to fix the RPATH issue +by building on [[pochu]]'s `file_exec_file_name()` +[patches](http://lists.gnu.org/archive/html/bug-hurd/2010-08/msg00023.html). + +I have succeeded in building a Hotspot-enabled `libjvm.so`, +although the current toolchain issues +([[toolchain/ELFOSABI_HURD]]) +have so far prevented me from testing it. + + +##### Open Items + + * [!] [[tschwinge]] to have a look at [[pochu]]'s `file_exec_file_name()` + patches, whether it's generally the right idea. + + * Assuming it is, continue with getting `$ORIGIN` working. + + * [!] [[Samuel|samuelthibault]]/[[tschwinge]]/[[jkoenig]]: + [[toolchain/ELFOSABI_HURD]]. + + * They seem to have a rather heavy-weight process for such projects: confer + <http://mail.openjdk.java.net/pipermail/announce/2011-January/000092.html>, + for example. Do we need this, too? + + * Eclipse + + OK for testing -- but I'd very much hope that it *just works* as soon as we + provide the required Java platform. But it may perhaps have some + Linux-specifics (needlessly?) in its basement. Is it available for Debian + GNU/kFreeBSD already? + + +### Java bindings for Mach + + +#### Plans + +(just started.) + + +##### Open Items + + * [[tschwinge]] has to read about RMI and CORBA. + + * MIG + + * Hacking [[microkernel/mach/MIG]] shouldn't be too difficult. + + * (Unless you want to make MIG's own code (that is, not the generated + code, but MIG itself) look a bit more nice, too.) ;-) + + * There are also alternatives to MIG. If there is interest, the following + could be considered: + + * FLICK ([[!GNU_Savannah_task 5723]]). [[tschwinge]] has no idea yet if + there would be any benefits over MIG, like better modularity (for the + backends)? If we feel like it, we could spend a little bit of time on + this. + + * For [[microkernel/Viengoos]], Neal has written a RPC stub generator + entirely in C Preprocessor macros. While this is obviously not + directly applicable, perhaps we can get some ideas from it. + + * Anything else that would be worth having a look at? (What are other + microkernels using?) + + * `mach_msg` + + * Seems like the right approach to [[tschwinge]], but he hasn't digested + all the pecularities yet. Will definitely need more time. + + +## Postponed + +Might get back to these as time/interest permits. + + +### GCJ + + * [[tschwinge]] has the feeling that Java in GCC (that is, GCJ) is mostly + dead? (True?) + + * Thus perhaps not too much effort should be spent with it. + + If the POSIX threads signal semantics makes it going, then great, otherwise + we should get a feeling what else is missing. + + +### Joe-E. diff --git a/user/jkoenig/java/discussion.mdwn b/user/jkoenig/java/discussion.mdwn index 0131d8d5..266a7bcc 100644 --- a/user/jkoenig/java/discussion.mdwn +++ b/user/jkoenig/java/discussion.mdwn @@ -56,114 +56,6 @@ one person's summer.) Such bits could be moved to [[open_issues]] pages, either new ones or existing ones, as applicable. -# POSIX Threads Signal Semantics - - * Great! [[tschwinge]] had a brief look, and should have a deeper one. - - * If [[jkoenig]] thinks it's mature enough: should ask Samuel to test this - (that is, only the refactoring patches for starters?) on the buildds. - - * Then: should ask Roland to review. - - * Documentations bits should probably be moved to [[glibc/signal]]. - - -## libthreads (cthreads) Integration - - * [[tschwinge]] suggests to leave them as-is? - - -## [[libpthread]] integration - - * To be done. - - -# Java - - * [[tschwinge]] has to read about RMI and CORBA. - - -# Joe-E - - * For later. - - -# GCJ - - * [[tschwinge]] has the feeling that Java in GCC (that is, GCJ) is mostly - dead? (True?) - - * Thus perhaps not too much effort should be spent with it. - - If the POSIX threads signal semantics makes it going, then great, otherwise - we should get a feeling what else is missing. - - -# OpenJDK - - * All in all, [[tschwinge]] has the feeling that a working OpenJDK will be - more useful/powerful than GCJ. - - * We need to get a feeling how difficult such an OS port will be. - - * [[jkoenig]] suggests OpenJDK 6 -- should we directly go for version 7 - instead? - - * What are the differences (regarding the OS port) between the two - versions? Or this there something even more recent to be worked upon, - for new OS ports? - - * Perhaps the different versions' OS port specific stuff is not at - all very different, so that both v6 and v7 could be done? - - * They seem to have a rather heavy-weight process for such projects: confer - <http://mail.openjdk.java.net/pipermail/announce/2011-January/000092.html>, - for example. Do we need this, too? - - -# Eclipse - -OK for testing -- but I'd very much hope that it *just works* as soon as we -provide the required Java platform. - - -# Java Bindings - - -## Design Principles - - * Generally ack. - - -### MIG - - * Hacking [[microkernel/mach/MIG]] shouldn't be too difficult. - - * (Unless you want to make MIG's own code (that is, not the generated - code, but MIG itself) look a bit more nice, too.) ;-) - - * There are also alternatives to MIG. If there is interest, the following - could be considered: - - * FLICK ([[!GNU_Savannah_task 5723]]). [[tschwinge]] has no idea yet if - there would be any benefits over MIG, like better modularity (for the - backends)? If we feel like it, we could spend a little bit of time on - this. - - * For [[microkernel/Viengoos]], Neal has written a RPC stub generator - entirely in C Preprocessor macros. While this is obviously not - directly applicable, perhaps we can get some ideas from it. - - * Anything else that would be worth having a look at? (What are other - microkernels using?) - - -### `mach_msg` - - * Seems like the right approach to [[tschwinge]], but hasn't digested all the - pecularities yet. Will definitely need more time. - - # GSoC Site Discussion * Discussion items from diff --git a/user/jkoenig/java/proposal.mdwn b/user/jkoenig/java/proposal.mdwn new file mode 100644 index 00000000..4052f455 --- /dev/null +++ b/user/jkoenig/java/proposal.mdwn @@ -0,0 +1,628 @@ + +# Java for Hurd (and vice versa) + +Contact information: + + * Full name: Jérémie Koenig + * Email: jk@jk.fr.eu.org + * IRC: jkoenig on Freenode and OFTC + +## Introductions + +I am a first year M.Sc. student +in Computer Science at University of Strasbourg (France). +My interests include capability-based security, +programming languages and formal methods +(in particular, object-capability languages and proof-carrying code). + +### Proposal summary + +This project would consist in improving Java support on Hurd. +The first part would consist in +fixing bugs and porting Java-related packages. +The second part would consist in +creating low-level Java bindings for the Hurd interfaces, +as well as libraries to make translator development easier. + +### Previous involvement + +I started contributing to Hurd last summer, +during which I participated to Google Summer of Code +as a student for the Debian project. +I worked on porting Debian-Installer to Hurd. +This project was mostly a success, +although we still have to use a special mirror for installation +with a few modified packages +and tweaked priorities +to work around some uninstallable packages +with Priority: standard. + +Shortly afterwards, +I rewrote the procfs translator +to fix some issues with memory leaks, +make it more reliable, +and improve compatibility with Linux-based tools +such as `procps` or `htop`. + +Although I have not had as much time +as I would have liked to dedicate to the Hurd +since that time, +I have continued to maintain the mirror in question, +and I have started to work +on implementing POSIX threads signal semantics in glibc. + +### Project-related skills and interests + +I have used Java mostly for university assignments. +This includes non-trivial projects +using threads and distributed programming frameworks +such as Java RMI or CORBA. +I have also used it to experiment with +Google App Engine +(web applications) +and Google Web Toolkit +(a compiler from Java to Javascript which helps with AJAX code), +and I have some limited experience with JNI +(the Java Native Interface, to link Java with C code). + +My knowledge of the Hurd and Debian GNU/Hurd is reasonable, +as the Debian-Installer and procfs projects +gave me the opportunity to fiddle with many parts of the system. + +Initially, +I started working on this project because I wanted to use +[Joe-E](http://code.google.com/p/joe-e/) +(a subset of Java) +to investigate the potential +[[applications of object-capability languages|objcap]] +in a Hurd context. +I also believe that improving Java support on Hurd +would be an important milestone. + +### Organisational matters + +I am subscribed to bug-hurd@g.o and +I do have a permanent internet connexion. + +I would be able to attend the regular IRC meetings, +and otherwise communicate with my mentor +through any means they would prefer +(though I expect email and IRC would be the most practical). +Since I'm already familiar with the Hurd, +I don't expect I would require too much time from them. + +My exams end on May 20 so I would be able to start coding +right at the beginning of the GSoC period. +Next year's term would probably begin around September 15, +so that would not be an issue either. +I expect I would work around 40 hours per week, +and my waking hours would be flexible. + +I don't have any other plans for the summer +and would not make any if my project were to be accepted. + +Full disclosure: +I also submitted a proposal to the Jikes RVM project +(which is a research-oriented Java Virtual Machine, +itself written in Java) +for implementing a new garbage collector into the MMTk subsystem. + +## Improve Java support + +### Justification + +Java is a popular language and platform used by many desktop and web +applications (mostly on the server side). As a consequence, competitive Java +support is important for any general-purpose operating system. +Better Java support would also be a prerequisite +for the second part of my proposal. + +### Current situation + +Java is currently supported on Hurd with the GNU Java suite: + + * [GCJ](http://gcc.gnu.org/java/), + the GNU Compiler for Java, is part of GCC and can compile Java + source code to Java bytecode, and both source code and bytecode to + native code; + * libgcj is the implementation of the Java runtime which GCJ uses. + It is based on [GNU Classpath](http://www.gnu.org/software/classpath/). + It includes a bytecode interpreter which enables + Java applications compiled to native code to dynamically load and execute + Java bytecode from class files. + * The gij command is a wrapper around the above-mentioned virtual machine + functionality of libgcj and can be used as a replacement for the java + command. + +However, GCJ does not work flawlessly on Hurd.r +For instance, some parts of libgcj relies on +the POSIX threads signal semantics, which are not yet implemented. +In particular, this makes ant hang waiting for child processes, +which makes some packages fail to build on Hurd +(“ant” is the “make” of the Java world). + +### Tasks + + * **Finish implementing POSIX thread semantics** in glibc (high priority). + According to POSIX, signal dispositions should be global to a process, + while signal blocking masks should be thread-specific. Signals sent to the + process as a whole are to be delivered to any thread which does not block + them. By contrast, Hurd has per-thread signal dispositions and signals + sent to a process are delivered to the main thread only. I have been + working on refactoring the glibc signal code and implementing the POSIX + semantics as a per-thread option. However, due to lack of time I have not + yet been able to test and debug my code properly. Finishing this work + would be my first task. + * **Fix further problems with GCJ on Hurd** (high priority). While I’m not + aware of any other problems with GCJ at the moment, I suspect some might + turn up as I progress with the other tasks. Fixing these problems would + also be a high-priority task. + * **Port OpenJDK 6** (medium priority). While GCJ is fine, it is not yet + 100% complete. It is also slower than OpenJDK on architectures where a + just-in-time compiler is available. Porting OpenJDK would therefore + improve Java support on Hurd in scope and quality. Besides, it would also + be a good way to test GCJ, which is used for bootstrapping by the Debian + OpenJDK packages. Also note that OpenJDK 6 is now the default Java + Runtime Environment on all released Linux-based Debian architectures; + bringing Hurd in line with this would probably be a good thing. + * **Port Eclipse and other Java applications** (low priority). Eclipse is a + popular, state-of-the-art IDE and tool suite used for Java and other + languages. It is a dependency of the Joe-E verifier (see part 3 of this + proposal). Porting Eclipse would be a good opportunity to test GCJ and + OpenJDK. + +### Deliverables + + * The glibc pthreads patch and any other fixes on the Hurd side + would be submitted upstream + * Patches against Debian source packages + required to make them build on Hurd would be submitted + to the [Debian bug tracking system](http://bugs.debian.org/). + + +## Create Java bindings for the Hurd interfaces + +### Justification + +Java is used for many applications and often taught to +introduce object-oriented programming. The fact that Java is a +garbage-collected language makes it easier to use, especially for the less +experienced programmers. Besides, its object-oriented nature is a +natural fit for the capability-based design of Hurd. +The JVM is also used as a target for many other languages, +all of which would benefit from the access provided by these bindings. + +Advantages over other garbage-collected, object-oriented languages include +performance, type safety and the possibility to compile a Java translator to +native code and +[link it statically](http://gcc.gnu.org/wiki/Statically_linking_libgcj) +using GCJ, should anyone want to use a +translator written in Java for booting. +Note that Java is +[being](http://www.linuxjournal.com/article/8757) +[used](http://oss.readytalk.com/avian/) +in this manner for embedded development. +Since GCJ can take bytecode as its input, +this expect this possibility would apply to any JVM-based language. + +Java bindings would lower the bar for newcomers +to begin experimenting with what makes Hurd unique +without being faced right away with the complexity of +low-level systems programming. + +### Tasks summary + + * Implement Java bindings for Mach + * Implement a libports-like library for Java + * Modify MIG to output Java code + * Implement libfoofs-like Java libraries + +### Design principles + +The principles I would use to guide the design +of these Java bindings would be the following ones: + + * The system should be hooked into at a low level, + to ensure that Java is a "first class citizen" + as far as the access to the Hurd's interfaces is concerned. + * At the same time, the memory safety of Java should be maintained + and extended to Mach primitives such as port names and + out-of-line memory regions. + * Higher-level interfaces should be provided as well + in order to make translator development + as easy as possible. + * A minimum amount of JNI code (ie. C code) should be used. + Most of the system should be built using Java itself + on top of a few low-level primitives. + * Hurd objects would map to Java objects. + * Using the same interfaces, + objects corresponding to local ports would be accessed directly, + and remote objects would be accessed over IPC. + +One approach used previously to interface programming languages with the Hurd +has been to create bindings for helper libraries such as libtrivfs. Instead, +for Java I would like to take a lower-level approach by providing access to +Mach primitives and extending MIG to generate Java code from the interface +description files. + +This approach would be initially more involved, and would introduces several +issues related to overcoming the "impedance mismatch" between Java and Mach. +However, once an initial implementation is done it would be easier to maintain +in the long run and we would be able to provide Java bindings for a large +percentage of the Hurd’s interfaces. + +### Bindings for Mach system calls + +In this low-level approach, my intention is to enable Java code to use Mach +system calls (in particular, mach_msg) more or less directly. This would +ensure full access to the system from Java code, but it raises a number of +issues: + + * the Java code must be able to manipulate Mach-level entities, such as port + rights or page-aligned buffers mapped outside of the garbage-collected + heap (for out-of-line transfers); + * putting together IPC messages requires control of the low-level + representation of data. + +In order to address these concerns, classes would be encapsulating these +low-level entities so that they can be referenced through normal, safe objects +from standard Java code. Bindings for Mach system calls can then be provided +in terms of these classes. Their implementation would use C code through the +Java Native Interface (JNI). + +More specifically, this functionality would be provided by the `org.gnu.mach` +package, which would contain at least the following classes: + + * `MachPort` would encapsulate a `mach_port_t`. (Some of) its constructors + would act as an interface for the `mach_port_allocate()` system call. + `MachPort` objects would also be instantiated from other parts of the JNI + C code to represent port rights received through IPC. The `deallocate()` + method would call `mach_port_deallocate()` and replace the encapsulated + port name with `MACH_PORT_DEAD`. We would recommend that users call it + when a port is no longer used, but the finalizer would also deallocate the + port when the `MachPort` object is garbage collected. + * `Buffer` would represent a page-aligned buffer allocated outside of the + Java heap, to be transferred (or having been received) as out-of-line + memory. The JNI code would would provide methods to read and write data at + an arbitrary offset (but within bounds) and would use `vm_allocate()` and + `vm_deallocate()` in the same spirit as for `MachPort` objects. + * `Message` would allow Java code to put together Mach messages. The + constructor would allocate a `byte[]` member array of a given size. + Additional methods would be provided to fill in or query the information + in the message header and additional data items, including `MachPort` and + `Buffer` objects which would be translated to the corresponding port names + and out-of-line pointers. + A global map from port names to the corresponding `MachPort` object + would probably be needed to ensure that there is a one-to-one + correspondence. + * `Syscall` would provide static JNI methods for performing system calls not + covered by the above classes, such as `mach_msg()` or + `mach_thread_self()`. These methods would accept or return `MachPort`, + `Buffer` and `Message` objects when appropriate. The associated C code + would access the contents of such objects directly in order to perform the + required unsafe operations, such as constructing `MachPort` and `Buffer` + objects directly from port names and C pointers. + +Note that careful consideration should be given to the interfaces of these +classes to avoid “safety leaks” which would compromise the safety guarantees +provided by Java. Potential problematic scenarios include the following +examples: + + * It must not be possible to write an integer at some position in a + `Message` object, and to read it back as a `MachPort` or `Buffer` object, + since this would allow unsafe access to arbitrary memory addresses and + mach port names. + * Providing the `mach_task_self()` system call would also provide access to + arbitrary addresses and ports by using the `vm_*` family of RPC operations + with the returned `MachPort` object. This means that the relevant task + operations should be provided by the `Syscall` class instead. + +Finally, access should be provided to the initial ports and file descriptors +in `_hurd_ports` and provided by the `getdport()` function, +for instance through static methods such as +`getCRDir()`, `getCWDir()`, `getProc()`, ... in a dedicated class such as +`org.gnu.hurd.InitPorts`. + +A realistic example of code based on such interfaces would be: + + import org.gnu.mach.MsgType; + import org.gnu.mach.MachPort; + import org.gnu.mach.Buffer; + import org.gnu.mach.Message; + import org.gnu.mach.Syscall; + import org.gnu.hurd.InitPorts; + + public class Hello + { + public static main(String argv[]) + /* Parent class for all Mach-related exceptions */ + throws org.gnu.mach.MachException + { + /* Allocate a reply port */ + MachPort reply = new MachPort(); + + /* Allocate an out-of-line buffer */ + Buffer data = new Buffer(MsgType.CHAR, 13); + data.writeString(0, "Hello, World!"); + + /* Craft an io_write message */ + Message msg = new Message(1024); + msg.setRemotePort(InitPorts.getdport(1)); + msg.setLocalPort(reply, Message.Type.MAKE_SEND_ONCE); + msg.setId(21000); + msg.addBuffer(data); + + /* Make the call, MACH_MSG_SEND | MACH_MSG_RECEIVE */ + Syscall.machMsg(msg, true, true, reply); + + /* Extract the returned value */ + msg.assertId(21100); + int retCode = msg.readInt(0); + int amount = msg.readInt(1); + } + } + +Should this paradigm prove insufficient, +more ideas could be borrowed from the +[`org.vmmagic`](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.151.5253&rep=rep1&type=pdf) +package used by [Jikes RVM](http://jikesrvm.org/), +a research Java virtual machine itself written in Java. + +### Generating Java stubs with MIG + +Once the basic machinery is in place to interface with Mach, Java programs +have more or less equal access to the system functionality without resorting +to more JNI code. However, as illustrated above, this access is far from +convenient. + +As a solution I would modify MIG to add the option to output Java code. MIG +would emit a Java interface, a client class able to implement the interface +given a Mach port send right, an a server class which would be able to handle +incoming messages. The class diagram below, although it is by no means +complete or exempt of any problem, illustrates the general idea: + +[[gsoc2011_classes.png]] + +This structure is somewhat reminiscent of +[Java RMI](http://en.wikipedia.org/wiki/Java_remote_method_invocation) +or similar systems, +which aim to provide more or less transparent access to remote objects. +The exact way the Java code would be generated still needs to be determined, +but basically: + + * An interface, corresponding to the header files generated by MIG, would + enumerate the operations listed in a given .defs files. Method names would + be transformed to adhere to Java conventions (for instance, + `some_random_identifier` would become `someRandomIdentifier`). + * A user class, corresponding to the `*User.c` files, + would implement this interface by doing RPC over a given MachPort object. + * A server class, corresponding to `*Server.c`, would be able to handle + incoming messages using a user-provided implementation of the interface. + (Possibly, a skeleton class providing methods which would raise + `NotImplementedException`s would be provided as well. + Users would derive from this class and override the relevant methods. + This would allow them not to implement some operations, + and would avoid pre-existing code from breaking when new operations are + introduced.) + +In order to help with the implementation of servers, some kind of library +would be needed to associate Mach receive rights with server objects and to +handle incoming messages on dedicated threads, in the spirit of libports. +This would probably require support for port sets at the level of the Mach +primitives described in the previous section. + +When possible, operations involving the transmission of send rights +of some kind would be expressed in terms of the MIG-generated interfaces +instead of `MachPort` objects. +Upon reception of a send right, +a `FooUser` object would be created +and associated with the corresponding `MachPort` object. +If the received send right corresponds to a local port +to which a server object has been associated, +this object would be used instead. +This way, +subsequent operations on the received send right +would be handled as direct method calls +instead of going through RPC mechanisms. + +Some issues will still need to be solved regarding how MIG will convert +interface description files to Java interfaces. For instance: + + * `.defs` files are not explicitly associated with a type. For instance in + the example above, MIG would have to somehow infer that io_t corresponds + to `this` in the `Io` interface. + * More generally, a correspondence between MIG and Java types would have + to be determined. Ideally this would be automated and not hardcoded + too much. + * Initially, reply port parameters would be ignored. However they may be + needed for some applications. + +So the details would need to be flushed out during the community bonding +period and as the implementation progresses. However I’m confident that a +satisfactory solution can be designed. + +Using these new features, the example above could be rewritten as: + + import org.gnu.hurd.InitPorts; + import org.gnu.hurd.Io; + import org.gnu.hurd.IoUser; + + class Hello { + static void main(String argv[]) throws ... + { + Io stdout = new IoUser(InitPorts.getdport(1)); + String hello = “Hello, World!\n”; + + int amount = stdout.write(hello.getBytes(), -1); + + /* (A retCode corresponding to an error + would be signalled as an exception.) */ + } + } + +An example of server implementation would be: + + import org.gnu.hurd.Io; + import java.util.Arrays; + + class HelloIo implements Io { + final byte[] contents = “Hello, World!\n”.getBytes(); + + int write(byte[] data, int offset) { + return SOME_ERROR_CODE; + } + + byte[] read(int offset, int amount) { + return Arrays.copyOfRange(contents, offset, + offset + amount - 1); + } + + /* ... */ + } + +A new server object could then be created with `new IoServer(new HelloIo())`, +and associated with some receive right at the level of the ports management +library. + +### Base classes for common types of translators + +Once MIG can target Java code, and a libports equivalent is available, +creating new translators in Java would be greatly facilitated. However, +we would probably want to introduce basic implementations of file system +translators in the spirit of libtrivfs or libnetfs. They could take the form +of base classes implementing the relevant MIG-generated interfaces which +would then be derived by users, +or could define a simpler interface +which would then be used by adapter classes +to implement the required ones. + +I would draw inspiration from libtrivfs and libnetfs +to design and implement similar solutions for Java. + +### Deliverables + + * A hurd-java package would contain the Java code developed + in the context of this project. + * The Java code would be documented using javadoc + and a tutorial for writing translators would be written as well. + * Modifications to MIG would be submitted upstream, + or a patched MIG package would be made available. + +The Java libraries resulting from this work, +including any MIG support classes +as well as the class files built from the MIG-generated code +for the Mach and Hurd interface definition files, +would be provided as single `hurd-java` package for +Debian GNU/Hurd. +This package would be separate from both Hurd and Mach, +so as not to impose unreasonable build dependencies on them. + +I expect I would be able to act as its maintainer in the foreseeable future, +either as an individual or as a part of the Hurd team. +Hopefully, +my code would be claimed by the Hurd project as their own, +and consequently the modifications to MIG +(which would at least conceptually depend on the Mach Java package) +could be integrated upstream. + +Since by design, +the Java code would use only a small number of stable interfaces, +it would not be subject to excessive amounts of bitrot. +Consequently, +maintenance would primarily consist in +fixing bugs as they are reported, +and adding new features as they are requested. +A large number of such requests +would mean the package is useful, +so I expect that the overall amount of work +would be correlated with the willingness of more people +to help with maintenance +should I become overwhelmed or get hit by a bus. + + +## Timeline + +The dates listed are deadlines for the associated tasks. + + * *Community bonding period.* + Discuss, refine and complete the design of the Java bindings + (in particular the MIG and "libports" parts) + * *May 23.* + Coding starts. + * *May 30.* + Finish implementing pthread signal semantics. + * *June 5.* + Port OpenJDK + * *June 12.* + Fix the remaining problems with GCJ and/or OpenJDK, + possibly port Eclipse or other big Java packages. + * *June 19.* + Create the bindings for Mach. + * *June 26.* + Work on some kind of basic Java libports + to handle receive rights. + * *July 3.* + Test, write some documentation and examples. + * *July 17 (two weeks).* + Add the Java target to MIG. + * *July 24.* + Test, write some documentation and examples. + * *August 7 (two weeks).* + Implement a modular libfoofs to help with translator development. + Try to write a basic but non-trivial translator + to evaluate the performance and ease of use of the result, + rectify any rough edges this would uncover. + * *August 22. (last two weeks)* + Polish the code and packaging, + finish writing the documentation. + + +## Conclusion + +This project is arguably ambitious. +However, I have been thinking about it for some time now +and I'm confident I would be able to accomplish most of it. + +In the event multiple language bindings projects +would be accepted, +some work could probably be done in common. +In particular, +[ArneBab](http://www.bddebian.com/~hurd-web/community/weblogs/ArneBab/2011-04-06-application-pyhurd/) +seems to favor a low-level approach for his Python bindings as I do for Java, +and I would be happy to discuss API design and coordinate MIG changes with him. +I would also have an extra month after the end of the GSoC period +before I go back to school, +which I would be able to use to finish the project +if there is some remaining work. +(Last year's rewrite of procfs was done during this period.) + +As for the project's benefits, +I believe that good support for Java +is a must-have for the Hurd. +Java bindings would also further the Hurd's agenda +of user freedom by extending this freedom to more people: +I expect the set of developers +who would be able to write Java code against a well-written libfoofs +is much larger than +those who master the intricacies of low-level systems C programming. +From a more strategic point of view, +this would also help recruit new contributors +by providing an easier path to learning the inner workings of the Hurd. + +Further developments +which would build on the results of this project +include my planned [[experiment with Joe-E|objcap]] +(which I would possibly take on as a university project next year). +Another possibility would be to reimplement some parts +of the Java standard library +directly in terms of the Hurd interfaces +instead of using the POSIX ones through glibc. +This would possibly improve the performance +of some Java applications (though probably not by much), +and would otherwise be a good project +for someone trying to get acquainted with Hurd. + +Overall, I believe this project would be fun, interesting and useful. +I hope that you will share this sentiment +and give me the opportunity to spend another summer working on Hurd. + |