From e0d173f99e59a969333e56ebbd30a2bebb771592 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Tue, 31 May 2011 19:24:23 +0200 Subject: irc: Regular meetings on Wednesdays. General streamlining. --- irc.mdwn | 65 +++++++++++++++++++++++----------------------------------------- 1 file changed, 23 insertions(+), 42 deletions(-) diff --git a/irc.mdwn b/irc.mdwn index 455641ea..66f3c4c4 100644 --- a/irc.mdwn +++ b/irc.mdwn @@ -11,41 +11,22 @@ License|/fdl]]."]]"""]] [[!meta title="IRC"]] -While all official development takes place on the mailing lists and the Savannah trackers, -a lot of discussions are had on IRC as well. Everybody is welcome to join and follow these channels, but please -respect the below guidelines if you want to participate. - -# Asking Questions - -[[!inline pages=asking_questions raw=yes feeds=no]] - -# Staying On-Topic +There are some IRC channels where Hurd topics are discussed. -Please try to stay on topic. +Everybody is welcome to join and participate in the discussions, but please +respect the below guidelines if you want to participate. -* emacs vs. vi **is not** on topic -* If it is appropriate for a **slashdot comment**, it's **not appropriate** - here -* why GNU sucks is **off-topic** -* when the next release of the Hurd will be **is inappropriate** -* you should not advocate your favorite **GNU/Linux** ditribution +Please try to stay on topic. We're not interested in Emacs vs. vi discussions, +we don't want to hear why you think that the GNU project sucks, or what your +favorite GNU/Linux distribution is. -# Pasting Logs +[[!inline pages=asking_questions raw=yes feeds=no]] Sometimes providing a log or some other excerpt of text can -help solve a problem or answer a question. **Do not** paste -the log in the channel itself. Instead use a +help solve a problem or answer a question. Do not paste +the log in the channel itself if it's more than a few lines. Instead use a [paste bin](http://paste.debian.net). -# Rich Text - -Don't use it. Don't use colors. Don't use bold. Don't use emphasis. - -# Greeting - -If you never contribute to the discussion, there is no need -to always greet the channel when you enter and before leave. - # Regular Meetings @@ -54,33 +35,33 @@ Starting in early 2008, there have been regular IRC meetings held between the (now former) [[Google Summer of Code|community/gsoc]] students and their mentors. These meetings turned out to considerably help student-mentor interactions, and other developers regularely took part, too. - Currently, the meetings take place in the **`#hurd` channel every -Tuesday at 19:00 UTC** and are open to any interested party. So, +Wednesday at 19:00 UTC** and are open to any interested party. So, everyone, take your chance to chat with GNU Hurd developers! # Channels -All Hurd IRC channels are hosted on [Freenode.net](http://freenode.net/). +## Freenode + +`irc.freenode.net`, + + * `#hurd`, the official GNU Hurd IRC channel. Some of the Hurd developers + and users hang out there. [Logs](http://richtlijn.be/~larstiq/hurd/). + + * `#hurdfr`, the French chapter. -* #hurd - The official Hurd IRC channel. Some of the Hurd developers and users hang out there, and discussions about GNU Hurd, GNU/Hurd and -Debian GNU/Hurd are had there. + * `#archhurd`, [[hurd/running/Arch_Hurd]]. + [Logs](http://files.archhurd.org/irclogs). -Local user channels include: -* #hurd-it - Italian -* #hurd-es - Spanish -* #hurdfr - French -* #hurd-de - German -* #hurd.in - Indian regional languages(but primarily English) +## OFTC -# Channel logs +`irc.oftc.net`, -* [#hurd logs](http://richtlijn.be/~larstiq/hurd/) + * `#debian-hurd`, [[Debian GNU/Hurd|hurd/running/debian]]. # Related -- cgit v1.3.1 From 8fba1a1ef076b60e10f03ccdf5d1e8714bd9c16a Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Tue, 31 May 2011 19:37:14 +0200 Subject: community/meetings/ghm2011: New. --- community/meetings.mdwn | 13 +++++++++---- community/meetings/ghm2010.mdwn | 4 ++-- community/meetings/ghm2011.mdwn | 13 +++++++++++++ contributing/web_pages/news/moth_next.mdwn | 3 +++ 4 files changed, 27 insertions(+), 6 deletions(-) create mode 100644 community/meetings/ghm2011.mdwn diff --git a/community/meetings.mdwn b/community/meetings.mdwn index 939d5269..6c15d144 100644 --- a/community/meetings.mdwn +++ b/community/meetings.mdwn @@ -1,5 +1,5 @@ -[[!meta copyright="Copyright © 2007, 2008, 2009, 2010 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2007, 2008, 2009, 2010, 2011 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -13,13 +13,18 @@ License|/fdl]]."]]"""]] # Upcoming - * [[FOSDEM_2011]] + * [[GNU Hackers Meeting, 2011, Paris|ghm2011]] + +## In the Future + * [[Self-organised]] + # Past + * [[FOSDEM_2011]] * [[DebConf10]] - * [[GNU Hackers Meeting in the Hague 2010|ghm2010]] + * [[GNU Hackers Meeting, 2010, Den Haag|ghm2010]] * [[FOSDEM_2010]] * [[EuroSys_2009]] * [[FOSDEM_2008]] diff --git a/community/meetings/ghm2010.mdwn b/community/meetings/ghm2010.mdwn index b7d7e880..c8a9d8c3 100644 --- a/community/meetings/ghm2010.mdwn +++ b/community/meetings/ghm2010.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -8,7 +8,7 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -[[!meta title="GNU Hackers Meeting in the Hague 2010"]] +[[!meta title="GNU Hackers Meeting, 2010, Den Haag"]] diff --git a/community/meetings/ghm2011.mdwn b/community/meetings/ghm2011.mdwn new file mode 100644 index 00000000..7a2df8a0 --- /dev/null +++ b/community/meetings/ghm2011.mdwn @@ -0,0 +1,13 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="GNU Hackers Meeting, 2011, Paris"]] + + diff --git a/contributing/web_pages/news/moth_next.mdwn b/contributing/web_pages/news/moth_next.mdwn index 4c8b9969..dafb12b5 100644 --- a/contributing/web_pages/news/moth_next.mdwn +++ b/contributing/web_pages/news/moth_next.mdwn @@ -39,6 +39,9 @@ else=" * Plan for Debian GNU/Hurd: Get over XX% of packages ported and become a full Debian project till YYYY-MM. +* [[community/meetings/ghm2011]] + + This month [hurd hacker] [item] Also … -- cgit v1.3.1 From 1cc4f789bb83cec5c8139da75d799b136a8441b5 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Fri, 10 Jun 2011 12:37:36 +0200 Subject: open_issues/nightly_builds: Hudson is now Jenkins, or similar. --- open_issues/nightly_builds.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/open_issues/nightly_builds.mdwn b/open_issues/nightly_builds.mdwn index 5d1257fb..6eef7d19 100644 --- a/open_issues/nightly_builds.mdwn +++ b/open_issues/nightly_builds.mdwn @@ -20,7 +20,7 @@ Resources: * As reported in the [[news/2010-05-31]] news, there's Hydra doing nightly builds / Nix packages. - * + * , * -- cgit v1.3.1 From 21f7f80cd190b56a14e1faf2ba24c7137d3faef5 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Fri, 10 Jun 2011 18:47:19 +0200 Subject: More Git fun. --- open_issues/git-core-2.mdwn | 37 ++++++++++++++++++++++++++++++++- open_issues/git_nfs_mmap.mdwn | 48 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+), 1 deletion(-) create mode 100644 open_issues/git_nfs_mmap.mdwn diff --git a/open_issues/git-core-2.mdwn b/open_issues/git-core-2.mdwn index 2cac56f2..a1fdbac1 100644 --- a/open_issues/git-core-2.mdwn +++ b/open_issues/git-core-2.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -98,3 +98,38 @@ differences to HEAD. fatal: cannot pread pack file: Interrupted system call fatal: index-pack failed error: Could not fetch savannah + +--- + +2011-06-10, coulomb.SCHWINGE, checking out [[binutils]]' master branch, +starting from an empty working directory (after an external `git push`): + + $ git checkout -f + fatal: cannot create directory at 'gas/testsuite/gas/bfin': Interrupted system call + $ git checkout -f + error: unable to create file gas/testsuite/gas/i386/ilp32/x86-64-sse4_1-intel.d (File exists) + warning: unable to unlink gas/testsuite/gas/m68k-coff: Operation not permitted + fatal: cannot create directory at 'gas/testsuite/gas/m68k-coff': Operation not permitted + $ git checkout -f + error: unable to create file gas/testsuite/gas/h8300/h8300.exp (File exists) + error: unable to create file gas/testsuite/gas/i386/x86-64-addr32-intel.d (File exists) + error: unable to create file gas/testsuite/gas/ia64/secname.d (File exists) + error: unable to create file gas/testsuite/gas/m68k/pr11676.s (File exists) + Checking out files: 100% (12315/12315), done. + $ git status + # On branch master + # Changes not staged for commit: + # (use "git add ..." to update what will be committed) + # (use "git checkout -- ..." to discard changes in working directory) + # + # modified: gas/testsuite/gas/h8300/h8300.exp + # modified: gas/testsuite/gas/i386/x86-64-addr32-intel.d + # modified: gas/testsuite/gas/ia64/secname.d + # modified: gas/testsuite/gas/m68k/pr11676.s + # + no changes added to commit (use "git add" and/or "git commit -a") + $ rm gas/testsuite/gas/h8300/h8300.exp gas/testsuite/gas/i386/x86-64-addr32-intel.d gas/testsuite/gas/ia64/secname.d gas/testsuite/gas/m68k/pr11676.s + $ git checkout -f + $ git status + # On branch master + nothing to commit (working directory clean) diff --git a/open_issues/git_nfs_mmap.mdwn b/open_issues/git_nfs_mmap.mdwn new file mode 100644 index 00000000..b7e86491 --- /dev/null +++ b/open_issues/git_nfs_mmap.mdwn @@ -0,0 +1,48 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd open_issue_porting]] + + $ git-new-workdir /media/kepler-data/home/thomas/tmp/source/binutils/git master master + fatal: Out of memory? mmap failed: No such device + $ echo $? + 128 + $ showtrans /media/kepler-data + /hurd/nfs kepler.schwinge.homeip.net:/media/data + +With `sh -x`: + + [...] + + ln -s /media/kepler-data/home/thomas/tmp/source/binutils/git/.git/remotes master/.git/remotes + + ln -s /media/kepler-data/home/thomas/tmp/source/binutils/git/.git/rr-cache master/.git/rr-cache + + ln -s /media/kepler-data/home/thomas/tmp/source/binutils/git/.git/svn master/.git/svn + + cd master + + cp /media/kepler-data/home/thomas/tmp/source/binutils/git/.git/HEAD .git/HEAD + + git checkout -f master + fatal: Out of memory? mmap failed: No such device + +As one can easily guess (and confirm with [[hurd/debugging/rpctrace]]), `git` +tries to [[glibc/mmap]] a file via the [[hurd/translator/nfs]] translator, this +fails, and it isn't prepared to cope with that: + + [...] + 88->dir_lookup (".git/objects/pack/pack-37ca560e7877fa0cc6e5ddcd556aa73e5a3e3f40.idx" 2049 0) = 0 3 "/media/kepler-data/home/thomas/tmp/source/binutils/git/.git/objects/pack/pack-37" (null) + 62->dir_lookup ("media/kepler-data/home/thomas/tmp/source/binutils/git/.git/objects/pack/pack-37c" 2049 0) = 0 1 "/home/thomas/tmp/source/binutils/git/.git/objects/pack/pack-37ca560e7877fa0cc6e5" 61 + 61->dir_lookup ("home/thomas/tmp/source/binutils/git/.git/objects/pack/pack-37ca560e7877fa0cc6e5d" 2049 0) = 0 1 "" 84 + task3741-> 3206 (pn{ 33}) = 0 + 84->term_getctty () = 0xfffffed1 ((ipc/mig) bad request message ID) + 84->io_stat_request () = 0 {1 704 0 36308992 0 0 -1 33060 1 1000 1000 4712 0 1307711395 0 1307657003 0 1307657003 0 4096 16 0 1000 0 0 100663296 1836017780 29537 0 0 0 0} + 84->io_map_request () = 0x4000002d (Operation not supported) + 84->io_map_request () = 0x4000002d (Operation not supported) + 76->io_write_request ("fatal: Out of memory? mmap failed: No such device + " -1) = 0 50 + 64->proc_mark_exit_request (32768 0) = 0 + task3741-> 2008 () = 0 + Child 3741 exited with 128 -- cgit v1.3.1 From 3a7f9cfe8334a67de82935865c78fad451cd7bcd Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Fri, 10 Jun 2011 23:29:54 +0200 Subject: open_issues/ext2fs_file_corruption: New. --- open_issues/ext2fs_file_corruption.mdwn | 199 ++++++++++++++++++++++++++++++++ 1 file changed, 199 insertions(+) create mode 100644 open_issues/ext2fs_file_corruption.mdwn diff --git a/open_issues/ext2fs_file_corruption.mdwn b/open_issues/ext2fs_file_corruption.mdwn new file mode 100644 index 00000000..2af35bb5 --- /dev/null +++ b/open_issues/ext2fs_file_corruption.mdwn @@ -0,0 +1,199 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +[[!toc]] + + +# Log + +Create two 5 GiB partitions. + + # # Would default to 256 byte inodes otherwise. + # mke2fs -I 128 -o linux /dev/hd1s1 + [...] + # mke2fs -o hurd /dev/hd1s2 + [...] + +The only difference (apart from UUID, time stamps, hash seeds) in ext2 file +system meta data, according to `dumpe2fs -h`: + + [...] + -Filesystem features: ext_attr resize_inode dir_index filetype sparse_super large_file + +Filesystem features: ext_attr resize_inode dir_index sparse_super large_file + Filesystem flags: signed_directory_hash + Default mount options: (none) + Filesystem state: clean + Errors behavior: Continue + -Filesystem OS type: Linux + +Filesystem OS type: Hurd + [...] + +According to *mke2fs(8)*: + +> *filetype* +> Store file type information in directory entries. + + # settrans -c /media/hd1s1 /hurd/ext2fs /dev/hd1s1 + # settrans -c /media/hd1s2 /hurd/ext2fs /dev/hd1s2 + # mkdir /media/hd1s{1,2}/tmp + # chmod 1777 /media/hd1s{1,2}/tmp + + $ git-new-workdir ~/tmp/binutils/git /media/hd1s1/tmp/master master + error: unable to create file gas/testsuite/gas/arm/attr-mfpu-vfpv3-d16.d (Interrupted system call) + Checking out files: 100% (12315/12315), done. + Already on 'master' + $ cd /media/hd1s1/tmp/master + $ git status + # On branch master + # Changes not staged for commit: + # (use "git add ..." to update what will be committed) + # (use "git checkout -- ..." to discard changes in working directory) + # + # modified: gas/testsuite/gas/arm/attr-mfpu-vfpv3-d16.d + # + no changes added to commit (use "git add" and/or "git commit -a") + $ git checkout -f + $ git status + # On branch master + nothing to commit (working directory clean) + +([[Git issue|git-core-2]] is known.) + + $ git-new-workdir ~/tmp/binutils/git /media/hd1s2/tmp/master master + error: unable to create file bfd/elf32-dlx.c (Interrupted system call) + error: unable to create file bfd/sunos.c (Interrupted system call) + error: unable to create file gas/testsuite/gas/arm/attr-mfpu-vfpv3-d16.d (Interrupted system call) + error: unable to create file gas/testsuite/gas/mmix/regx-op.d (Interrupted system call) + error: unable to create file gas/testsuite/gas/tic6x/reloc-bad-4.s (Interrupted system call) + error: unable to create file gold/testsuite/script_test_2.t (Interrupted system call) + error: unable to create file ld/testsuite/ld-mmix/loc7m.d (Interrupted system call) + error: unable to create file ld/testsuite/ld-powerpc/tlsexe.g (Interrupted system call) + Checking out files: 100% (12315/12315), done. + Already on 'master' + $ cd /media/hd1s2/tmp/master + $ git status + # On branch master + # Changes not staged for commit: + # (use "git add ..." to update what will be committed) + # (use "git checkout -- ..." to discard changes in working directory) + # + # modified: bfd/elf32-dlx.c + # modified: bfd/sunos.c + # modified: gas/testsuite/gas/arm/attr-mfpu-vfpv3-d16.d + # modified: gas/testsuite/gas/mmix/regx-op.d + # modified: gas/testsuite/gas/tic6x/reloc-bad-4.s + # modified: gold/testsuite/script_test_2.t + # modified: ld/testsuite/ld-mmix/loc7m.d + # modified: ld/testsuite/ld-powerpc/tlsexe.g + # + no changes added to commit (use "git add" and/or "git commit -a") + $ git checkout -f + $ git status + # On branch master + nothing to commit (working directory clean) + +Now you'd expect these directories to have identical content, but: + + $ diff -x .git -ru /media/hd1s{1,2}/tmp/master/ > /tmp/diff + $ ls -l /tmp/diff + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:12 /tmp/diff + $ grep '^[^ @+-]' < /tmp/diff + diff -x .git -ru /media/hd1s1/tmp/master//ld/configure /media/hd1s2/tmp/master//ld/configure + +(Note that this isn't a file that Git had issues with.) + +Try again: + + $ diff -x .git -ru /media/hd1s{1,2}/tmp/master/ > /tmp/diff_ + $ ls -l /tmp/diff* + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:12 /tmp/diff + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:17 /tmp/diff_ + $ cmp /tmp/diff{,_}; echo $? + 0 + +At least it's consistent. Force a reload: + + # settrans -ag /media/hd1s1 + # settrans -ag /media/hd1s2 + +Try again: + + $ diff -x .git -ru /media/hd1s{1,2}/tmp/master/ > /tmp/diff__ + $ ls -l /tmp/diff* + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:12 /tmp/diff + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:17 /tmp/diff_ + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:30 /tmp/diff__ + $ cmp /tmp/diff{,__}; echo $? + 0 + +Consistent; thus very likely corrupt on-disk. + +After a few tries, the pattern generally is that for the files where there are +differences, once the file regularely ends, its content appears once more. +That is, the files' content appears once (regularely), and then the same again. +From three retries, I've seen this only ever happen on the `-o +linux`/`filetype` option file system. + +Some more copying: + + $ (cd /media/hd1s1/tmp/ && cp -a master master_) + $ (cd /media/hd1s2/tmp/ && cp -a master master_) + $ diff -x .git -ru /media/hd1s1/tmp/master{,_}/ > /tmp/diff1 + $ diff -x .git -ru /media/hd1s2/tmp/master{,_}/ > /tmp/diff2 + $ ls -l /tmp/diff{1,2} + -rw-r--r-- 1 thomas thomas 0 10. Jun 19:46 /tmp/diff1 + -rw-r--r-- 1 thomas thomas 0 10. Jun 19:46 /tmp/diff2 + +No further difference. + + +# Conclusion? + +The ext2 `filetype` option, as used for Linux-type file systems by default, +might cause some confusion inside [[hurd/translator/ext2fs]]? + +[[!tag open_issue_documentation]] The strange thing is that in +`[hurd]/ext2fs/dir.c` the `EXT2_FEATURE_INCOMPAT_FILETYPE` is generally masked +out (is not even considered) when adding a node to a directory in +`diskfs_direnter_hard` and when reading in `diskfs_get_directs`, so unless +setting this file system flag has some other effects on the meta data (which it +doesn't seem to have according to the Linux kernel source code), it shouldn't +make any difference. Also, this information would only be used for setting +directory listing's informational `d_type` field (`readdir`, etc.). Hrm. +Perhaps only a red herring? + + +## `e2fsck` + +Running `e2fsck` on a file system with the `filetypes` option, there are +messages that the `filetype` for a lot of files is set (to either 1 (regular +file, `EXT2_FT_REG_FILE`), or 2 (directory, `EXT2_FT_DIR`)). The Hurd's ext2fs +should unconditionally set this field to 0 (`EXT2_FT_UNKNOWN`). + + +# Next + +Start over with the `filetype` option reversed: + + # mke2fs -I 128 -o linux -O ^filetype /dev/hd1s1 + # mke2fs -o hurd -O filetype /dev/hd1s2 + +In fact, for `-o hurd`, `-O filetype` doesn't have any effect (according to +`dumpe2fs -h`), so the meta data now only differs in *Filesystem OS type*. + +Invoke `git-new-workdir` once more (for each). + +Now there is corruption in files on the `-o hurd -O filetype` file system. But +this time not caused by `git-new-workdir`, but instead when copying a binutils +checkout to a new directory. Huh. Does `-O filetype` in fact have any effect +with `-o hurd`? The `e2fsprogs` source code should tell, but is not available +right now. -- cgit v1.3.1 From 40fce0b2c547fe7a73267f5285fb1f596f8ff947 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Mon, 13 Jun 2011 16:44:12 +0200 Subject: open_issues/git_duplicated_content: Progress? --- hurd/translator/ext2fs.mdwn | 16 ++- hurd/translator/ext2fs/filetype.mdwn | 33 ++++++ open_issues/ext2fs_file_corruption.mdwn | 199 -------------------------------- open_issues/git-core-2.mdwn | 7 ++ open_issues/git_duplicated_content.mdwn | 125 ++++++++++++++++++++ 5 files changed, 177 insertions(+), 203 deletions(-) create mode 100644 hurd/translator/ext2fs/filetype.mdwn delete mode 100644 open_issues/ext2fs_file_corruption.mdwn create mode 100644 open_issues/git_duplicated_content.mdwn diff --git a/hurd/translator/ext2fs.mdwn b/hurd/translator/ext2fs.mdwn index 305576b8..fff2e74b 100644 --- a/hurd/translator/ext2fs.mdwn +++ b/hurd/translator/ext2fs.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2007, 2008, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2007, 2008, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -9,14 +9,20 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -# Large Stores +# Implementation + + * [[filetype]] option + + +## Large Stores The `ext2fs` translator from the upstream Hurd code base can only handle file systems with sizes of less than roughly 2 GiB. [[!tag open_issue_hurd]] -## Ognyan's Work + +### Ognyan's Work * Ognyan Kulev, [[*Supporting Large ext2 File Systems in the Hurd*|ogi-fosdem2005.mgp]], 2005, at FOSDEM @@ -34,4 +40,6 @@ small backend stores, like floppy devices. # Documentation - + * + + * diff --git a/hurd/translator/ext2fs/filetype.mdwn b/hurd/translator/ext2fs/filetype.mdwn new file mode 100644 index 00000000..5d85bac9 --- /dev/null +++ b/hurd/translator/ext2fs/filetype.mdwn @@ -0,0 +1,33 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +The *ext2fs* translator doesn't support the ext2 format's *filetype* option. + +According to *mke2fs(8)*: + +> **filetype**: Store file type information in directory entries. + +By setting directory listings' informational `d_type` field (`readdir`, etc.), +this may avoid the need for subsequent `stat` calls. + +Not all file systems can support this option. + +In `[hurd]/ext2fs/dir.c` the `EXT2_FEATURE_INCOMPAT_FILETYPE` is generally +masked out (is not even considered) when adding a node to a directory in +`diskfs_direnter_hard` and when reading in `diskfs_get_directs`. The Hurd's +ext2fs unconditionally sets this field to 0 (`EXT2_FT_UNKNOWN`). + + +# `e2fsck` + +Running `e2fsck` on a file system with the *filetype* option, will correct the +*filetype* for a lot of files (all `EXT2_FT_UNKNOWN`?) to either 1 (regular +file, `EXT2_FT_REG_FILE`), or 2 (directory, `EXT2_FT_DIR`), and likely others. +The Hurd's ext2fs will again ignore these fields, of course. diff --git a/open_issues/ext2fs_file_corruption.mdwn b/open_issues/ext2fs_file_corruption.mdwn deleted file mode 100644 index 2af35bb5..00000000 --- a/open_issues/ext2fs_file_corruption.mdwn +++ /dev/null @@ -1,199 +0,0 @@ -[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -[[!tag open_issue_hurd]] - -[[!toc]] - - -# Log - -Create two 5 GiB partitions. - - # # Would default to 256 byte inodes otherwise. - # mke2fs -I 128 -o linux /dev/hd1s1 - [...] - # mke2fs -o hurd /dev/hd1s2 - [...] - -The only difference (apart from UUID, time stamps, hash seeds) in ext2 file -system meta data, according to `dumpe2fs -h`: - - [...] - -Filesystem features: ext_attr resize_inode dir_index filetype sparse_super large_file - +Filesystem features: ext_attr resize_inode dir_index sparse_super large_file - Filesystem flags: signed_directory_hash - Default mount options: (none) - Filesystem state: clean - Errors behavior: Continue - -Filesystem OS type: Linux - +Filesystem OS type: Hurd - [...] - -According to *mke2fs(8)*: - -> *filetype* -> Store file type information in directory entries. - - # settrans -c /media/hd1s1 /hurd/ext2fs /dev/hd1s1 - # settrans -c /media/hd1s2 /hurd/ext2fs /dev/hd1s2 - # mkdir /media/hd1s{1,2}/tmp - # chmod 1777 /media/hd1s{1,2}/tmp - - $ git-new-workdir ~/tmp/binutils/git /media/hd1s1/tmp/master master - error: unable to create file gas/testsuite/gas/arm/attr-mfpu-vfpv3-d16.d (Interrupted system call) - Checking out files: 100% (12315/12315), done. - Already on 'master' - $ cd /media/hd1s1/tmp/master - $ git status - # On branch master - # Changes not staged for commit: - # (use "git add ..." to update what will be committed) - # (use "git checkout -- ..." to discard changes in working directory) - # - # modified: gas/testsuite/gas/arm/attr-mfpu-vfpv3-d16.d - # - no changes added to commit (use "git add" and/or "git commit -a") - $ git checkout -f - $ git status - # On branch master - nothing to commit (working directory clean) - -([[Git issue|git-core-2]] is known.) - - $ git-new-workdir ~/tmp/binutils/git /media/hd1s2/tmp/master master - error: unable to create file bfd/elf32-dlx.c (Interrupted system call) - error: unable to create file bfd/sunos.c (Interrupted system call) - error: unable to create file gas/testsuite/gas/arm/attr-mfpu-vfpv3-d16.d (Interrupted system call) - error: unable to create file gas/testsuite/gas/mmix/regx-op.d (Interrupted system call) - error: unable to create file gas/testsuite/gas/tic6x/reloc-bad-4.s (Interrupted system call) - error: unable to create file gold/testsuite/script_test_2.t (Interrupted system call) - error: unable to create file ld/testsuite/ld-mmix/loc7m.d (Interrupted system call) - error: unable to create file ld/testsuite/ld-powerpc/tlsexe.g (Interrupted system call) - Checking out files: 100% (12315/12315), done. - Already on 'master' - $ cd /media/hd1s2/tmp/master - $ git status - # On branch master - # Changes not staged for commit: - # (use "git add ..." to update what will be committed) - # (use "git checkout -- ..." to discard changes in working directory) - # - # modified: bfd/elf32-dlx.c - # modified: bfd/sunos.c - # modified: gas/testsuite/gas/arm/attr-mfpu-vfpv3-d16.d - # modified: gas/testsuite/gas/mmix/regx-op.d - # modified: gas/testsuite/gas/tic6x/reloc-bad-4.s - # modified: gold/testsuite/script_test_2.t - # modified: ld/testsuite/ld-mmix/loc7m.d - # modified: ld/testsuite/ld-powerpc/tlsexe.g - # - no changes added to commit (use "git add" and/or "git commit -a") - $ git checkout -f - $ git status - # On branch master - nothing to commit (working directory clean) - -Now you'd expect these directories to have identical content, but: - - $ diff -x .git -ru /media/hd1s{1,2}/tmp/master/ > /tmp/diff - $ ls -l /tmp/diff - -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:12 /tmp/diff - $ grep '^[^ @+-]' < /tmp/diff - diff -x .git -ru /media/hd1s1/tmp/master//ld/configure /media/hd1s2/tmp/master//ld/configure - -(Note that this isn't a file that Git had issues with.) - -Try again: - - $ diff -x .git -ru /media/hd1s{1,2}/tmp/master/ > /tmp/diff_ - $ ls -l /tmp/diff* - -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:12 /tmp/diff - -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:17 /tmp/diff_ - $ cmp /tmp/diff{,_}; echo $? - 0 - -At least it's consistent. Force a reload: - - # settrans -ag /media/hd1s1 - # settrans -ag /media/hd1s2 - -Try again: - - $ diff -x .git -ru /media/hd1s{1,2}/tmp/master/ > /tmp/diff__ - $ ls -l /tmp/diff* - -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:12 /tmp/diff - -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:17 /tmp/diff_ - -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:30 /tmp/diff__ - $ cmp /tmp/diff{,__}; echo $? - 0 - -Consistent; thus very likely corrupt on-disk. - -After a few tries, the pattern generally is that for the files where there are -differences, once the file regularely ends, its content appears once more. -That is, the files' content appears once (regularely), and then the same again. -From three retries, I've seen this only ever happen on the `-o -linux`/`filetype` option file system. - -Some more copying: - - $ (cd /media/hd1s1/tmp/ && cp -a master master_) - $ (cd /media/hd1s2/tmp/ && cp -a master master_) - $ diff -x .git -ru /media/hd1s1/tmp/master{,_}/ > /tmp/diff1 - $ diff -x .git -ru /media/hd1s2/tmp/master{,_}/ > /tmp/diff2 - $ ls -l /tmp/diff{1,2} - -rw-r--r-- 1 thomas thomas 0 10. Jun 19:46 /tmp/diff1 - -rw-r--r-- 1 thomas thomas 0 10. Jun 19:46 /tmp/diff2 - -No further difference. - - -# Conclusion? - -The ext2 `filetype` option, as used for Linux-type file systems by default, -might cause some confusion inside [[hurd/translator/ext2fs]]? - -[[!tag open_issue_documentation]] The strange thing is that in -`[hurd]/ext2fs/dir.c` the `EXT2_FEATURE_INCOMPAT_FILETYPE` is generally masked -out (is not even considered) when adding a node to a directory in -`diskfs_direnter_hard` and when reading in `diskfs_get_directs`, so unless -setting this file system flag has some other effects on the meta data (which it -doesn't seem to have according to the Linux kernel source code), it shouldn't -make any difference. Also, this information would only be used for setting -directory listing's informational `d_type` field (`readdir`, etc.). Hrm. -Perhaps only a red herring? - - -## `e2fsck` - -Running `e2fsck` on a file system with the `filetypes` option, there are -messages that the `filetype` for a lot of files is set (to either 1 (regular -file, `EXT2_FT_REG_FILE`), or 2 (directory, `EXT2_FT_DIR`)). The Hurd's ext2fs -should unconditionally set this field to 0 (`EXT2_FT_UNKNOWN`). - - -# Next - -Start over with the `filetype` option reversed: - - # mke2fs -I 128 -o linux -O ^filetype /dev/hd1s1 - # mke2fs -o hurd -O filetype /dev/hd1s2 - -In fact, for `-o hurd`, `-O filetype` doesn't have any effect (according to -`dumpe2fs -h`), so the meta data now only differs in *Filesystem OS type*. - -Invoke `git-new-workdir` once more (for each). - -Now there is corruption in files on the `-o hurd -O filetype` file system. But -this time not caused by `git-new-workdir`, but instead when copying a binutils -checkout to a new directory. Huh. Does `-O filetype` in fact have any effect -with `-o hurd`? The `e2fsprogs` source code should tell, but is not available -right now. diff --git a/open_issues/git-core-2.mdwn b/open_issues/git-core-2.mdwn index a1fdbac1..acd82442 100644 --- a/open_issues/git-core-2.mdwn +++ b/open_issues/git-core-2.mdwn @@ -133,3 +133,10 @@ starting from an empty working directory (after an external `git push`): $ git status # On branch master nothing to commit (working directory clean) + +--- + +2011-06-13 + +This seems to be fixed after a rebuild of the Debian git 1.7.5.3-1 with a +recent toolchain. diff --git a/open_issues/git_duplicated_content.mdwn b/open_issues/git_duplicated_content.mdwn new file mode 100644 index 00000000..3b4bc44b --- /dev/null +++ b/open_issues/git_duplicated_content.mdwn @@ -0,0 +1,125 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_porting]] + + $ git-new-workdir ~/tmp/binutils/git /media/hd1s1/tmp/master master + error: unable to create file gas/testsuite/gas/arm/attr-mfpu-vfpv3-d16.d (Interrupted system call) + Checking out files: 100% (12315/12315), done. + Already on 'master' + $ cd /media/hd1s1/tmp/master + $ git status + # On branch master + # Changes not staged for commit: + # (use "git add ..." to update what will be committed) + # (use "git checkout -- ..." to discard changes in working directory) + # + # modified: gas/testsuite/gas/arm/attr-mfpu-vfpv3-d16.d + # + no changes added to commit (use "git add" and/or "git commit -a") + $ git checkout -f + $ git status + # On branch master + nothing to commit (working directory clean) + +([[Git issue|git-core-2]] is known.) + + $ git-new-workdir ~/tmp/binutils/git /media/hd1s2/tmp/master master + error: unable to create file bfd/elf32-dlx.c (Interrupted system call) + error: unable to create file bfd/sunos.c (Interrupted system call) + error: unable to create file gas/testsuite/gas/arm/attr-mfpu-vfpv3-d16.d (Interrupted system call) + error: unable to create file gas/testsuite/gas/mmix/regx-op.d (Interrupted system call) + error: unable to create file gas/testsuite/gas/tic6x/reloc-bad-4.s (Interrupted system call) + error: unable to create file gold/testsuite/script_test_2.t (Interrupted system call) + error: unable to create file ld/testsuite/ld-mmix/loc7m.d (Interrupted system call) + error: unable to create file ld/testsuite/ld-powerpc/tlsexe.g (Interrupted system call) + Checking out files: 100% (12315/12315), done. + Already on 'master' + $ cd /media/hd1s2/tmp/master + $ git status + # On branch master + # Changes not staged for commit: + # (use "git add ..." to update what will be committed) + # (use "git checkout -- ..." to discard changes in working directory) + # + # modified: bfd/elf32-dlx.c + # modified: bfd/sunos.c + # modified: gas/testsuite/gas/arm/attr-mfpu-vfpv3-d16.d + # modified: gas/testsuite/gas/mmix/regx-op.d + # modified: gas/testsuite/gas/tic6x/reloc-bad-4.s + # modified: gold/testsuite/script_test_2.t + # modified: ld/testsuite/ld-mmix/loc7m.d + # modified: ld/testsuite/ld-powerpc/tlsexe.g + # + no changes added to commit (use "git add" and/or "git commit -a") + $ git checkout -f + $ git status + # On branch master + nothing to commit (working directory clean) + +Now you'd expect these directories to have identical content, but: + + $ diff -x .git -ru /media/hd1s{1,2}/tmp/master/ > /tmp/diff + $ ls -l /tmp/diff + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:12 /tmp/diff + $ grep '^[^ @+-]' < /tmp/diff + diff -x .git -ru /media/hd1s1/tmp/master//ld/configure /media/hd1s2/tmp/master//ld/configure + +(Note that this isn't a file that Git had issues with.) + +Try again: + + $ diff -x .git -ru /media/hd1s{1,2}/tmp/master/ > /tmp/diff_ + $ ls -l /tmp/diff* + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:12 /tmp/diff + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:17 /tmp/diff_ + $ cmp /tmp/diff{,_}; echo $? + 0 + +At least it's consistent. Force a reload: + + # settrans -ag /media/hd1s1 + # settrans -ag /media/hd1s2 + +Try again: + + $ diff -x .git -ru /media/hd1s{1,2}/tmp/master/ > /tmp/diff__ + $ ls -l /tmp/diff* + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:12 /tmp/diff + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:17 /tmp/diff_ + -rw-r--r-- 1 thomas thomas 613677 10. Jun 19:30 /tmp/diff__ + $ cmp /tmp/diff{,__}; echo $? + 0 + +Consistent; thus very likely corrupt on-disk. + +After a few tries, the pattern generally is that for the files where there are +differences, once the file regularely ends, its content appears once more. +That is, the files' content appears once (regularely), and then the same again. + +Some more copying: + + $ (cd /media/hd1s1/tmp/ && cp -a master master_) + $ (cd /media/hd1s2/tmp/ && cp -a master master_) + $ diff -x .git -ru /media/hd1s1/tmp/master{,_}/ > /tmp/diff1 + $ diff -x .git -ru /media/hd1s2/tmp/master{,_}/ > /tmp/diff2 + $ ls -l /tmp/diff{1,2} + -rw-r--r-- 1 thomas thomas 0 10. Jun 19:46 /tmp/diff1 + -rw-r--r-- 1 thomas thomas 0 10. Jun 19:46 /tmp/diff2 + +No further difference. + +--- + + $ git-new-workdir git master master + $ diff -x .git -ur tar_master/ master/ > master.diff + + $ rm -rf ar_master* && (cd git/ && git archive master) | (mkdir ar_master && cd ar_master/ && tar -x) && diff -x .git -ru tar_master/ ar_master/ > ar_master.diff; ls -l ar_master.diff + $ (cd git/ && git archive master) | md5sum -- cgit v1.3.1 From 6a05a26aafa04e07c97d0c134a08ce75b3410252 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Mon, 13 Jun 2011 21:51:59 +0200 Subject: open_issues/git-core-2: SA_RESTART? --- open_issues/git-core-2.mdwn | 54 +++++++++++++++++++++++++++++++-- open_issues/git_duplicated_content.mdwn | 6 ++++ 2 files changed, 57 insertions(+), 3 deletions(-) diff --git a/open_issues/git-core-2.mdwn b/open_issues/git-core-2.mdwn index acd82442..2d8ad96b 100644 --- a/open_issues/git-core-2.mdwn +++ b/open_issues/git-core-2.mdwn @@ -13,6 +13,13 @@ License|/fdl]]."]]"""]] [[!tag open_issue_porting]] +[[!toc]] + + +# Log + +December, 2008. + On the otherwise-idle flubber: $ git clone git://sources.redhat.com/git/glibc.git @@ -134,9 +141,50 @@ starting from an empty working directory (after an external `git push`): # On branch master nothing to commit (working directory clean) ---- + +# Analysis 2011-06-13 -This seems to be fixed after a rebuild of the Debian git 1.7.5.3-1 with a -recent toolchain. +Running `git checkout -f` under GDB: + + error: git checkout-index: unable to create file gas/testsuite/gas/cris/string-1.s (File exists) + error: git checkout-index: unable to create file gas/testsuite/gas/i386/x86-64-sse-check.d (File exists) + error: git checkout-index: unable to create file gas/testsuite/gas/i386/x86-64-sse4_1.d (File exists) + error: git checkout-index: unable to create file gas/testsuite/gas/ppc/astest.d (File exists) + error: git checkout-index: unable to create file gas/testsuite/gas/tic6x/reloc-bad-4.s (File exists) + warning: unable to unlink include/cgen: Operation not permitted + fatal: cannot create directory at 'include/cgen': Operation not permitted + +Again: + + error: git checkout-index: unable to create file gas/config/te-vxworks.h (File exists) + error: git checkout-index: unable to create file gas/testsuite/gas/cris/string-1.s (File exists) + error: git checkout-index: unable to create file gas/testsuite/gas/d10v/warning-019.s (File exists) + error: git checkout-index: unable to create file gas/testsuite/gas/i860/dual03.s (File exists) + error: git checkout-index: unable to create file ld/testsuite/ld-mmix/sec-7a.s (File exists) + warning: unable to unlink ld/testsuite/ld-powerpc: Operation not permitted + fatal: cannot create directory at 'ld/testsuite/ld-powerpc': Operation not permitted + +And: [[git_duplicated_content]]. + +All these (very likely) have the same root cause: `SA_RESTART` restarting too +much. + +With `git checkout`, Git uses in progress.c a SIGALRM handler (`SA_RESTART`; +invoked every second via `setitimer(ITIMER_REAL)`) to display status messages: +*x % already checked out*. + +To avoid the status update signals every second, in +`[git]/progress.c:start_progress_delay` we can just return `NULL` (manually in +GDB, for example), then both the *error: git checkout-index* and the +[[duplicated content|git_duplicated_content]] issues go away. + +I'm guessing that when returning from a `SA_RESTART` signal handler, too much +of the \`\`syscall''s is being restarted. For example, if a file has already +been created, the restarted creation attempt would fail: *File exists*. If +data has been written, it might get written again (duplication issue). Then, +there are cases where `unlink` apparently returns EINTR, which is not kosher +either. Etc. + +Do we have problems with `SA_RESTART` vs. the atomicity of our syscall-alikes? diff --git a/open_issues/git_duplicated_content.mdwn b/open_issues/git_duplicated_content.mdwn index 3b4bc44b..cbc171a7 100644 --- a/open_issues/git_duplicated_content.mdwn +++ b/open_issues/git_duplicated_content.mdwn @@ -123,3 +123,9 @@ No further difference. $ rm -rf ar_master* && (cd git/ && git archive master) | (mkdir ar_master && cd ar_master/ && tar -x) && diff -x .git -ru tar_master/ ar_master/ > ar_master.diff; ls -l ar_master.diff $ (cd git/ && git archive master) | md5sum + +--- + +2011-06-13 + +-> [[git-core-2]] -- cgit v1.3.1 From e24c06d392601c1c3a5ead08aea237a4bfa79d03 Mon Sep 17 00:00:00 2001 From: Jeremie Koenig Date: Wed, 15 Jun 2011 16:27:00 +0200 Subject: user/jkoenig: java status report --- user/jkoenig/java.mdwn | 738 ++++++---------------------------------- user/jkoenig/java/proposal.mdwn | 628 ++++++++++++++++++++++++++++++++++ 2 files changed, 740 insertions(+), 626 deletions(-) create mode 100644 user/jkoenig/java/proposal.mdwn diff --git a/user/jkoenig/java.mdwn b/user/jkoenig/java.mdwn index 4052f455..90f51028 100644 --- a/user/jkoenig/java.mdwn +++ b/user/jkoenig/java.mdwn @@ -1,628 +1,114 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +# Improve Java on Hurd (GSoC 2011) + + +## Description + +The project consists in improving Java support on Hurd. +This includes porting OpenJDK, +creating low-level Java bindings for Mach and Hurd, +as well as creating Java libraries to help with translator development. + +For details, see my original [[proposal]]. + + +## Current status + + +### Apt repository + +Modified Debian packages are available in this repository: + + deb http://jk.fr.eu.org/debian experimental/ + deb-src http://jk.fr.eu.org/debian experimental/ + + +### Glibc signal code improvements + +I have submitted +[preliminary patches](http://lists.gnu.org/archive/html/bug-hurd/2011-05/msg00182.html) +for global signal dispositions, +which I'm currently testing. +I have since fixed a few thinks and implemented `SA_SIGINFO` +(which is required by OpenJDK.) +My latest code is available on +[github](http://github.com/jeremie-koenig/glibc/commits/master-beware-rebase), +and modified Debian packages +are available in my apt repository. + +One question is how the new symbols introduced by my patches +should be handled. +Weak symbols turned out to be impractical, +so I'm currently considering using a Debian-specific +symbol version in the interim period (`GLIBC_2.13_DEBIAN_7` so far). +The ultimate symbol version to be used will depend on +the time at which the patches get integrated upstream, +at which point we will alias the interim version +to the new one in debian packages. + +I have modified libc0.3 to include a `deb-symbols(5)` file +so that we get an accurate libc dependency in `hurd` and other packages +when the symbols in question are pulled in. + +Another issue which came up with OpenJDK is the expansion +by the dynamic linker of `$ORIGIN` in the `RPATH` header, +see below. + +#### Plans + +I will submit revised series for review later this week, +as well as matching Debian patches. +I expect only the last patch (implement global dispositions) will change, +and new ones will be added on top of it. + + +### Port OpenJDK + +As suggested by [[tschwinge]], I have targeted OpenJDK 7 at first. +I don't expect it will be too hard to backport my patches to OpenJDK 6. +I have succeeded in building a working JIT-less ("zero") version, +although the dynamic linker issue must be worked around. +Porting Hotspot (the original just-in-time compiler of OpenJDK) +should not be too hard. +If that fails we can fall back on Shark +(a portable alternative JIT which uses LLVM). + +The dynamic linker issue is as follows. +An executable-specific search path can be provided in the ELF RPATH header. +RPATH directories can include the special string `$ORIGIN`, +which is to be expanded to the directory the executable was loaded from. +OpenJDK's `java` command uses this feature to locate +the right `libjli.so` at runtime. +However, +on Hurd this information is not available to the dynamic linker +and as a consequence RPATH components which include `$ORIGIN` +are silently discarded. + +This can be worked around by defining +the `LD_ORIGIN_PATH` environment variable. +(which have I used to build and test OpenJDK so far.) + +#### Plans + +I intend to fix the RPATH issue +by building on [[pochu]]'s `file_exec_file_name()` patches. + +I have succeeded in building a Hotspot-enabled `libjvm.so`, +although the current toolchain issues +have so far prevented me from testing it. + + +### Java bindings for Mach + +(just started.) -# Java for Hurd (and vice versa) - -Contact information: - - * Full name: Jérémie Koenig - * Email: jk@jk.fr.eu.org - * IRC: jkoenig on Freenode and OFTC - -## Introductions - -I am a first year M.Sc. student -in Computer Science at University of Strasbourg (France). -My interests include capability-based security, -programming languages and formal methods -(in particular, object-capability languages and proof-carrying code). - -### Proposal summary - -This project would consist in improving Java support on Hurd. -The first part would consist in -fixing bugs and porting Java-related packages. -The second part would consist in -creating low-level Java bindings for the Hurd interfaces, -as well as libraries to make translator development easier. - -### Previous involvement - -I started contributing to Hurd last summer, -during which I participated to Google Summer of Code -as a student for the Debian project. -I worked on porting Debian-Installer to Hurd. -This project was mostly a success, -although we still have to use a special mirror for installation -with a few modified packages -and tweaked priorities -to work around some uninstallable packages -with Priority: standard. - -Shortly afterwards, -I rewrote the procfs translator -to fix some issues with memory leaks, -make it more reliable, -and improve compatibility with Linux-based tools -such as `procps` or `htop`. - -Although I have not had as much time -as I would have liked to dedicate to the Hurd -since that time, -I have continued to maintain the mirror in question, -and I have started to work -on implementing POSIX threads signal semantics in glibc. - -### Project-related skills and interests - -I have used Java mostly for university assignments. -This includes non-trivial projects -using threads and distributed programming frameworks -such as Java RMI or CORBA. -I have also used it to experiment with -Google App Engine -(web applications) -and Google Web Toolkit -(a compiler from Java to Javascript which helps with AJAX code), -and I have some limited experience with JNI -(the Java Native Interface, to link Java with C code). - -My knowledge of the Hurd and Debian GNU/Hurd is reasonable, -as the Debian-Installer and procfs projects -gave me the opportunity to fiddle with many parts of the system. - -Initially, -I started working on this project because I wanted to use -[Joe-E](http://code.google.com/p/joe-e/) -(a subset of Java) -to investigate the potential -[[applications of object-capability languages|objcap]] -in a Hurd context. -I also believe that improving Java support on Hurd -would be an important milestone. - -### Organisational matters - -I am subscribed to bug-hurd@g.o and -I do have a permanent internet connexion. - -I would be able to attend the regular IRC meetings, -and otherwise communicate with my mentor -through any means they would prefer -(though I expect email and IRC would be the most practical). -Since I'm already familiar with the Hurd, -I don't expect I would require too much time from them. - -My exams end on May 20 so I would be able to start coding -right at the beginning of the GSoC period. -Next year's term would probably begin around September 15, -so that would not be an issue either. -I expect I would work around 40 hours per week, -and my waking hours would be flexible. - -I don't have any other plans for the summer -and would not make any if my project were to be accepted. - -Full disclosure: -I also submitted a proposal to the Jikes RVM project -(which is a research-oriented Java Virtual Machine, -itself written in Java) -for implementing a new garbage collector into the MMTk subsystem. - -## Improve Java support - -### Justification - -Java is a popular language and platform used by many desktop and web -applications (mostly on the server side). As a consequence, competitive Java -support is important for any general-purpose operating system. -Better Java support would also be a prerequisite -for the second part of my proposal. - -### Current situation - -Java is currently supported on Hurd with the GNU Java suite: - - * [GCJ](http://gcc.gnu.org/java/), - the GNU Compiler for Java, is part of GCC and can compile Java - source code to Java bytecode, and both source code and bytecode to - native code; - * libgcj is the implementation of the Java runtime which GCJ uses. - It is based on [GNU Classpath](http://www.gnu.org/software/classpath/). - It includes a bytecode interpreter which enables - Java applications compiled to native code to dynamically load and execute - Java bytecode from class files. - * The gij command is a wrapper around the above-mentioned virtual machine - functionality of libgcj and can be used as a replacement for the java - command. - -However, GCJ does not work flawlessly on Hurd.r -For instance, some parts of libgcj relies on -the POSIX threads signal semantics, which are not yet implemented. -In particular, this makes ant hang waiting for child processes, -which makes some packages fail to build on Hurd -(“ant” is the “make” of the Java world). - -### Tasks - - * **Finish implementing POSIX thread semantics** in glibc (high priority). - According to POSIX, signal dispositions should be global to a process, - while signal blocking masks should be thread-specific. Signals sent to the - process as a whole are to be delivered to any thread which does not block - them. By contrast, Hurd has per-thread signal dispositions and signals - sent to a process are delivered to the main thread only. I have been - working on refactoring the glibc signal code and implementing the POSIX - semantics as a per-thread option. However, due to lack of time I have not - yet been able to test and debug my code properly. Finishing this work - would be my first task. - * **Fix further problems with GCJ on Hurd** (high priority). While I’m not - aware of any other problems with GCJ at the moment, I suspect some might - turn up as I progress with the other tasks. Fixing these problems would - also be a high-priority task. - * **Port OpenJDK 6** (medium priority). While GCJ is fine, it is not yet - 100% complete. It is also slower than OpenJDK on architectures where a - just-in-time compiler is available. Porting OpenJDK would therefore - improve Java support on Hurd in scope and quality. Besides, it would also - be a good way to test GCJ, which is used for bootstrapping by the Debian - OpenJDK packages. Also note that OpenJDK 6 is now the default Java - Runtime Environment on all released Linux-based Debian architectures; - bringing Hurd in line with this would probably be a good thing. - * **Port Eclipse and other Java applications** (low priority). Eclipse is a - popular, state-of-the-art IDE and tool suite used for Java and other - languages. It is a dependency of the Joe-E verifier (see part 3 of this - proposal). Porting Eclipse would be a good opportunity to test GCJ and - OpenJDK. - -### Deliverables - - * The glibc pthreads patch and any other fixes on the Hurd side - would be submitted upstream - * Patches against Debian source packages - required to make them build on Hurd would be submitted - to the [Debian bug tracking system](http://bugs.debian.org/). - - -## Create Java bindings for the Hurd interfaces - -### Justification - -Java is used for many applications and often taught to -introduce object-oriented programming. The fact that Java is a -garbage-collected language makes it easier to use, especially for the less -experienced programmers. Besides, its object-oriented nature is a -natural fit for the capability-based design of Hurd. -The JVM is also used as a target for many other languages, -all of which would benefit from the access provided by these bindings. - -Advantages over other garbage-collected, object-oriented languages include -performance, type safety and the possibility to compile a Java translator to -native code and -[link it statically](http://gcc.gnu.org/wiki/Statically_linking_libgcj) -using GCJ, should anyone want to use a -translator written in Java for booting. -Note that Java is -[being](http://www.linuxjournal.com/article/8757) -[used](http://oss.readytalk.com/avian/) -in this manner for embedded development. -Since GCJ can take bytecode as its input, -this expect this possibility would apply to any JVM-based language. - -Java bindings would lower the bar for newcomers -to begin experimenting with what makes Hurd unique -without being faced right away with the complexity of -low-level systems programming. - -### Tasks summary - - * Implement Java bindings for Mach - * Implement a libports-like library for Java - * Modify MIG to output Java code - * Implement libfoofs-like Java libraries - -### Design principles - -The principles I would use to guide the design -of these Java bindings would be the following ones: - - * The system should be hooked into at a low level, - to ensure that Java is a "first class citizen" - as far as the access to the Hurd's interfaces is concerned. - * At the same time, the memory safety of Java should be maintained - and extended to Mach primitives such as port names and - out-of-line memory regions. - * Higher-level interfaces should be provided as well - in order to make translator development - as easy as possible. - * A minimum amount of JNI code (ie. C code) should be used. - Most of the system should be built using Java itself - on top of a few low-level primitives. - * Hurd objects would map to Java objects. - * Using the same interfaces, - objects corresponding to local ports would be accessed directly, - and remote objects would be accessed over IPC. - -One approach used previously to interface programming languages with the Hurd -has been to create bindings for helper libraries such as libtrivfs. Instead, -for Java I would like to take a lower-level approach by providing access to -Mach primitives and extending MIG to generate Java code from the interface -description files. - -This approach would be initially more involved, and would introduces several -issues related to overcoming the "impedance mismatch" between Java and Mach. -However, once an initial implementation is done it would be easier to maintain -in the long run and we would be able to provide Java bindings for a large -percentage of the Hurd’s interfaces. - -### Bindings for Mach system calls - -In this low-level approach, my intention is to enable Java code to use Mach -system calls (in particular, mach_msg) more or less directly. This would -ensure full access to the system from Java code, but it raises a number of -issues: - - * the Java code must be able to manipulate Mach-level entities, such as port - rights or page-aligned buffers mapped outside of the garbage-collected - heap (for out-of-line transfers); - * putting together IPC messages requires control of the low-level - representation of data. - -In order to address these concerns, classes would be encapsulating these -low-level entities so that they can be referenced through normal, safe objects -from standard Java code. Bindings for Mach system calls can then be provided -in terms of these classes. Their implementation would use C code through the -Java Native Interface (JNI). - -More specifically, this functionality would be provided by the `org.gnu.mach` -package, which would contain at least the following classes: - - * `MachPort` would encapsulate a `mach_port_t`. (Some of) its constructors - would act as an interface for the `mach_port_allocate()` system call. - `MachPort` objects would also be instantiated from other parts of the JNI - C code to represent port rights received through IPC. The `deallocate()` - method would call `mach_port_deallocate()` and replace the encapsulated - port name with `MACH_PORT_DEAD`. We would recommend that users call it - when a port is no longer used, but the finalizer would also deallocate the - port when the `MachPort` object is garbage collected. - * `Buffer` would represent a page-aligned buffer allocated outside of the - Java heap, to be transferred (or having been received) as out-of-line - memory. The JNI code would would provide methods to read and write data at - an arbitrary offset (but within bounds) and would use `vm_allocate()` and - `vm_deallocate()` in the same spirit as for `MachPort` objects. - * `Message` would allow Java code to put together Mach messages. The - constructor would allocate a `byte[]` member array of a given size. - Additional methods would be provided to fill in or query the information - in the message header and additional data items, including `MachPort` and - `Buffer` objects which would be translated to the corresponding port names - and out-of-line pointers. - A global map from port names to the corresponding `MachPort` object - would probably be needed to ensure that there is a one-to-one - correspondence. - * `Syscall` would provide static JNI methods for performing system calls not - covered by the above classes, such as `mach_msg()` or - `mach_thread_self()`. These methods would accept or return `MachPort`, - `Buffer` and `Message` objects when appropriate. The associated C code - would access the contents of such objects directly in order to perform the - required unsafe operations, such as constructing `MachPort` and `Buffer` - objects directly from port names and C pointers. - -Note that careful consideration should be given to the interfaces of these -classes to avoid “safety leaks” which would compromise the safety guarantees -provided by Java. Potential problematic scenarios include the following -examples: - - * It must not be possible to write an integer at some position in a - `Message` object, and to read it back as a `MachPort` or `Buffer` object, - since this would allow unsafe access to arbitrary memory addresses and - mach port names. - * Providing the `mach_task_self()` system call would also provide access to - arbitrary addresses and ports by using the `vm_*` family of RPC operations - with the returned `MachPort` object. This means that the relevant task - operations should be provided by the `Syscall` class instead. - -Finally, access should be provided to the initial ports and file descriptors -in `_hurd_ports` and provided by the `getdport()` function, -for instance through static methods such as -`getCRDir()`, `getCWDir()`, `getProc()`, ... in a dedicated class such as -`org.gnu.hurd.InitPorts`. - -A realistic example of code based on such interfaces would be: - - import org.gnu.mach.MsgType; - import org.gnu.mach.MachPort; - import org.gnu.mach.Buffer; - import org.gnu.mach.Message; - import org.gnu.mach.Syscall; - import org.gnu.hurd.InitPorts; - - public class Hello - { - public static main(String argv[]) - /* Parent class for all Mach-related exceptions */ - throws org.gnu.mach.MachException - { - /* Allocate a reply port */ - MachPort reply = new MachPort(); - - /* Allocate an out-of-line buffer */ - Buffer data = new Buffer(MsgType.CHAR, 13); - data.writeString(0, "Hello, World!"); - - /* Craft an io_write message */ - Message msg = new Message(1024); - msg.setRemotePort(InitPorts.getdport(1)); - msg.setLocalPort(reply, Message.Type.MAKE_SEND_ONCE); - msg.setId(21000); - msg.addBuffer(data); - - /* Make the call, MACH_MSG_SEND | MACH_MSG_RECEIVE */ - Syscall.machMsg(msg, true, true, reply); - - /* Extract the returned value */ - msg.assertId(21100); - int retCode = msg.readInt(0); - int amount = msg.readInt(1); - } - } - -Should this paradigm prove insufficient, -more ideas could be borrowed from the -[`org.vmmagic`](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.151.5253&rep=rep1&type=pdf) -package used by [Jikes RVM](http://jikesrvm.org/), -a research Java virtual machine itself written in Java. - -### Generating Java stubs with MIG - -Once the basic machinery is in place to interface with Mach, Java programs -have more or less equal access to the system functionality without resorting -to more JNI code. However, as illustrated above, this access is far from -convenient. - -As a solution I would modify MIG to add the option to output Java code. MIG -would emit a Java interface, a client class able to implement the interface -given a Mach port send right, an a server class which would be able to handle -incoming messages. The class diagram below, although it is by no means -complete or exempt of any problem, illustrates the general idea: - -[[gsoc2011_classes.png]] - -This structure is somewhat reminiscent of -[Java RMI](http://en.wikipedia.org/wiki/Java_remote_method_invocation) -or similar systems, -which aim to provide more or less transparent access to remote objects. -The exact way the Java code would be generated still needs to be determined, -but basically: - - * An interface, corresponding to the header files generated by MIG, would - enumerate the operations listed in a given .defs files. Method names would - be transformed to adhere to Java conventions (for instance, - `some_random_identifier` would become `someRandomIdentifier`). - * A user class, corresponding to the `*User.c` files, - would implement this interface by doing RPC over a given MachPort object. - * A server class, corresponding to `*Server.c`, would be able to handle - incoming messages using a user-provided implementation of the interface. - (Possibly, a skeleton class providing methods which would raise - `NotImplementedException`s would be provided as well. - Users would derive from this class and override the relevant methods. - This would allow them not to implement some operations, - and would avoid pre-existing code from breaking when new operations are - introduced.) - -In order to help with the implementation of servers, some kind of library -would be needed to associate Mach receive rights with server objects and to -handle incoming messages on dedicated threads, in the spirit of libports. -This would probably require support for port sets at the level of the Mach -primitives described in the previous section. - -When possible, operations involving the transmission of send rights -of some kind would be expressed in terms of the MIG-generated interfaces -instead of `MachPort` objects. -Upon reception of a send right, -a `FooUser` object would be created -and associated with the corresponding `MachPort` object. -If the received send right corresponds to a local port -to which a server object has been associated, -this object would be used instead. -This way, -subsequent operations on the received send right -would be handled as direct method calls -instead of going through RPC mechanisms. - -Some issues will still need to be solved regarding how MIG will convert -interface description files to Java interfaces. For instance: - - * `.defs` files are not explicitly associated with a type. For instance in - the example above, MIG would have to somehow infer that io_t corresponds - to `this` in the `Io` interface. - * More generally, a correspondence between MIG and Java types would have - to be determined. Ideally this would be automated and not hardcoded - too much. - * Initially, reply port parameters would be ignored. However they may be - needed for some applications. - -So the details would need to be flushed out during the community bonding -period and as the implementation progresses. However I’m confident that a -satisfactory solution can be designed. - -Using these new features, the example above could be rewritten as: - - import org.gnu.hurd.InitPorts; - import org.gnu.hurd.Io; - import org.gnu.hurd.IoUser; - - class Hello { - static void main(String argv[]) throws ... - { - Io stdout = new IoUser(InitPorts.getdport(1)); - String hello = “Hello, World!\n”; - - int amount = stdout.write(hello.getBytes(), -1); - - /* (A retCode corresponding to an error - would be signalled as an exception.) */ - } - } - -An example of server implementation would be: - - import org.gnu.hurd.Io; - import java.util.Arrays; - - class HelloIo implements Io { - final byte[] contents = “Hello, World!\n”.getBytes(); - - int write(byte[] data, int offset) { - return SOME_ERROR_CODE; - } - - byte[] read(int offset, int amount) { - return Arrays.copyOfRange(contents, offset, - offset + amount - 1); - } - - /* ... */ - } - -A new server object could then be created with `new IoServer(new HelloIo())`, -and associated with some receive right at the level of the ports management -library. - -### Base classes for common types of translators - -Once MIG can target Java code, and a libports equivalent is available, -creating new translators in Java would be greatly facilitated. However, -we would probably want to introduce basic implementations of file system -translators in the spirit of libtrivfs or libnetfs. They could take the form -of base classes implementing the relevant MIG-generated interfaces which -would then be derived by users, -or could define a simpler interface -which would then be used by adapter classes -to implement the required ones. - -I would draw inspiration from libtrivfs and libnetfs -to design and implement similar solutions for Java. - -### Deliverables - - * A hurd-java package would contain the Java code developed - in the context of this project. - * The Java code would be documented using javadoc - and a tutorial for writing translators would be written as well. - * Modifications to MIG would be submitted upstream, - or a patched MIG package would be made available. - -The Java libraries resulting from this work, -including any MIG support classes -as well as the class files built from the MIG-generated code -for the Mach and Hurd interface definition files, -would be provided as single `hurd-java` package for -Debian GNU/Hurd. -This package would be separate from both Hurd and Mach, -so as not to impose unreasonable build dependencies on them. - -I expect I would be able to act as its maintainer in the foreseeable future, -either as an individual or as a part of the Hurd team. -Hopefully, -my code would be claimed by the Hurd project as their own, -and consequently the modifications to MIG -(which would at least conceptually depend on the Mach Java package) -could be integrated upstream. - -Since by design, -the Java code would use only a small number of stable interfaces, -it would not be subject to excessive amounts of bitrot. -Consequently, -maintenance would primarily consist in -fixing bugs as they are reported, -and adding new features as they are requested. -A large number of such requests -would mean the package is useful, -so I expect that the overall amount of work -would be correlated with the willingness of more people -to help with maintenance -should I become overwhelmed or get hit by a bus. - - -## Timeline - -The dates listed are deadlines for the associated tasks. - - * *Community bonding period.* - Discuss, refine and complete the design of the Java bindings - (in particular the MIG and "libports" parts) - * *May 23.* - Coding starts. - * *May 30.* - Finish implementing pthread signal semantics. - * *June 5.* - Port OpenJDK - * *June 12.* - Fix the remaining problems with GCJ and/or OpenJDK, - possibly port Eclipse or other big Java packages. - * *June 19.* - Create the bindings for Mach. - * *June 26.* - Work on some kind of basic Java libports - to handle receive rights. - * *July 3.* - Test, write some documentation and examples. - * *July 17 (two weeks).* - Add the Java target to MIG. - * *July 24.* - Test, write some documentation and examples. - * *August 7 (two weeks).* - Implement a modular libfoofs to help with translator development. - Try to write a basic but non-trivial translator - to evaluate the performance and ease of use of the result, - rectify any rough edges this would uncover. - * *August 22. (last two weeks)* - Polish the code and packaging, - finish writing the documentation. - - -## Conclusion - -This project is arguably ambitious. -However, I have been thinking about it for some time now -and I'm confident I would be able to accomplish most of it. - -In the event multiple language bindings projects -would be accepted, -some work could probably be done in common. -In particular, -[ArneBab](http://www.bddebian.com/~hurd-web/community/weblogs/ArneBab/2011-04-06-application-pyhurd/) -seems to favor a low-level approach for his Python bindings as I do for Java, -and I would be happy to discuss API design and coordinate MIG changes with him. -I would also have an extra month after the end of the GSoC period -before I go back to school, -which I would be able to use to finish the project -if there is some remaining work. -(Last year's rewrite of procfs was done during this period.) - -As for the project's benefits, -I believe that good support for Java -is a must-have for the Hurd. -Java bindings would also further the Hurd's agenda -of user freedom by extending this freedom to more people: -I expect the set of developers -who would be able to write Java code against a well-written libfoofs -is much larger than -those who master the intricacies of low-level systems C programming. -From a more strategic point of view, -this would also help recruit new contributors -by providing an easier path to learning the inner workings of the Hurd. - -Further developments -which would build on the results of this project -include my planned [[experiment with Joe-E|objcap]] -(which I would possibly take on as a university project next year). -Another possibility would be to reimplement some parts -of the Java standard library -directly in terms of the Hurd interfaces -instead of using the POSIX ones through glibc. -This would possibly improve the performance -of some Java applications (though probably not by much), -and would otherwise be a good project -for someone trying to get acquainted with Hurd. - -Overall, I believe this project would be fun, interesting and useful. -I hope that you will share this sentiment -and give me the opportunity to spend another summer working on Hurd. diff --git a/user/jkoenig/java/proposal.mdwn b/user/jkoenig/java/proposal.mdwn new file mode 100644 index 00000000..4052f455 --- /dev/null +++ b/user/jkoenig/java/proposal.mdwn @@ -0,0 +1,628 @@ + +# Java for Hurd (and vice versa) + +Contact information: + + * Full name: Jérémie Koenig + * Email: jk@jk.fr.eu.org + * IRC: jkoenig on Freenode and OFTC + +## Introductions + +I am a first year M.Sc. student +in Computer Science at University of Strasbourg (France). +My interests include capability-based security, +programming languages and formal methods +(in particular, object-capability languages and proof-carrying code). + +### Proposal summary + +This project would consist in improving Java support on Hurd. +The first part would consist in +fixing bugs and porting Java-related packages. +The second part would consist in +creating low-level Java bindings for the Hurd interfaces, +as well as libraries to make translator development easier. + +### Previous involvement + +I started contributing to Hurd last summer, +during which I participated to Google Summer of Code +as a student for the Debian project. +I worked on porting Debian-Installer to Hurd. +This project was mostly a success, +although we still have to use a special mirror for installation +with a few modified packages +and tweaked priorities +to work around some uninstallable packages +with Priority: standard. + +Shortly afterwards, +I rewrote the procfs translator +to fix some issues with memory leaks, +make it more reliable, +and improve compatibility with Linux-based tools +such as `procps` or `htop`. + +Although I have not had as much time +as I would have liked to dedicate to the Hurd +since that time, +I have continued to maintain the mirror in question, +and I have started to work +on implementing POSIX threads signal semantics in glibc. + +### Project-related skills and interests + +I have used Java mostly for university assignments. +This includes non-trivial projects +using threads and distributed programming frameworks +such as Java RMI or CORBA. +I have also used it to experiment with +Google App Engine +(web applications) +and Google Web Toolkit +(a compiler from Java to Javascript which helps with AJAX code), +and I have some limited experience with JNI +(the Java Native Interface, to link Java with C code). + +My knowledge of the Hurd and Debian GNU/Hurd is reasonable, +as the Debian-Installer and procfs projects +gave me the opportunity to fiddle with many parts of the system. + +Initially, +I started working on this project because I wanted to use +[Joe-E](http://code.google.com/p/joe-e/) +(a subset of Java) +to investigate the potential +[[applications of object-capability languages|objcap]] +in a Hurd context. +I also believe that improving Java support on Hurd +would be an important milestone. + +### Organisational matters + +I am subscribed to bug-hurd@g.o and +I do have a permanent internet connexion. + +I would be able to attend the regular IRC meetings, +and otherwise communicate with my mentor +through any means they would prefer +(though I expect email and IRC would be the most practical). +Since I'm already familiar with the Hurd, +I don't expect I would require too much time from them. + +My exams end on May 20 so I would be able to start coding +right at the beginning of the GSoC period. +Next year's term would probably begin around September 15, +so that would not be an issue either. +I expect I would work around 40 hours per week, +and my waking hours would be flexible. + +I don't have any other plans for the summer +and would not make any if my project were to be accepted. + +Full disclosure: +I also submitted a proposal to the Jikes RVM project +(which is a research-oriented Java Virtual Machine, +itself written in Java) +for implementing a new garbage collector into the MMTk subsystem. + +## Improve Java support + +### Justification + +Java is a popular language and platform used by many desktop and web +applications (mostly on the server side). As a consequence, competitive Java +support is important for any general-purpose operating system. +Better Java support would also be a prerequisite +for the second part of my proposal. + +### Current situation + +Java is currently supported on Hurd with the GNU Java suite: + + * [GCJ](http://gcc.gnu.org/java/), + the GNU Compiler for Java, is part of GCC and can compile Java + source code to Java bytecode, and both source code and bytecode to + native code; + * libgcj is the implementation of the Java runtime which GCJ uses. + It is based on [GNU Classpath](http://www.gnu.org/software/classpath/). + It includes a bytecode interpreter which enables + Java applications compiled to native code to dynamically load and execute + Java bytecode from class files. + * The gij command is a wrapper around the above-mentioned virtual machine + functionality of libgcj and can be used as a replacement for the java + command. + +However, GCJ does not work flawlessly on Hurd.r +For instance, some parts of libgcj relies on +the POSIX threads signal semantics, which are not yet implemented. +In particular, this makes ant hang waiting for child processes, +which makes some packages fail to build on Hurd +(“ant” is the “make” of the Java world). + +### Tasks + + * **Finish implementing POSIX thread semantics** in glibc (high priority). + According to POSIX, signal dispositions should be global to a process, + while signal blocking masks should be thread-specific. Signals sent to the + process as a whole are to be delivered to any thread which does not block + them. By contrast, Hurd has per-thread signal dispositions and signals + sent to a process are delivered to the main thread only. I have been + working on refactoring the glibc signal code and implementing the POSIX + semantics as a per-thread option. However, due to lack of time I have not + yet been able to test and debug my code properly. Finishing this work + would be my first task. + * **Fix further problems with GCJ on Hurd** (high priority). While I’m not + aware of any other problems with GCJ at the moment, I suspect some might + turn up as I progress with the other tasks. Fixing these problems would + also be a high-priority task. + * **Port OpenJDK 6** (medium priority). While GCJ is fine, it is not yet + 100% complete. It is also slower than OpenJDK on architectures where a + just-in-time compiler is available. Porting OpenJDK would therefore + improve Java support on Hurd in scope and quality. Besides, it would also + be a good way to test GCJ, which is used for bootstrapping by the Debian + OpenJDK packages. Also note that OpenJDK 6 is now the default Java + Runtime Environment on all released Linux-based Debian architectures; + bringing Hurd in line with this would probably be a good thing. + * **Port Eclipse and other Java applications** (low priority). Eclipse is a + popular, state-of-the-art IDE and tool suite used for Java and other + languages. It is a dependency of the Joe-E verifier (see part 3 of this + proposal). Porting Eclipse would be a good opportunity to test GCJ and + OpenJDK. + +### Deliverables + + * The glibc pthreads patch and any other fixes on the Hurd side + would be submitted upstream + * Patches against Debian source packages + required to make them build on Hurd would be submitted + to the [Debian bug tracking system](http://bugs.debian.org/). + + +## Create Java bindings for the Hurd interfaces + +### Justification + +Java is used for many applications and often taught to +introduce object-oriented programming. The fact that Java is a +garbage-collected language makes it easier to use, especially for the less +experienced programmers. Besides, its object-oriented nature is a +natural fit for the capability-based design of Hurd. +The JVM is also used as a target for many other languages, +all of which would benefit from the access provided by these bindings. + +Advantages over other garbage-collected, object-oriented languages include +performance, type safety and the possibility to compile a Java translator to +native code and +[link it statically](http://gcc.gnu.org/wiki/Statically_linking_libgcj) +using GCJ, should anyone want to use a +translator written in Java for booting. +Note that Java is +[being](http://www.linuxjournal.com/article/8757) +[used](http://oss.readytalk.com/avian/) +in this manner for embedded development. +Since GCJ can take bytecode as its input, +this expect this possibility would apply to any JVM-based language. + +Java bindings would lower the bar for newcomers +to begin experimenting with what makes Hurd unique +without being faced right away with the complexity of +low-level systems programming. + +### Tasks summary + + * Implement Java bindings for Mach + * Implement a libports-like library for Java + * Modify MIG to output Java code + * Implement libfoofs-like Java libraries + +### Design principles + +The principles I would use to guide the design +of these Java bindings would be the following ones: + + * The system should be hooked into at a low level, + to ensure that Java is a "first class citizen" + as far as the access to the Hurd's interfaces is concerned. + * At the same time, the memory safety of Java should be maintained + and extended to Mach primitives such as port names and + out-of-line memory regions. + * Higher-level interfaces should be provided as well + in order to make translator development + as easy as possible. + * A minimum amount of JNI code (ie. C code) should be used. + Most of the system should be built using Java itself + on top of a few low-level primitives. + * Hurd objects would map to Java objects. + * Using the same interfaces, + objects corresponding to local ports would be accessed directly, + and remote objects would be accessed over IPC. + +One approach used previously to interface programming languages with the Hurd +has been to create bindings for helper libraries such as libtrivfs. Instead, +for Java I would like to take a lower-level approach by providing access to +Mach primitives and extending MIG to generate Java code from the interface +description files. + +This approach would be initially more involved, and would introduces several +issues related to overcoming the "impedance mismatch" between Java and Mach. +However, once an initial implementation is done it would be easier to maintain +in the long run and we would be able to provide Java bindings for a large +percentage of the Hurd’s interfaces. + +### Bindings for Mach system calls + +In this low-level approach, my intention is to enable Java code to use Mach +system calls (in particular, mach_msg) more or less directly. This would +ensure full access to the system from Java code, but it raises a number of +issues: + + * the Java code must be able to manipulate Mach-level entities, such as port + rights or page-aligned buffers mapped outside of the garbage-collected + heap (for out-of-line transfers); + * putting together IPC messages requires control of the low-level + representation of data. + +In order to address these concerns, classes would be encapsulating these +low-level entities so that they can be referenced through normal, safe objects +from standard Java code. Bindings for Mach system calls can then be provided +in terms of these classes. Their implementation would use C code through the +Java Native Interface (JNI). + +More specifically, this functionality would be provided by the `org.gnu.mach` +package, which would contain at least the following classes: + + * `MachPort` would encapsulate a `mach_port_t`. (Some of) its constructors + would act as an interface for the `mach_port_allocate()` system call. + `MachPort` objects would also be instantiated from other parts of the JNI + C code to represent port rights received through IPC. The `deallocate()` + method would call `mach_port_deallocate()` and replace the encapsulated + port name with `MACH_PORT_DEAD`. We would recommend that users call it + when a port is no longer used, but the finalizer would also deallocate the + port when the `MachPort` object is garbage collected. + * `Buffer` would represent a page-aligned buffer allocated outside of the + Java heap, to be transferred (or having been received) as out-of-line + memory. The JNI code would would provide methods to read and write data at + an arbitrary offset (but within bounds) and would use `vm_allocate()` and + `vm_deallocate()` in the same spirit as for `MachPort` objects. + * `Message` would allow Java code to put together Mach messages. The + constructor would allocate a `byte[]` member array of a given size. + Additional methods would be provided to fill in or query the information + in the message header and additional data items, including `MachPort` and + `Buffer` objects which would be translated to the corresponding port names + and out-of-line pointers. + A global map from port names to the corresponding `MachPort` object + would probably be needed to ensure that there is a one-to-one + correspondence. + * `Syscall` would provide static JNI methods for performing system calls not + covered by the above classes, such as `mach_msg()` or + `mach_thread_self()`. These methods would accept or return `MachPort`, + `Buffer` and `Message` objects when appropriate. The associated C code + would access the contents of such objects directly in order to perform the + required unsafe operations, such as constructing `MachPort` and `Buffer` + objects directly from port names and C pointers. + +Note that careful consideration should be given to the interfaces of these +classes to avoid “safety leaks” which would compromise the safety guarantees +provided by Java. Potential problematic scenarios include the following +examples: + + * It must not be possible to write an integer at some position in a + `Message` object, and to read it back as a `MachPort` or `Buffer` object, + since this would allow unsafe access to arbitrary memory addresses and + mach port names. + * Providing the `mach_task_self()` system call would also provide access to + arbitrary addresses and ports by using the `vm_*` family of RPC operations + with the returned `MachPort` object. This means that the relevant task + operations should be provided by the `Syscall` class instead. + +Finally, access should be provided to the initial ports and file descriptors +in `_hurd_ports` and provided by the `getdport()` function, +for instance through static methods such as +`getCRDir()`, `getCWDir()`, `getProc()`, ... in a dedicated class such as +`org.gnu.hurd.InitPorts`. + +A realistic example of code based on such interfaces would be: + + import org.gnu.mach.MsgType; + import org.gnu.mach.MachPort; + import org.gnu.mach.Buffer; + import org.gnu.mach.Message; + import org.gnu.mach.Syscall; + import org.gnu.hurd.InitPorts; + + public class Hello + { + public static main(String argv[]) + /* Parent class for all Mach-related exceptions */ + throws org.gnu.mach.MachException + { + /* Allocate a reply port */ + MachPort reply = new MachPort(); + + /* Allocate an out-of-line buffer */ + Buffer data = new Buffer(MsgType.CHAR, 13); + data.writeString(0, "Hello, World!"); + + /* Craft an io_write message */ + Message msg = new Message(1024); + msg.setRemotePort(InitPorts.getdport(1)); + msg.setLocalPort(reply, Message.Type.MAKE_SEND_ONCE); + msg.setId(21000); + msg.addBuffer(data); + + /* Make the call, MACH_MSG_SEND | MACH_MSG_RECEIVE */ + Syscall.machMsg(msg, true, true, reply); + + /* Extract the returned value */ + msg.assertId(21100); + int retCode = msg.readInt(0); + int amount = msg.readInt(1); + } + } + +Should this paradigm prove insufficient, +more ideas could be borrowed from the +[`org.vmmagic`](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.151.5253&rep=rep1&type=pdf) +package used by [Jikes RVM](http://jikesrvm.org/), +a research Java virtual machine itself written in Java. + +### Generating Java stubs with MIG + +Once the basic machinery is in place to interface with Mach, Java programs +have more or less equal access to the system functionality without resorting +to more JNI code. However, as illustrated above, this access is far from +convenient. + +As a solution I would modify MIG to add the option to output Java code. MIG +would emit a Java interface, a client class able to implement the interface +given a Mach port send right, an a server class which would be able to handle +incoming messages. The class diagram below, although it is by no means +complete or exempt of any problem, illustrates the general idea: + +[[gsoc2011_classes.png]] + +This structure is somewhat reminiscent of +[Java RMI](http://en.wikipedia.org/wiki/Java_remote_method_invocation) +or similar systems, +which aim to provide more or less transparent access to remote objects. +The exact way the Java code would be generated still needs to be determined, +but basically: + + * An interface, corresponding to the header files generated by MIG, would + enumerate the operations listed in a given .defs files. Method names would + be transformed to adhere to Java conventions (for instance, + `some_random_identifier` would become `someRandomIdentifier`). + * A user class, corresponding to the `*User.c` files, + would implement this interface by doing RPC over a given MachPort object. + * A server class, corresponding to `*Server.c`, would be able to handle + incoming messages using a user-provided implementation of the interface. + (Possibly, a skeleton class providing methods which would raise + `NotImplementedException`s would be provided as well. + Users would derive from this class and override the relevant methods. + This would allow them not to implement some operations, + and would avoid pre-existing code from breaking when new operations are + introduced.) + +In order to help with the implementation of servers, some kind of library +would be needed to associate Mach receive rights with server objects and to +handle incoming messages on dedicated threads, in the spirit of libports. +This would probably require support for port sets at the level of the Mach +primitives described in the previous section. + +When possible, operations involving the transmission of send rights +of some kind would be expressed in terms of the MIG-generated interfaces +instead of `MachPort` objects. +Upon reception of a send right, +a `FooUser` object would be created +and associated with the corresponding `MachPort` object. +If the received send right corresponds to a local port +to which a server object has been associated, +this object would be used instead. +This way, +subsequent operations on the received send right +would be handled as direct method calls +instead of going through RPC mechanisms. + +Some issues will still need to be solved regarding how MIG will convert +interface description files to Java interfaces. For instance: + + * `.defs` files are not explicitly associated with a type. For instance in + the example above, MIG would have to somehow infer that io_t corresponds + to `this` in the `Io` interface. + * More generally, a correspondence between MIG and Java types would have + to be determined. Ideally this would be automated and not hardcoded + too much. + * Initially, reply port parameters would be ignored. However they may be + needed for some applications. + +So the details would need to be flushed out during the community bonding +period and as the implementation progresses. However I’m confident that a +satisfactory solution can be designed. + +Using these new features, the example above could be rewritten as: + + import org.gnu.hurd.InitPorts; + import org.gnu.hurd.Io; + import org.gnu.hurd.IoUser; + + class Hello { + static void main(String argv[]) throws ... + { + Io stdout = new IoUser(InitPorts.getdport(1)); + String hello = “Hello, World!\n”; + + int amount = stdout.write(hello.getBytes(), -1); + + /* (A retCode corresponding to an error + would be signalled as an exception.) */ + } + } + +An example of server implementation would be: + + import org.gnu.hurd.Io; + import java.util.Arrays; + + class HelloIo implements Io { + final byte[] contents = “Hello, World!\n”.getBytes(); + + int write(byte[] data, int offset) { + return SOME_ERROR_CODE; + } + + byte[] read(int offset, int amount) { + return Arrays.copyOfRange(contents, offset, + offset + amount - 1); + } + + /* ... */ + } + +A new server object could then be created with `new IoServer(new HelloIo())`, +and associated with some receive right at the level of the ports management +library. + +### Base classes for common types of translators + +Once MIG can target Java code, and a libports equivalent is available, +creating new translators in Java would be greatly facilitated. However, +we would probably want to introduce basic implementations of file system +translators in the spirit of libtrivfs or libnetfs. They could take the form +of base classes implementing the relevant MIG-generated interfaces which +would then be derived by users, +or could define a simpler interface +which would then be used by adapter classes +to implement the required ones. + +I would draw inspiration from libtrivfs and libnetfs +to design and implement similar solutions for Java. + +### Deliverables + + * A hurd-java package would contain the Java code developed + in the context of this project. + * The Java code would be documented using javadoc + and a tutorial for writing translators would be written as well. + * Modifications to MIG would be submitted upstream, + or a patched MIG package would be made available. + +The Java libraries resulting from this work, +including any MIG support classes +as well as the class files built from the MIG-generated code +for the Mach and Hurd interface definition files, +would be provided as single `hurd-java` package for +Debian GNU/Hurd. +This package would be separate from both Hurd and Mach, +so as not to impose unreasonable build dependencies on them. + +I expect I would be able to act as its maintainer in the foreseeable future, +either as an individual or as a part of the Hurd team. +Hopefully, +my code would be claimed by the Hurd project as their own, +and consequently the modifications to MIG +(which would at least conceptually depend on the Mach Java package) +could be integrated upstream. + +Since by design, +the Java code would use only a small number of stable interfaces, +it would not be subject to excessive amounts of bitrot. +Consequently, +maintenance would primarily consist in +fixing bugs as they are reported, +and adding new features as they are requested. +A large number of such requests +would mean the package is useful, +so I expect that the overall amount of work +would be correlated with the willingness of more people +to help with maintenance +should I become overwhelmed or get hit by a bus. + + +## Timeline + +The dates listed are deadlines for the associated tasks. + + * *Community bonding period.* + Discuss, refine and complete the design of the Java bindings + (in particular the MIG and "libports" parts) + * *May 23.* + Coding starts. + * *May 30.* + Finish implementing pthread signal semantics. + * *June 5.* + Port OpenJDK + * *June 12.* + Fix the remaining problems with GCJ and/or OpenJDK, + possibly port Eclipse or other big Java packages. + * *June 19.* + Create the bindings for Mach. + * *June 26.* + Work on some kind of basic Java libports + to handle receive rights. + * *July 3.* + Test, write some documentation and examples. + * *July 17 (two weeks).* + Add the Java target to MIG. + * *July 24.* + Test, write some documentation and examples. + * *August 7 (two weeks).* + Implement a modular libfoofs to help with translator development. + Try to write a basic but non-trivial translator + to evaluate the performance and ease of use of the result, + rectify any rough edges this would uncover. + * *August 22. (last two weeks)* + Polish the code and packaging, + finish writing the documentation. + + +## Conclusion + +This project is arguably ambitious. +However, I have been thinking about it for some time now +and I'm confident I would be able to accomplish most of it. + +In the event multiple language bindings projects +would be accepted, +some work could probably be done in common. +In particular, +[ArneBab](http://www.bddebian.com/~hurd-web/community/weblogs/ArneBab/2011-04-06-application-pyhurd/) +seems to favor a low-level approach for his Python bindings as I do for Java, +and I would be happy to discuss API design and coordinate MIG changes with him. +I would also have an extra month after the end of the GSoC period +before I go back to school, +which I would be able to use to finish the project +if there is some remaining work. +(Last year's rewrite of procfs was done during this period.) + +As for the project's benefits, +I believe that good support for Java +is a must-have for the Hurd. +Java bindings would also further the Hurd's agenda +of user freedom by extending this freedom to more people: +I expect the set of developers +who would be able to write Java code against a well-written libfoofs +is much larger than +those who master the intricacies of low-level systems C programming. +From a more strategic point of view, +this would also help recruit new contributors +by providing an easier path to learning the inner workings of the Hurd. + +Further developments +which would build on the results of this project +include my planned [[experiment with Joe-E|objcap]] +(which I would possibly take on as a university project next year). +Another possibility would be to reimplement some parts +of the Java standard library +directly in terms of the Hurd interfaces +instead of using the POSIX ones through glibc. +This would possibly improve the performance +of some Java applications (though probably not by much), +and would otherwise be a good project +for someone trying to get acquainted with Hurd. + +Overall, I believe this project would be fun, interesting and useful. +I hope that you will share this sentiment +and give me the opportunity to spend another summer working on Hurd. + -- cgit v1.3.1 From 97ef5d15ef5e44806b92da486c5f06311db14727 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Wed, 15 Jun 2011 19:01:24 +0200 Subject: user/jkoenig/java: Integrate most of the action items from the discussion page. --- user/jkoenig/java.mdwn | 91 +++++++++++++++++++++++++++++++- user/jkoenig/java/discussion.mdwn | 108 -------------------------------------- 2 files changed, 90 insertions(+), 109 deletions(-) diff --git a/user/jkoenig/java.mdwn b/user/jkoenig/java.mdwn index 90f51028..fcd316b7 100644 --- a/user/jkoenig/java.mdwn +++ b/user/jkoenig/java.mdwn @@ -38,7 +38,7 @@ I have submitted [preliminary patches](http://lists.gnu.org/archive/html/bug-hurd/2011-05/msg00182.html) for global signal dispositions, which I'm currently testing. -I have since fixed a few thinks and implemented `SA_SIGINFO` +I have since fixed a few things and implemented `SA_SIGINFO` (which is required by OpenJDK.) My latest code is available on [github](http://github.com/jeremie-koenig/glibc/commits/master-beware-rebase), @@ -56,6 +56,7 @@ at which point we will alias the interim version to the new one in debian packages. I have modified libc0.3 to include a `deb-symbols(5)` file +(alternatively see ) so that we get an accurate libc dependency in `hurd` and other packages when the symbols in question are pulled in. @@ -71,6 +72,28 @@ I expect only the last patch (implement global dispositions) will change, and new ones will be added on top of it. +##### Open Items + + * Test patches: in progress, [[jkoenig]], Svante. More volunteers welcome, + of course. + + * If [[jkoenig]] thinks it's mature enough: should ask Samuel to test this + (that is, only the refactoring patches for starters?) on the buildds. + + * Get patches reviewed (Roland?), and integrated into official sources: [!] + [[tschwinge]]. + + * Documentations bits (from [[proposal]] and elsewhere) should probably be + moved either into the appropriate glibc or Hurd documentation + files/reference manuals, or to [[glibc/signal]]. + + * libthreads (cthreads) integration. + + * [[tschwinge]] suggests to leave them as-is? + + * [[libpthread]] integration. + + ### Port OpenJDK As suggested by [[tschwinge]], I have targeted OpenJDK 7 at first. @@ -107,8 +130,74 @@ although the current toolchain issues have so far prevented me from testing it. +##### Open Items + + * They seem to have a rather heavy-weight process for such projects: confer + , + for example. Do we need this, too? + + * Eclipse + + OK for testing -- but I'd very much hope that it *just works* as soon as we + provide the required Java platform. But it may perhaps have some + Linux-specifics (needlessly?) in its basement. Is it available for Debian + GNU/kFreeBSD already? + + ### Java bindings for Mach + +#### Plans + (just started.) +##### Open Items + + * [[tschwinge]] has to read about RMI and CORBA. + + * MIG + + * Hacking [[microkernel/mach/MIG]] shouldn't be too difficult. + + * (Unless you want to make MIG's own code (that is, not the generated + code, but MIG itself) look a bit more nice, too.) ;-) + + * There are also alternatives to MIG. If there is interest, the following + could be considered: + + * FLICK ([[!GNU_Savannah_task 5723]]). [[tschwinge]] has no idea yet if + there would be any benefits over MIG, like better modularity (for the + backends)? If we feel like it, we could spend a little bit of time on + this. + + * For [[microkernel/Viengoos]], Neal has written a RPC stub generator + entirely in C Preprocessor macros. While this is obviously not + directly applicable, perhaps we can get some ideas from it. + + * Anything else that would be worth having a look at? (What are other + microkernels using?) + + * `mach_msg` + + * Seems like the right approach to [[tschwinge]], but he hasn't digested + all the pecularities yet. Will definitely need more time. + + +## Postponed + +Might get back to these as time/interest permits. + + +### GCJ + + * [[tschwinge]] has the feeling that Java in GCC (that is, GCJ) is mostly + dead? (True?) + + * Thus perhaps not too much effort should be spent with it. + + If the POSIX threads signal semantics makes it going, then great, otherwise + we should get a feeling what else is missing. + + +### Joe-E. diff --git a/user/jkoenig/java/discussion.mdwn b/user/jkoenig/java/discussion.mdwn index 0131d8d5..266a7bcc 100644 --- a/user/jkoenig/java/discussion.mdwn +++ b/user/jkoenig/java/discussion.mdwn @@ -56,114 +56,6 @@ one person's summer.) Such bits could be moved to [[open_issues]] pages, either new ones or existing ones, as applicable. -# POSIX Threads Signal Semantics - - * Great! [[tschwinge]] had a brief look, and should have a deeper one. - - * If [[jkoenig]] thinks it's mature enough: should ask Samuel to test this - (that is, only the refactoring patches for starters?) on the buildds. - - * Then: should ask Roland to review. - - * Documentations bits should probably be moved to [[glibc/signal]]. - - -## libthreads (cthreads) Integration - - * [[tschwinge]] suggests to leave them as-is? - - -## [[libpthread]] integration - - * To be done. - - -# Java - - * [[tschwinge]] has to read about RMI and CORBA. - - -# Joe-E - - * For later. - - -# GCJ - - * [[tschwinge]] has the feeling that Java in GCC (that is, GCJ) is mostly - dead? (True?) - - * Thus perhaps not too much effort should be spent with it. - - If the POSIX threads signal semantics makes it going, then great, otherwise - we should get a feeling what else is missing. - - -# OpenJDK - - * All in all, [[tschwinge]] has the feeling that a working OpenJDK will be - more useful/powerful than GCJ. - - * We need to get a feeling how difficult such an OS port will be. - - * [[jkoenig]] suggests OpenJDK 6 -- should we directly go for version 7 - instead? - - * What are the differences (regarding the OS port) between the two - versions? Or this there something even more recent to be worked upon, - for new OS ports? - - * Perhaps the different versions' OS port specific stuff is not at - all very different, so that both v6 and v7 could be done? - - * They seem to have a rather heavy-weight process for such projects: confer - , - for example. Do we need this, too? - - -# Eclipse - -OK for testing -- but I'd very much hope that it *just works* as soon as we -provide the required Java platform. - - -# Java Bindings - - -## Design Principles - - * Generally ack. - - -### MIG - - * Hacking [[microkernel/mach/MIG]] shouldn't be too difficult. - - * (Unless you want to make MIG's own code (that is, not the generated - code, but MIG itself) look a bit more nice, too.) ;-) - - * There are also alternatives to MIG. If there is interest, the following - could be considered: - - * FLICK ([[!GNU_Savannah_task 5723]]). [[tschwinge]] has no idea yet if - there would be any benefits over MIG, like better modularity (for the - backends)? If we feel like it, we could spend a little bit of time on - this. - - * For [[microkernel/Viengoos]], Neal has written a RPC stub generator - entirely in C Preprocessor macros. While this is obviously not - directly applicable, perhaps we can get some ideas from it. - - * Anything else that would be worth having a look at? (What are other - microkernels using?) - - -### `mach_msg` - - * Seems like the right approach to [[tschwinge]], but hasn't digested all the - pecularities yet. Will definitely need more time. - - # GSoC Site Discussion * Discussion items from -- cgit v1.3.1 From 5693033a406e3c5f1f536435b8d6c40bd2ef4df3 Mon Sep 17 00:00:00 2001 From: Jeremie Koenig Date: Wed, 15 Jun 2011 21:10:03 +0200 Subject: link to pochu's patches --- user/jkoenig/java.mdwn | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/user/jkoenig/java.mdwn b/user/jkoenig/java.mdwn index fcd316b7..5702a0f2 100644 --- a/user/jkoenig/java.mdwn +++ b/user/jkoenig/java.mdwn @@ -123,7 +123,8 @@ the `LD_ORIGIN_PATH` environment variable. #### Plans I intend to fix the RPATH issue -by building on [[pochu]]'s `file_exec_file_name()` patches. +by building on [[pochu]]'s `file_exec_file_name()` +[patches](http://lists.gnu.org/archive/html/bug-hurd/2010-08/msg00023.html). I have succeeded in building a Hotspot-enabled `libjvm.so`, although the current toolchain issues -- cgit v1.3.1 From c0391c22faaca583d91a5ddb88fb59d409a1d855 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Wed, 15 Jun 2011 22:06:16 +0200 Subject: user/jkoenig/java: IRC meeting updates. --- user/jkoenig/java.mdwn | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/user/jkoenig/java.mdwn b/user/jkoenig/java.mdwn index fcd316b7..af2b908e 100644 --- a/user/jkoenig/java.mdwn +++ b/user/jkoenig/java.mdwn @@ -60,6 +60,12 @@ I have modified libc0.3 to include a `deb-symbols(5)` file so that we get an accurate libc dependency in `hurd` and other packages when the symbols in question are pulled in. +[[hurd/libthreads]] (cthreads library) will not be changed. There's no reason +why its behavior should change, whereas for [[libpthread]] it's needed for +conformance. Patches posted on 2011-05-25, but there's a more recent one in +the modified hurd package (adds `_hurd_sigstate_delete` and removes the weak +symbols). + Another issue which came up with OpenJDK is the expansion by the dynamic linker of `$ORIGIN` in the `RPATH` header, see below. @@ -77,8 +83,8 @@ and new ones will be added on top of it. * Test patches: in progress, [[jkoenig]], Svante. More volunteers welcome, of course. - * If [[jkoenig]] thinks it's mature enough: should ask Samuel to test this - (that is, only the refactoring patches for starters?) on the buildds. + * If [[jkoenig]] thinks it's mature enough: should ask + [[Samuel|samuelthibault]] to test these patches on the buildds. * Get patches reviewed (Roland?), and integrated into official sources: [!] [[tschwinge]]. @@ -87,11 +93,12 @@ and new ones will be added on top of it. moved either into the appropriate glibc or Hurd documentation files/reference manuals, or to [[glibc/signal]]. - * libthreads (cthreads) integration. + * `SA_SIGINFO` patche is based on [[Samuel|samuelthibault]]'s earlier work. + Thus, have him review the new patch? - * [[tschwinge]] suggests to leave them as-is? + * Perhaps have a look at `SA_NOCLDWAIT`. - * [[libpthread]] integration. + * Which numeric values to use for `SA_SIGINFO` (and `SA_NOCLDWAIT`)? ### Port OpenJDK @@ -105,6 +112,11 @@ should not be too hard. If that fails we can fall back on Shark (a portable alternative JIT which uses LLVM). +Complexity of porting HotSpot: probably low. The complex things should be +arch- rather than OS-specific. Not many Linux-specific interfaces used. +Garbage collection/memory management, etc. and/or most of other Linux-specific +interfaces are already dealt with for the zero build. + The dynamic linker issue is as follows. An executable-specific search path can be provided in the ELF RPATH header. RPATH directories can include the special string `$ORIGIN`, @@ -127,11 +139,19 @@ by building on [[pochu]]'s `file_exec_file_name()` patches. I have succeeded in building a Hotspot-enabled `libjvm.so`, although the current toolchain issues +(*Hurd OS ABI*) have so far prevented me from testing it. ##### Open Items + * [!] [[tschwinge]] to have a look at [[pochu]]'s `file_exec_file_name()` + patches (2010-05), whether it's generally the right idea. + + * Assuming it is, continue with getting `$ORIGIN` working. + + * [!] [[Samuel|samuelthibault]]/[[tschwinge]]/[[jkoenig]]: *Hurd OS ABI*. + * They seem to have a rather heavy-weight process for such projects: confer , for example. Do we need this, too? -- cgit v1.3.1 From a5b95baab6b4512ceaef8749a29ed7b3685b4125 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Wed, 15 Jun 2011 22:54:09 +0200 Subject: Some more bits from IRC and elsewhere. --- hurd/running/qemu/discussion.mdwn | 92 +++++++++++++++++++++++++++++++++ microkernel/mach/gnumach/ports/xen.mdwn | 24 +++++++-- toolchain.mdwn | 8 ++- toolchain/elfosabi_hurd.mdwn | 83 +++++++++++++++++++++++++++++ user/jkoenig/java.mdwn | 5 +- 5 files changed, 204 insertions(+), 8 deletions(-) create mode 100644 toolchain/elfosabi_hurd.mdwn diff --git a/hurd/running/qemu/discussion.mdwn b/hurd/running/qemu/discussion.mdwn index fd0df4c5..1ce14b01 100644 --- a/hurd/running/qemu/discussion.mdwn +++ b/hurd/running/qemu/discussion.mdwn @@ -50,3 +50,95 @@ The problem is actually that the linux block cache doesn't make any consistency between /dev/hda and /dev/hda6, so if you give /dev/hda to qemu, qemu writings won't be consistent with mounting /dev/hda6 in linux. You can give /dev/hda6 directly to qemu and it will be fine. + + +# Host-side Writeback Caching + +IRC, freenode, #hurd, 2011-06-07 + + hm, i guess i should have used cache=writeback with kvm before + starting the debian installer :/ + ah yes, much better + this shows how poor the state of our I/O drivers and subsystem is + :/ + indeed... still no clustered pageout :-( + and no I/O scheduler either + although an I/O scheduler has limited value without clustered + pageouts + since one of its goals is to pack related I/O requests together eh + i wonder if the wiki mentions using cache=writeback to speed up + qemu performances + it would help those unable to use kvm a lot + and even those running kvm too + kvm -m $RAM \ -monitor stdio \ -drive + cache=writeback,index=0,media=disk,file=hd0.img \ + etc.. + the idea is that qemu doesn't open its disk file synchronously + changes are queued in the host page cache before being flushed to + the disk image + but if you brutally close your qemu instance, you're likely to + loose file system consistency + ext2fs will think it has committed its metadata to the disk, but + the disk image won't be updated synchronously + on my machine (which is quite fast), my kvm has installed debian + like 10 times faster than without the option + braunr: I don't think killing qemu should hurt in this + case... probably only matters when the host machine dies + antrik: ah yes, right + it really makes everything faster, even downloading, since I/O + requests aren't interleaved between networking RPCs + regarding I/O sheduler... this discussion came up before, but I + don't remember the outcome -- doesn't the glued Linux driver actually + come with one? + i don't remember either + braunr: err... I don't think interleaving has anything to do with + it... I guess it's simply the fact that downloading writes the result to + disk, which suffers from lacking clustered pageout like everything else + (my internet connection is too slow though to notice :-) ) + well, if there is no I/O during downloading, downloading is faster + :) + +IRC, freenode, #hurd, 2011-06-08 + + youpi: does xen provide disk caching options ? + through a blktap, probably + ok + +([[microkernel/mach/gnumach/ports/Xen]], *Host-side Writeback Caching*.) + + we should find the pages mentioning qemu on the wiki and add the + options to enable disk image caching + it really makes the hurd run a lot faster + as a workaround for emulators until I/O is reworked, ofc + +IRC, freenode, #hurd, 2011-06-09 + + braunr recommends to use writeback caching with kvm. Is this + really recommended with the frequent crashes I experience? + provided that you terminate your kvm normaly (i.e. quitting it, not + killing it), there should be no difference + I think the host's stability is what matters + the data presumably sits in linux's cache even if qemu dies + violently + But the freezes I see force me to kill kvm :-( + maybe kvm doesn't even do caching indeed, I don't know + gnu_srs: you can quit even when frozen + use the console + (the kvm console) + gnu_srs, "Writeback caching will report data writes as completed + as soon as the data is present in the host page cache. This is safe as + long as you trust your host. If your host crashes or loses power, then + the guest may experience data corruption." (from the qemu manpage) + +IRC, freenode, #hurd, 2011-06-11 + + braunr: If you are online. For me setting the parameters -drive + cache=writeback,index=0,media=disk,file=hd0.img does not show any speed + improvement at all compared to the default. + gnu_srs: what's your complete qemu command line ? + kvm -m 1024 -net nic,model=rtl8139 -net + user,hostfwd=tcp::5556-:22 -drive + cache=writeback,index=0,media=disk,file=hd0.img -cdrom netinst.iso + what qemu version ? + qemu-kvm 0.14.1+dfsg-1: Sorry, I cannot be online until + tomorrow again. diff --git a/microkernel/mach/gnumach/ports/xen.mdwn b/microkernel/mach/gnumach/ports/xen.mdwn index bf26410a..af431c92 100644 --- a/microkernel/mach/gnumach/ports/xen.mdwn +++ b/microkernel/mach/gnumach/ports/xen.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2007, 2008, 2009 Free Software Foundation, +[[!meta copyright="Copyright © 2007, 2008, 2009, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -6,11 +6,12 @@ id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!toc]] + # Xen dom0, hypervisor /!\ Now that GNU Mach handles PAE you can use a PAE-enabled hypervisor. @@ -30,6 +31,7 @@ If you have a free partition, you can fdisk to type 0x83, create a filesystem us Replace /dev/sda4 with your partition. Install and use crosshurd to setup a GNU/Hurd system on this partition. + # /etc/xen/hurd configuration Here is a sample /etc/xen/hurd configuration @@ -49,6 +51,7 @@ Suggestions about [[networking_configuration]] are available. If you need stable MAC addresses, use a syntax like `vif = [ 'mac=00:16:3e:XX:XX:XX, bridge=br0' ]`. + # Running Hurd with Xen To run Hurd with Xen, use: @@ -63,6 +66,7 @@ and gnumach should get started. Proceed with native-install. - If `xm` complains about networking (`vif could not be connected`), it's Xen scripts' fault, see Xen documentation for how to configure the network. The simplest way is network-bridge with fixed IPs (note that you need the bridge-utils package for this). You can also just disable networking by commenting the vif line in the config. - If `xm` complains `Error: (2, 'Invalid kernel', 'xc_dom_compat_check: guest type xen-3.0-x86_32 not supported by xen kernel, sorry\n')`, you most probably have a PAE-enabled hypervisor and a non-PAE gnumach. Either install and boot non-PAE hypervisor and kernel, or rebuilt gnumach in PAE mode. + # Building from sources If you want to generate these images, first get the `gnumach-1-branch-Xen-branch` branch from gnumach CVS. @@ -75,7 +79,6 @@ Then use The current `hurd-modules` was built from the debian packages `hurd 20070606-2` and `libc0.3 2.6.1-1`. /!\ This means that when using this image, your GNU/Hurd system also needs to be a glibc version 2.6 or later-based one! ---- # Miscellaneous @@ -83,7 +86,6 @@ The current `hurd-modules` was built from the debian packages `hurd 20070606-2` [[!GNU_Savannah_task 5468]], [[!GNU_Savannah_task 6584]]. ---- # `pv-grub` @@ -91,3 +93,15 @@ From Xen 4.0 on you'll be able to run the GNU Hurd directly using `pv-grub`, without the need to [prepare a special bootstrap image](http://youpibouh.thefreecat.org/hurd-xen/build_hurd-modules) (like an initrd). + + +# Host-side Writeback Caching + +Optimization possible as it is with [[QEMU|hurd/running/qemu/discussion]], +*Host-side Writeback Caching*? + +IRC, freenode, #hurd, 2011-06-08 + + youpi: does xen provide disk caching options ? + through a blktap, probably + ok diff --git a/toolchain.mdwn b/toolchain.mdwn index b08dba95..0059317b 100644 --- a/toolchain.mdwn +++ b/toolchain.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2007, 2008, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2007, 2008, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -24,4 +24,10 @@ for example, how things are done differently for GNU/Hurd than they are done for GNU/Linux. +--- + + * [[ELFOSABI_HURD]] + +--- + * [[cross-gnu]] diff --git a/toolchain/elfosabi_hurd.mdwn b/toolchain/elfosabi_hurd.mdwn new file mode 100644 index 00000000..4d60f761 --- /dev/null +++ b/toolchain/elfosabi_hurd.mdwn @@ -0,0 +1,83 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="ELFOSABI_HURD"]] + +[[!tag open_issue_binutils open_issue_glibc]] + + +# [[!debbug 630180]] + + +# [[open_issues/binutils]] commit 51b2f560ad035dffad3371093f8e5c80608d196c + +Usage of `ELFOSABI_LINUX`/`STB_GNU_UNIQUE`. Has also been wrong before already +with respect to `STT_GNU_IFUNC`? + + +# IRC + +IRC, freenode, #debian-hurd, 2011-06-11 + + youpi: not that there would be any hope in that, but id you try + asking doko about the gcc miscompiling (wrong elf format) issue? + I didn't + I'm still investigating + maybe it's a binutils change actually + youpi: hm, are you sure it could be binutils? after all, even + some .o files are produced with format gnu/linux, so there's no binutils + involved up to that point of thecompilation yet? + as is + "as", I mean + i see + since it's so unclear, I really prefer to investigate before + bothering doko + youpi: maybe i could be wrong, + in binutils, bfd/elf.c, around lines 9580 + the faulty thing seems to be gnu_unique_object in the source .s + file produced by g++ + that's what that comment (which changed wrt binutils from eg + march) says + seems to concur with my comment above :) + http://paste.debian.net/119542/ ‘¡û extract of diff + ok, that really seems the culprit + starting reportbug + who's the fault then? + binutils + it shouldn't hardcode LINUX + g++ emitting those symbols, or binutil considering them "linux"? + it's a GNU thing, not a Linux thing + ah ok + it's the same dynamic linker actually + youpi: http://sourceware.org/bugzilla/show_bug.cgi?id=10549 + see the reporter :) + heh + youpi: see also gas/config/obj-elf.c:1725 + (another change related to that bug, it seems) + +IRC, freenode, #hurd, 2011-06-15 + + also I still get an "ELF file OS ABI invalid" error with binutils + 2.21.52.20110606-2+hurd.1, is that expected? + tschwinge: oops, the OS ABI invalid is actually due to the file + being marked GNU/Hurd + I guess the linker is simply not aware that it should accept + GNU/Hurd + youpi: So we got to work on glibc'S ld.so to teach it aboput + the Hurd OS ABI? (Or probably simply make that equivalent to the Linux + one?) + probably simply an equivalent + ELFOSABI_HURD is missing from elf/elf.h, for a start... + linux' glibc has tests in lsdodefs.h + the VALID_ELF_OSABI macro + it's thus apparently a matter of providing an ldsodefs.h file with + VALID_ELF_HEADER, VALID_ELF_OSABI and VALID_ELF_ABIVERSION definitions + (and include_next the generic one) + I've prepared a patch for ldsodefs.h, I'll test it diff --git a/user/jkoenig/java.mdwn b/user/jkoenig/java.mdwn index 6aeef9fe..8e1fb097 100644 --- a/user/jkoenig/java.mdwn +++ b/user/jkoenig/java.mdwn @@ -140,7 +140,7 @@ by building on [[pochu]]'s `file_exec_file_name()` I have succeeded in building a Hotspot-enabled `libjvm.so`, although the current toolchain issues -(*Hurd OS ABI*) +([[toolchain/ELFOSABI_HURD]]) have so far prevented me from testing it. @@ -151,7 +151,8 @@ have so far prevented me from testing it. * Assuming it is, continue with getting `$ORIGIN` working. - * [!] [[Samuel|samuelthibault]]/[[tschwinge]]/[[jkoenig]]: *Hurd OS ABI*. + * [!] [[Samuel|samuelthibault]]/[[tschwinge]]/[[jkoenig]]: + [[toolchain/ELFOSABI_HURD]]. * They seem to have a rather heavy-weight process for such projects: confer , -- cgit v1.3.1 From 0eb99732fdee36b575de01215548a8ed5324f136 Mon Sep 17 00:00:00 2001 From: "https://me.yahoo.com/a/_NM8WaIwh4f4wFIbBoanB_kTeIAy51oft76mag--#3f4b4" Date: Thu, 16 Jun 2011 17:49:35 +0000 Subject: --- documentation.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation.mdwn b/documentation.mdwn index 5c666f3f..954ef1a9 100644 --- a/documentation.mdwn +++ b/documentation.mdwn @@ -27,7 +27,7 @@ Documentation for... * 2004-07-02 - Ognyan Kulev, *presentation of the Hurd*, at the seminar *LIO and friends*, + Ognyan Kulev, [[presentation|http://loseeweightfast.webs.com]] of the Hurd, at the seminar *LIO and friends*, , in Bulgarian. -- cgit v1.3.1 From 45bcbc3f3a1d7181eb963a6da571e802556185f7 Mon Sep 17 00:00:00 2001 From: "https://me.yahoo.com/a/_NM8WaIwh4f4wFIbBoanB_kTeIAy51oft76mag--#3f4b4" Date: Thu, 16 Jun 2011 17:55:27 +0000 Subject: --- documentation.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation.mdwn b/documentation.mdwn index 954ef1a9..78be5627 100644 --- a/documentation.mdwn +++ b/documentation.mdwn @@ -27,7 +27,7 @@ Documentation for... * 2004-07-02 - Ognyan Kulev, [[presentation|http://loseeweightfast.webs.com]] of the Hurd, at the seminar *LIO and friends*, + Ognyan Kulev, presentation of the Hurd, at the seminar *LIO and friends*, , in Bulgarian. -- cgit v1.3.1 From 0af46c58452d63160097c8d5480cd16eac4b49c8 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Fri, 17 Jun 2011 06:45:54 +0200 Subject: Restore original state, ban user. This reverts commits 45bcbc3f3a1d7181eb963a6da571e802556185f7 and 0eb99732fdee36b575de01215548a8ed5324f136. --- documentation.mdwn | 2 +- ikiwiki.setup | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/documentation.mdwn b/documentation.mdwn index 78be5627..5c666f3f 100644 --- a/documentation.mdwn +++ b/documentation.mdwn @@ -27,7 +27,7 @@ Documentation for... * 2004-07-02 - Ognyan Kulev, presentation of the Hurd, at the seminar *LIO and friends*, + Ognyan Kulev, *presentation of the Hurd*, at the seminar *LIO and friends*, , in Bulgarian. diff --git a/ikiwiki.setup b/ikiwiki.setup index 3ef1bb73..9ae1f669 100644 --- a/ikiwiki.setup +++ b/ikiwiki.setup @@ -39,8 +39,8 @@ IkiWiki::Setup::Standard->import({ # users who are wiki admins adminuser => [qw{tschwinge}], # users who are banned from the wiki - banned_users => [qw{AlbertF bernhart ColetCris flamberian jasclaine - NateNash + banned_users => [qw{AlbertF bernhart ColetCris flamberian Jack_Dark + jasclaine NateNash http://calvinyoung.myopenid.com/ http://heaton.myopenid.com/ http://hilarybunton.myopenid.com/}], -- cgit v1.3.1