Merge commit '2462ca1a8be02689b5e0c5a13038472ad29a7888'

7 files changed, 282 insertions, 7 deletions

diff --git a/community/gsoc/project_ideas.mdwn b/community/gsoc/project_ideas.mdwn
index da247847..030c18f3 100644
--- a/community/gsoc/project_ideas.mdwn
+++ b/community/gsoc/project_ideas.mdwn
@@ -89,10 +89,10 @@ Here are these pages' content:
 [[!inline pages="community/gsoc/project_ideas/server_overriding" show=0 feeds=no actions=yes]]
 [[!inline pages="community/gsoc/project_ideas/tcp_ip_stack" show=0 feeds=no actions=yes]]
 [[!inline pages="community/gsoc/project_ideas/nfs" show=0 feeds=no actions=yes]]
-[[!inline pages="open_issues/locking" show=0 feeds=no actions=yes]]
+[[!inline pages="community/gsoc/project_ideas/libdiskfs_locking" show=0 feeds=no actions=yes]]
 [[!inline pages="community/gsoc/project_ideas/pthreads" show=0 feeds=no actions=yes]]
 [[!inline pages="community/gsoc/project_ideas/sound" show=0 feeds=no actions=yes]]
-[[!inline pages="open_issues/performance/io_system" show=0 feeds=no actions=yes]]
+[[!inline pages="community/gsoc/project_ideas/disk_io_performance" show=0 feeds=no actions=yes]]
 [[!inline pages="community/gsoc/project_ideas/vm_tuning" show=0 feeds=no actions=yes]]
 [[!inline pages="community/gsoc/project_ideas/mtab" show=0 feeds=no actions=yes]]
 [[!inline pages="community/gsoc/project_ideas/gnumach_cleanup" show=0 feeds=no actions=yes]]
@@ -114,4 +114,4 @@ Here are these pages' content:
 [[!inline pages="open_issues/unit_testing" show=0 feeds=no actions=yes]]
 [[!inline pages="community/gsoc/project_ideas/libcap" show=0 feeds=no actions=yes]]
 [[!inline pages="community/gsoc/project_ideas/xattr" show=0 feeds=no actions=yes]]
-[[!inline pages="open_issues/valgrind" show=0 feeds=no actions=yes]]
+[[!inline pages="community/gsoc/project_ideas/valgrind" show=0 feeds=no actions=yes]]
diff --git a/community/gsoc/project_ideas/disk_io_performance.mdwn b/community/gsoc/project_ideas/disk_io_performance.mdwn
new file mode 100644
index 00000000..b6f223c8
--- /dev/null
+++ b/community/gsoc/project_ideas/disk_io_performance.mdwn
@@ -0,0 +1,48 @@
+[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation,
+Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
+
+[[!meta title="Disk I/O Performance Tuning"]]
+
+The most obvious reason for the Hurd feeling slow compared to mainstream
+systems like GNU/Linux, is a low I/O system performance, in particular very
+slow hard disk access.
+
+The reason for this slowness is lack and/or bad implementation of common
+optimization techniques, like scheduling reads and writes to minimize head
+movement; effective block caching; effective reads/writes to partial blocks;
+[[reading/writing multiple blocks at once|clustered_page_faults]]; and
+[[read-ahead]].  The
+[[ext2_filesystem_server|hurd/translator/ext2fs]] might also need some
+optimizations at a higher logical level.
+
+The goal of this project is to analyze the current situation, and implement/fix
+various optimizations, to achieve significantly better disk performance. It
+requires understanding the data flow through the various layers involved in
+disk access on the Hurd ([[filesystem|hurd/virtual_file_system]],
+[[pager|hurd/libpager]], driver), and general experience with
+optimizing complex systems.  That said, the killing feature we are definitely
+missing is the [[read-ahead]], and even a very simple implementation would bring
+very big performance speedups.
+
+Here are some real testcases:
+
+  * [[binutils_ld_64ksec]];
+
+  * running the Git testsuite which is mostly I/O bound;
+
+  * use [[TopGit]] on a non-toy repository.
+
+
+Possible mentors: Samuel Thibault (youpi)
+
+Exercise: Look through all the code involved in disk I/O, and try something
+easy to improve. It's quite likely though that you will find nothing obvious --
+in this case, please contact us about a different exercise task.
diff --git a/community/gsoc/project_ideas/dtrace.mdwn b/community/gsoc/project_ideas/dtrace.mdwn
new file mode 100644
index 00000000..252c303a
--- /dev/null
+++ b/community/gsoc/project_ideas/dtrace.mdwn
@@ -0,0 +1,49 @@
+[[!meta copyright="Copyright © 2008, 2009, 2011 Free Software Foundation,
+Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
+
+[[!meta title="dtrace Support"]]
+
+One of the main problems of the current Hurd implementation is very poor
+[[performance]]. While we have a bunch of ideas what could cause the performance
+problems, these are mostly just guesses. Better understanding what really
+causes bad performance is necessary to improve the situation.
+
+For that, we need tools for performance measurements. While all kinds of more
+or less specific [[profiling]] tools could be conceived, the most promising and
+generic approach seems to be a framework for logging certain events in the
+running system (both in the microkernel and in the Hurd servers). This would
+allow checking how much time is spent in certain modules, how often certain
+situations occur, how things interact, etc. It could also prove helpful in
+debugging some issues that are otherwise hard to find because of complex
+interactions.
+
+The most popular framework for that is Sun's dtrace; but there might be others.
+The student has to evaluate the existing options, deciding which makes most
+sense for the Hurd; and implement that one. (Apple's implementation of dtrace
+in their Mach-based kernel might be helpful here...)
+
+This project requires ability to evaluate possible solutions, and experience
+with integrating existing components as well as low-level programming.
+
+Possible mentors: Samuel Thibault (youpi)
+
+Related: [[profiling]], [[LTTng]], [[SystemTap]]
+
+Exercise: In lack of a good exercise directly related to this task, just pick
+one of the kernel-related or generally low-level tasks from the bug/task
+trackers on savannah, and make a go at it. You might not be able to finish the
+task in a limited amount of time, but you should at least be able to make a
+detailed analysis of the issue.
+
+*Status*: Andei Barbu was working on
+[SystemTap](http://csclub.uwaterloo.ca/~abarbu/hurd/) for GSoC 2008, but it
+turned out too Linux-specific. He implemented kernel probes, but there is no
+nice frontend yet.
diff --git a/community/gsoc/project_ideas/libdiskfs_locking.mdwn b/community/gsoc/project_ideas/libdiskfs_locking.mdwn
new file mode 100644
index 00000000..0e38b6a0
--- /dev/null
+++ b/community/gsoc/project_ideas/libdiskfs_locking.mdwn
@@ -0,0 +1,47 @@
+[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation,
+Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
+
+[[!meta title="Fix libdiskfs Locking Issues"]]
+
+Every now and then, new locking issues are discovered in
+[[hurd/libdiskfs]] or [[hurd/translator/ext2fs]], for example.  Nowadays
+these in fact seem to be the most often encountered cause of Hurd crashes
+/ lockups.
+
+One of these could be traced
+recently, and turned out to be a lock inside [[hurd/libdiskfs]] that was taken
+and not released in some cases.  There is reason to believe that there are more
+faulty paths causing these lockups.
+
+The task is systematically checking the [[hurd/libdiskfs]] code for this kind of locking
+issues. To achieve this, some kind of test harness has to be implemented: For
+example instrumenting the code to check locking correctness constantly at
+runtime. Or implementing a [[unit testing]] framework that explicitly checks
+locking in various code paths. (The latter could serve as a template for
+implementing unit tests in other parts of the Hurd codebase...)
+
+(A [[systematic code review|security]] would probably suffice to find the
+existing locking
+issues; but it wouldn't document the work in terms of actual code produced, and
+thus it's not suitable for a GSoC project...)
+
+This task requires experience with debugging locking issues in
+[[multithreaded|multithreading]] applications.
+
+Tools have been written for automated [[code analysis]]; these can help to
+locate and fix such errors.
+
+Possible mentors: Samuel Thibault (youpi)
+
+Exercise: If you could actually track down and fix one of the existing locking
+errors before the end of the application process, that would be excellent. This
+might be rather tough though, so probably you need to talk to us about an
+alternative exercise task...
diff --git a/community/gsoc/project_ideas/procfs.mdwn b/community/gsoc/project_ideas/procfs.mdwn
new file mode 100644
index 00000000..d4760aae
--- /dev/null
+++ b/community/gsoc/project_ideas/procfs.mdwn
@@ -0,0 +1,45 @@
+[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
+is included in the section entitled
+[[GNU Free Documentation License|/fdl]]."]]"""]]
+
+[[!meta title="procfs"]]
+
+Although there is no standard (POSIX or other) for the layout of the `/proc`
+pseudo-filesystem, it turned out a very useful facility in GNU/Linux and other
+systems, and many tools concerned with process management use it. (`ps`, `top`,
+`htop`, `gtop`, `killall`, `pkill`, ...)
+
+Instead of porting all these tools to use [[hurd/libps]] (Hurd's official method for
+accessing process information), they could be made to run out of the box, by
+implementing a Linux-compatible `/proc` filesystem for the Hurd.
+
+The goal is to implement all `/proc` functionality needed for the various process
+management tools to work.  (On Linux, the `/proc` filesystem is used also for
+debugging purposes; but this is highly system-specific anyways, so there is
+probably no point in trying to duplicate this functionality as well...)
+
+The [[existing_partially_working_procfs_implementation|hurd/translator/procfs]]
+can serve as a starting point, but needs to be largely rewritten.  (It should
+use [[hurd/libnetfs]] rather than [[hurd/libtrivfs]]; the data format needs to
+change to be more Linux-compatible; and it needs adaptation to newer system
+interfaces.)
+
+This project requires learning [[hurd/translator]] programming, and
+understanding some of the internals of process management in the Hurd.  It
+should not be too hard coding-wise; and the task is very nicely defined by the
+existing Linux `/proc` interface -- no design considerations necessary.
+
+**Note**: We already have several applications for this task.
+
+Possible mentors: Olaf Buddenhagen (antrik)
+
+Exercise: Add or fix one piece in the existing procfs translator.
+
+*Status*: Madhusudan.C.S has implemented a new, fully functional [[procfs|madhusudancs]] for
+GSoC 2008. He is still working on some outstanding issues.
diff --git a/community/gsoc/project_ideas/valgrind.mdwn b/community/gsoc/project_ideas/valgrind.mdwn
new file mode 100644
index 00000000..79c8bd1d
--- /dev/null
+++ b/community/gsoc/project_ideas/valgrind.mdwn
@@ -0,0 +1,81 @@
+[[!meta copyright="Copyright © 2009, 2010, 2011 Free Software Foundation,
+Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
+
+[[!meta title="Porting Valgrind to the Hurd"]]
+
+[Valgrind](http://valgrind.org/) is an extremely useful debugging tool for memory errors.
+(And some other kinds of hard-to-find errors too.)
+Aside from being useful for program development in general,
+a Hurd port will help finding out why certain programs segfault on the Hurd,
+although they work on Linux.
+Even more importantly, it will help finding bugs in the Hurd servers themselfs.
+
+To keep track of memory use,
+Valgrind however needs to know how each [[system call]] affects the validity of memory regions.
+This knowledge is highly kernel-specific,
+and thus Valgrind needs to be explicitely ported for every system.
+
+Such a port involves two major steps:
+making Valgrind understand how kernel traps work in general on the system in question;
+and how all the individual kernel calls affect memory.
+The latter step is where most of the work is,
+as the behaviour of each single [[system call]] needs to be described.
+
+Compared to Linux,
+[[microkernel/Mach]] (the microkernel used by the Hurd) has very few kernel traps.
+Almost all [[system call]]s are implemented as [[RPC]]s instead --
+either handled by Mach itself, or by the various [[Hurd servers|hurd/translator]].
+All RPCs use a pair of `mach_msg` invocations:
+one to send a request message, and one to receive a reply.
+However, while all RPCs use the same `mach_msg` trap,
+the actual effect of the call varies greatly depending on which RPC is invoked --
+similar to the `ioctl` call on Linux.
+Each request thus must be handled individually.
+
+Unlike `ioctl`,
+the RPC invocations have explicit type information for the parameters though,
+which can be retrieved from the message header.
+By analyzing the parameters of the RPC reply message,
+Valgrind can know exactly which memory regions are affected by that call,
+even without specific knowledge of the RPC in question.
+Thus implementing a general parser for the reply messages
+will already give Valgrind a fairly good approximation of memory validity --
+without having to specify the exact semantic of each RPC by hand.
+
+While this should make Valgrind quite usable on the Hurd already, it's not perfect:
+some RPCs might return a buffer that is only partially filled with valid data;
+or some reply parameters might be optional,
+and only contain valid data under certain conditions.
+Such specific semantics can't be deduced from the message headers alone.
+Thus for a complete port,
+it will still be necessary to go through the list of all known RPCs,
+and implement special handling in Valgrind for those RPCs that need it.
+
+The goal of this task is at minimum to make Valgrind grok Mach traps,
+and to implement the generic RPC handler.
+Ideally, specific handling for RPCs needing it should also be implemented.
+
+Completing this project will require digging into Valgrind's handling of [[system call]]s,
+and into Hurd RPCs.
+It is not an easy task, but a fairly predictable one --
+there shouldn't be any unexpected difficulties,
+and no major design work is necessary.
+It doesn't require any specific previous knowledge:
+only good programming skills in general.
+On the other hand,
+the student will obtain a good understanding of Hurd RPCs while working on this task,
+and thus perfect qualifications for Hurd development in general :-)
+
+Possible mentors: Samuel Thibault (youpi)
+
+Exercise: As a starter,
+students can try to teach valgrind a couple of Linux ioctls,
+as this will make them learn how to use the read/write primitives of valgrind.
diff --git a/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn b/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn
index d55527a7..00d09094 100644
--- a/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn
+++ b/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn
@@ -56,7 +56,9 @@ For that I want to use:
 ### Tarfs
 
     apt-get install zip libz-dev libbz2-dev
-    hg clone http://bitbucket.org/ArneBab/hurd-tarfs tarfs
+    git clone git://git.sv.gnu.org/hurd/incubator.git tarfs
+    cd tarfs/
+    git checkout tarfs/master
     cd tarfs
     make
     make install
@@ -86,9 +88,12 @@ For that I want to use:
     
     cd ../..
     mkdir test
-    settrans -c test2 /usr/local/bin/nsmux test
-    tar -cf test/intro.tar repos/hurd_intro 
-    ls test2/intro.tar,,tarfs
+    touch test/hello
+    settrans -ca test2 /usr/local/bin/nsmux test
+    # tar -cvf test/intro.tar repos/hurd_intro 
+    cat test2/hello
+    cat test2/hello,,hello
+    # Hello, World!
 
 ### clisp