From bd54087dd78ef166297769fe2e8be34e1cc89c48 Mon Sep 17 00:00:00 2001 From: "http://arnebab.livejournal.com/" Date: Wed, 16 Mar 2011 09:53:27 +0000 Subject: Added link to Arch Hurd LiveCD --- hurd/running/live_cd.mdwn | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/hurd/running/live_cd.mdwn b/hurd/running/live_cd.mdwn index 42d85c5b..c9360594 100644 --- a/hurd/running/live_cd.mdwn +++ b/hurd/running/live_cd.mdwn @@ -1,4 +1,6 @@ -You can download a gzipped iso of a Hurd Live CD at +[[Arch Hurd|hurd/running/arch_hurd/]] offers Hurd LiveCDs at . + +Also you can download a gzipped iso of a Hurd Live CD at . The Superunpriveleged crew also offers a tiny Hurd Live CD that is under 10 -- cgit v1.2.3 From d6807eea355a50cedddb778d2a732cba8316128d Mon Sep 17 00:00:00 2001 From: "http://arnebab.livejournal.com/" Date: Wed, 16 Mar 2011 10:10:52 +0000 Subject: create qemu: Link to the d-i images --- hurd/running/qemu.mdwn | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/hurd/running/qemu.mdwn b/hurd/running/qemu.mdwn index c5b2cf1c..141ab2b1 100644 --- a/hurd/running/qemu.mdwn +++ b/hurd/running/qemu.mdwn @@ -27,6 +27,12 @@ Also you can use qemu to easily try one of our [[Hurd_LiveCDs|hurd/running/live_ # What is Needed to create a QEMU image +## Debian Installer + +Instructions for creating a qemu image from the install CDs from debian installer can be found in the README alongside the d-i Hurd images: + +## Old method + 1. First thing is to install [[/QEMU]]. 2. A [[grub]] boot disk for the floppy disk image needed for booting. The [0\.97 version](ftp://alpha.gnu.org/gnu/grub/grub-0.97-i386-pc.ext2fs) works fine. I downloaded it and renamed to `floppy.img`. Alternatively, the Debian grub-disk package (up till version 0.97-28) is fine as well. 3. You will need a [Debian/Hurd installation CD](http://www.debian.org/ports/hurd/hurd-cd). K16 works fine. -- cgit v1.2.3 From 01c30100822bdf5cf3312733b75b487212af6592 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide Date: Thu, 24 Mar 2011 20:07:16 +0100 Subject: tar cf → tar cvf MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn b/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn index d55527a7..006966e9 100644 --- a/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn +++ b/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn @@ -87,7 +87,7 @@ For that I want to use: cd ../.. mkdir test settrans -c test2 /usr/local/bin/nsmux test - tar -cf test/intro.tar repos/hurd_intro + tar -cvf test/intro.tar repos/hurd_intro ls test2/intro.tar,,tarfs ### clisp -- cgit v1.2.3 From d6f55e4eddd3ea30de9e4685059264cc4b3f21d8 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide Date: Thu, 24 Mar 2011 20:40:34 +0100 Subject: working nsmux test with the hello translator --- community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn b/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn index 006966e9..00d09094 100644 --- a/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn +++ b/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn @@ -56,7 +56,9 @@ For that I want to use: ### Tarfs apt-get install zip libz-dev libbz2-dev - hg clone http://bitbucket.org/ArneBab/hurd-tarfs tarfs + git clone git://git.sv.gnu.org/hurd/incubator.git tarfs + cd tarfs/ + git checkout tarfs/master cd tarfs make make install @@ -86,9 +88,12 @@ For that I want to use: cd ../.. mkdir test - settrans -c test2 /usr/local/bin/nsmux test - tar -cvf test/intro.tar repos/hurd_intro - ls test2/intro.tar,,tarfs + touch test/hello + settrans -ca test2 /usr/local/bin/nsmux test + # tar -cvf test/intro.tar repos/hurd_intro + cat test2/hello + cat test2/hello,,hello + # Hello, World! ### clisp -- cgit v1.2.3 From 2462ca1a8be02689b5e0c5a13038472ad29a7888 Mon Sep 17 00:00:00 2001 From: antrik Date: Wed, 16 Mar 2011 03:00:08 +0100 Subject: gsoc/project_ideas: Restore all project ideas here While there is certainly some overlap with other areas, it is *not* acceptable to drop mentors and exercises from GSoC tasks, nor to add random crap, nor do any other changes that make them less useful as GSoC tasks -- and this is *not* obvious if they do not live in the project_ideas namespace. It's also confusing in general. I tried to preserve all valid changes to the task descriptions themself -- though I might have messed up some things. I did leave the now redundant entries in open_tasks in place. Not sure how to deal with them. As the content is virtually identical anyways, they probably should be just turned into stubs pointing here. Or don't list them explicitely at all -- we point out in other places that GSoC ideas are useful in other contexts too... For the future, please refrain from reorganising things here without prior discussion. --- community/gsoc/project_ideas.mdwn | 6 +- .../gsoc/project_ideas/disk_io_performance.mdwn | 48 +++++++++++++ community/gsoc/project_ideas/dtrace.mdwn | 49 +++++++++++++ .../gsoc/project_ideas/libdiskfs_locking.mdwn | 47 +++++++++++++ community/gsoc/project_ideas/procfs.mdwn | 45 ++++++++++++ community/gsoc/project_ideas/valgrind.mdwn | 81 ++++++++++++++++++++++ 6 files changed, 273 insertions(+), 3 deletions(-) create mode 100644 community/gsoc/project_ideas/disk_io_performance.mdwn create mode 100644 community/gsoc/project_ideas/dtrace.mdwn create mode 100644 community/gsoc/project_ideas/libdiskfs_locking.mdwn create mode 100644 community/gsoc/project_ideas/procfs.mdwn create mode 100644 community/gsoc/project_ideas/valgrind.mdwn diff --git a/community/gsoc/project_ideas.mdwn b/community/gsoc/project_ideas.mdwn index da247847..030c18f3 100644 --- a/community/gsoc/project_ideas.mdwn +++ b/community/gsoc/project_ideas.mdwn @@ -89,10 +89,10 @@ Here are these pages' content: [[!inline pages="community/gsoc/project_ideas/server_overriding" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/tcp_ip_stack" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/nfs" show=0 feeds=no actions=yes]] -[[!inline pages="open_issues/locking" show=0 feeds=no actions=yes]] +[[!inline pages="community/gsoc/project_ideas/libdiskfs_locking" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/pthreads" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/sound" show=0 feeds=no actions=yes]] -[[!inline pages="open_issues/performance/io_system" show=0 feeds=no actions=yes]] +[[!inline pages="community/gsoc/project_ideas/disk_io_performance" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/vm_tuning" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/mtab" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/gnumach_cleanup" show=0 feeds=no actions=yes]] @@ -114,4 +114,4 @@ Here are these pages' content: [[!inline pages="open_issues/unit_testing" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/libcap" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/xattr" show=0 feeds=no actions=yes]] -[[!inline pages="open_issues/valgrind" show=0 feeds=no actions=yes]] +[[!inline pages="community/gsoc/project_ideas/valgrind" show=0 feeds=no actions=yes]] diff --git a/community/gsoc/project_ideas/disk_io_performance.mdwn b/community/gsoc/project_ideas/disk_io_performance.mdwn new file mode 100644 index 00000000..b6f223c8 --- /dev/null +++ b/community/gsoc/project_ideas/disk_io_performance.mdwn @@ -0,0 +1,48 @@ +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Disk I/O Performance Tuning"]] + +The most obvious reason for the Hurd feeling slow compared to mainstream +systems like GNU/Linux, is a low I/O system performance, in particular very +slow hard disk access. + +The reason for this slowness is lack and/or bad implementation of common +optimization techniques, like scheduling reads and writes to minimize head +movement; effective block caching; effective reads/writes to partial blocks; +[[reading/writing multiple blocks at once|clustered_page_faults]]; and +[[read-ahead]]. The +[[ext2_filesystem_server|hurd/translator/ext2fs]] might also need some +optimizations at a higher logical level. + +The goal of this project is to analyze the current situation, and implement/fix +various optimizations, to achieve significantly better disk performance. It +requires understanding the data flow through the various layers involved in +disk access on the Hurd ([[filesystem|hurd/virtual_file_system]], +[[pager|hurd/libpager]], driver), and general experience with +optimizing complex systems. That said, the killing feature we are definitely +missing is the [[read-ahead]], and even a very simple implementation would bring +very big performance speedups. + +Here are some real testcases: + + * [[binutils_ld_64ksec]]; + + * running the Git testsuite which is mostly I/O bound; + + * use [[TopGit]] on a non-toy repository. + + +Possible mentors: Samuel Thibault (youpi) + +Exercise: Look through all the code involved in disk I/O, and try something +easy to improve. It's quite likely though that you will find nothing obvious -- +in this case, please contact us about a different exercise task. diff --git a/community/gsoc/project_ideas/dtrace.mdwn b/community/gsoc/project_ideas/dtrace.mdwn new file mode 100644 index 00000000..252c303a --- /dev/null +++ b/community/gsoc/project_ideas/dtrace.mdwn @@ -0,0 +1,49 @@ +[[!meta copyright="Copyright © 2008, 2009, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="dtrace Support"]] + +One of the main problems of the current Hurd implementation is very poor +[[performance]]. While we have a bunch of ideas what could cause the performance +problems, these are mostly just guesses. Better understanding what really +causes bad performance is necessary to improve the situation. + +For that, we need tools for performance measurements. While all kinds of more +or less specific [[profiling]] tools could be conceived, the most promising and +generic approach seems to be a framework for logging certain events in the +running system (both in the microkernel and in the Hurd servers). This would +allow checking how much time is spent in certain modules, how often certain +situations occur, how things interact, etc. It could also prove helpful in +debugging some issues that are otherwise hard to find because of complex +interactions. + +The most popular framework for that is Sun's dtrace; but there might be others. +The student has to evaluate the existing options, deciding which makes most +sense for the Hurd; and implement that one. (Apple's implementation of dtrace +in their Mach-based kernel might be helpful here...) + +This project requires ability to evaluate possible solutions, and experience +with integrating existing components as well as low-level programming. + +Possible mentors: Samuel Thibault (youpi) + +Related: [[profiling]], [[LTTng]], [[SystemTap]] + +Exercise: In lack of a good exercise directly related to this task, just pick +one of the kernel-related or generally low-level tasks from the bug/task +trackers on savannah, and make a go at it. You might not be able to finish the +task in a limited amount of time, but you should at least be able to make a +detailed analysis of the issue. + +*Status*: Andei Barbu was working on +[SystemTap](http://csclub.uwaterloo.ca/~abarbu/hurd/) for GSoC 2008, but it +turned out too Linux-specific. He implemented kernel probes, but there is no +nice frontend yet. diff --git a/community/gsoc/project_ideas/libdiskfs_locking.mdwn b/community/gsoc/project_ideas/libdiskfs_locking.mdwn new file mode 100644 index 00000000..0e38b6a0 --- /dev/null +++ b/community/gsoc/project_ideas/libdiskfs_locking.mdwn @@ -0,0 +1,47 @@ +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Fix libdiskfs Locking Issues"]] + +Every now and then, new locking issues are discovered in +[[hurd/libdiskfs]] or [[hurd/translator/ext2fs]], for example. Nowadays +these in fact seem to be the most often encountered cause of Hurd crashes +/ lockups. + +One of these could be traced +recently, and turned out to be a lock inside [[hurd/libdiskfs]] that was taken +and not released in some cases. There is reason to believe that there are more +faulty paths causing these lockups. + +The task is systematically checking the [[hurd/libdiskfs]] code for this kind of locking +issues. To achieve this, some kind of test harness has to be implemented: For +example instrumenting the code to check locking correctness constantly at +runtime. Or implementing a [[unit testing]] framework that explicitly checks +locking in various code paths. (The latter could serve as a template for +implementing unit tests in other parts of the Hurd codebase...) + +(A [[systematic code review|security]] would probably suffice to find the +existing locking +issues; but it wouldn't document the work in terms of actual code produced, and +thus it's not suitable for a GSoC project...) + +This task requires experience with debugging locking issues in +[[multithreaded|multithreading]] applications. + +Tools have been written for automated [[code analysis]]; these can help to +locate and fix such errors. + +Possible mentors: Samuel Thibault (youpi) + +Exercise: If you could actually track down and fix one of the existing locking +errors before the end of the application process, that would be excellent. This +might be rather tough though, so probably you need to talk to us about an +alternative exercise task... diff --git a/community/gsoc/project_ideas/procfs.mdwn b/community/gsoc/project_ideas/procfs.mdwn new file mode 100644 index 00000000..d4760aae --- /dev/null +++ b/community/gsoc/project_ideas/procfs.mdwn @@ -0,0 +1,45 @@ +[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled +[[GNU Free Documentation License|/fdl]]."]]"""]] + +[[!meta title="procfs"]] + +Although there is no standard (POSIX or other) for the layout of the `/proc` +pseudo-filesystem, it turned out a very useful facility in GNU/Linux and other +systems, and many tools concerned with process management use it. (`ps`, `top`, +`htop`, `gtop`, `killall`, `pkill`, ...) + +Instead of porting all these tools to use [[hurd/libps]] (Hurd's official method for +accessing process information), they could be made to run out of the box, by +implementing a Linux-compatible `/proc` filesystem for the Hurd. + +The goal is to implement all `/proc` functionality needed for the various process +management tools to work. (On Linux, the `/proc` filesystem is used also for +debugging purposes; but this is highly system-specific anyways, so there is +probably no point in trying to duplicate this functionality as well...) + +The [[existing_partially_working_procfs_implementation|hurd/translator/procfs]] +can serve as a starting point, but needs to be largely rewritten. (It should +use [[hurd/libnetfs]] rather than [[hurd/libtrivfs]]; the data format needs to +change to be more Linux-compatible; and it needs adaptation to newer system +interfaces.) + +This project requires learning [[hurd/translator]] programming, and +understanding some of the internals of process management in the Hurd. It +should not be too hard coding-wise; and the task is very nicely defined by the +existing Linux `/proc` interface -- no design considerations necessary. + +**Note**: We already have several applications for this task. + +Possible mentors: Olaf Buddenhagen (antrik) + +Exercise: Add or fix one piece in the existing procfs translator. + +*Status*: Madhusudan.C.S has implemented a new, fully functional [[procfs|madhusudancs]] for +GSoC 2008. He is still working on some outstanding issues. diff --git a/community/gsoc/project_ideas/valgrind.mdwn b/community/gsoc/project_ideas/valgrind.mdwn new file mode 100644 index 00000000..79c8bd1d --- /dev/null +++ b/community/gsoc/project_ideas/valgrind.mdwn @@ -0,0 +1,81 @@ +[[!meta copyright="Copyright © 2009, 2010, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Porting Valgrind to the Hurd"]] + +[Valgrind](http://valgrind.org/) is an extremely useful debugging tool for memory errors. +(And some other kinds of hard-to-find errors too.) +Aside from being useful for program development in general, +a Hurd port will help finding out why certain programs segfault on the Hurd, +although they work on Linux. +Even more importantly, it will help finding bugs in the Hurd servers themselfs. + +To keep track of memory use, +Valgrind however needs to know how each [[system call]] affects the validity of memory regions. +This knowledge is highly kernel-specific, +and thus Valgrind needs to be explicitely ported for every system. + +Such a port involves two major steps: +making Valgrind understand how kernel traps work in general on the system in question; +and how all the individual kernel calls affect memory. +The latter step is where most of the work is, +as the behaviour of each single [[system call]] needs to be described. + +Compared to Linux, +[[microkernel/Mach]] (the microkernel used by the Hurd) has very few kernel traps. +Almost all [[system call]]s are implemented as [[RPC]]s instead -- +either handled by Mach itself, or by the various [[Hurd servers|hurd/translator]]. +All RPCs use a pair of `mach_msg` invocations: +one to send a request message, and one to receive a reply. +However, while all RPCs use the same `mach_msg` trap, +the actual effect of the call varies greatly depending on which RPC is invoked -- +similar to the `ioctl` call on Linux. +Each request thus must be handled individually. + +Unlike `ioctl`, +the RPC invocations have explicit type information for the parameters though, +which can be retrieved from the message header. +By analyzing the parameters of the RPC reply message, +Valgrind can know exactly which memory regions are affected by that call, +even without specific knowledge of the RPC in question. +Thus implementing a general parser for the reply messages +will already give Valgrind a fairly good approximation of memory validity -- +without having to specify the exact semantic of each RPC by hand. + +While this should make Valgrind quite usable on the Hurd already, it's not perfect: +some RPCs might return a buffer that is only partially filled with valid data; +or some reply parameters might be optional, +and only contain valid data under certain conditions. +Such specific semantics can't be deduced from the message headers alone. +Thus for a complete port, +it will still be necessary to go through the list of all known RPCs, +and implement special handling in Valgrind for those RPCs that need it. + +The goal of this task is at minimum to make Valgrind grok Mach traps, +and to implement the generic RPC handler. +Ideally, specific handling for RPCs needing it should also be implemented. + +Completing this project will require digging into Valgrind's handling of [[system call]]s, +and into Hurd RPCs. +It is not an easy task, but a fairly predictable one -- +there shouldn't be any unexpected difficulties, +and no major design work is necessary. +It doesn't require any specific previous knowledge: +only good programming skills in general. +On the other hand, +the student will obtain a good understanding of Hurd RPCs while working on this task, +and thus perfect qualifications for Hurd development in general :-) + +Possible mentors: Samuel Thibault (youpi) + +Exercise: As a starter, +students can try to teach valgrind a couple of Linux ioctls, +as this will make them learn how to use the read/write primitives of valgrind. -- cgit v1.2.3