diff options
62 files changed, 1424 insertions, 305 deletions
diff --git a/.templates/autotag.tmpl b/.templates/autotag.tmpl index 87b76eef..1a963267 100644 --- a/.templates/autotag.tmpl +++ b/.templates/autotag.tmpl @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this diff --git a/community/gsoc.mdwn b/community/gsoc.mdwn index 2ce92d19..45d3a31d 100644 --- a/community/gsoc.mdwn +++ b/community/gsoc.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -6,49 +6,39 @@ id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!meta title="Google Summer of Code"]] -# 2010 +# 2011 -This year we are again participating in [Google Summer of Code](http://socghop.appspot.com) -under the [GNU umbrella](http://socghop.appspot.com/gsoc/org/show/google/gsoc2010/gnuproject). +We're in! The GNU Hurd project will again participate in the [Google Summer of +Code](http://socghop.appspot.com) under the [GNU +umbrella](http://www.gnu.org/software/soc-projects/). - * *[[Jeremie Koenig|jkoenig]]*, mentored by *[[Samuel - Thibault|samuelthibault]]*, is working on adapting the Debian Installer to - [produce working Debian GNU/Hurd installation - images](http://socghop.appspot.com/gsoc/student_project/show/google/gsoc2010/debian/t127230758239) - so we can easily offer up to date disc-sets. - ([Details](http://wiki.debian.org/SummerOfCode2010/HurdDebianInstaller/JeremieKoenig).) +On Monday, 2011-03-28 the *student application period* will +[begin](http://socghop.appspot.com/document/show/gsoc_program/google/gsoc2011/timeline), +the deadline is on Friday, 2011-04-08. That is plenty of time for discussing +your applications -- but please don't wait to the last minute! - * *[[Emilio Pozuelo Monfort|pochu]]*, mentored by *Carl Fredrik Hammar* (who - was a GSoC student in 2007), is working on a task that may be perceived as - less exciting from the outside, but yet is extremely valuable: [fixing - compatibility problems exposed by projects' - testsuites](http://socghop.appspot.com/gsoc/student_project/show/google/gsoc2010/gnuproject/t127230759396). - ([[Details|community/gsoc/project_ideas/testsuites]].) - * *[[Karim Allah Ahmed|kam]]*, mentored by *Sergio López*, is working on - [tuning the VM Subsystem in - GNU/Hurd](http://socghop.appspot.com/gsoc/student_project/show/google/gsoc2010/gnuproject/t127230759587) - to bring the virtual memory management in Hurd/Mach up to date. - ([[Details|community/gsoc/project_ideas/vm_tuning]].) +## Applying for a Task We always ask students that want to apply for a task (in the course of the Google Summer of Code) to mind our distinct [[student_application_form]]. -Apart from our own [[project_ideas]], there are also two [Hurd-related ideas -for X.Org](http://wiki.x.org/wiki/Hurd_Porting). +We have a list of [[project_ideas]], and students are likewise encouraged to +submit their own project proposals. -As the Google Summer of Code 2010 has already started, you can no longer -participate as a GSoC student. However, -working on one of these projects is still a good opportunity to get started with Hurd development. Please read up about [[contributing]] in general; and feel free to ask any questions you might have at one of our [[regular_IRC_meetings|IRC#regular_meetings]]. Generally it's a good idea to [[contact_us|communication]] when starting to work on some project. +Working on one of these projects is generally a good opportunity to get started +with Hurd development, even outside of the GSoC context. + + # History In 2006 and [[2007]], we participated in GSoC under the umbrella of the GNU @@ -59,5 +49,5 @@ In the following year, we successfully participated on our own, not only as a suborganization of the GNU project. Read about our five students' success on the [[2008]] page. -The next year, we participated under the GNU umbrella with one slot again. -Read about it on the [[2009]] page. +The next two year, we participated under the GNU umbrella with one slot in +[[2009]], and three in [[2010]]. diff --git a/community/gsoc/2010.mdwn b/community/gsoc/2010.mdwn new file mode 100644 index 00000000..4388636b --- /dev/null +++ b/community/gsoc/2010.mdwn @@ -0,0 +1,35 @@ +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Google Summer of Code 2010"]] + +In 2010 we have again been participating in the Google Summer of Code +under the GNU umbrella, with three slots: + + * *[[Jeremie Koenig|jkoenig]]*, mentored by *[[Samuel + Thibault|samuelthibault]]*, has been working on adapting the Debian Installer to + [produce working Debian GNU/Hurd installation + images](http://socghop.appspot.com/gsoc/student_project/show/google/gsoc2010/debian/t127230758239) + so we can easily offer up to date disc-sets. + ([Details](http://wiki.debian.org/SummerOfCode2010/HurdDebianInstaller/JeremieKoenig).) + + * *[[Emilio Pozuelo Monfort|pochu]]*, mentored by *Carl Fredrik Hammar* (who + was a GSoC student in [[2007]]), has been working on a task that may be perceived as + less exciting from the outside, but yet is extremely valuable: [fixing + compatibility problems exposed by projects' + testsuites](http://socghop.appspot.com/gsoc/student_project/show/google/gsoc2010/gnuproject/t127230759396). + ([[Details|community/gsoc/project_ideas/testsuites]].) + + * *[[Karim Allah Ahmed|kam]]*, mentored by *Sergio López*, has been working on + [tuning the VM Subsystem in + GNU/Hurd](http://socghop.appspot.com/gsoc/student_project/show/google/gsoc2010/gnuproject/t127230759587) + to bring the virtual memory management in Hurd/Mach up to date. + ([[Details|community/gsoc/project_ideas/vm_tuning]].) diff --git a/community/gsoc/project_ideas.mdwn b/community/gsoc/project_ideas.mdwn index 649e05c1..4f028e4f 100644 --- a/community/gsoc/project_ideas.mdwn +++ b/community/gsoc/project_ideas.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -76,16 +76,20 @@ will assist you as well as we can. See also the list of [Hurd-related X.org project ideas](http://wiki.x.org/wiki/Hurd_Porting). +<!-- These X.org tasks are also listed on +<http://xorg.freedesktop.org/wiki/SummerOfCodeIdeas>, which is the page used by +X.org for referring to their GSoC projects. Probabaly we should get rid of one +of these pages (to avoid duplication). --> + [[!inline pages="community/gsoc/project_ideas/language_bindings" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/virtualization" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/file_locking" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/server_overriding" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/tcp_ip_stack" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/nfs" show=0 feeds=no actions=yes]] -[[!inline pages="open_issues/locking" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/pthreads" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/sound" show=0 feeds=no actions=yes]] -[[!inline pages="open_issues/performance/io_system" show=0 feeds=no actions=yes]] +[[!inline pages="community/gsoc/project_ideas/disk_io_performance" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/vm_tuning" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/mtab" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/gnumach_cleanup" show=0 feeds=no actions=yes]] @@ -95,7 +99,6 @@ See also the list of [Hurd-related X.org project ideas](http://wiki.x.org/wiki/H [[!inline pages="community/gsoc/project_ideas/lexical_dot-dot" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/secure_chroot" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/package_manager" show=0 feeds=no actions=yes]] -[[!inline pages="community/gsoc/project_ideas/debian_installer" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/download_backends" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/libgtop" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/maxpath" show=0 feeds=no actions=yes]] @@ -104,6 +107,9 @@ See also the list of [Hurd-related X.org project ideas](http://wiki.x.org/wiki/H [[!inline pages="community/gsoc/project_ideas/cdparanoia" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/perl_python" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/testsuites" show=0 feeds=no actions=yes]] +[[!inline pages="community/gsoc/project_ideas/testing_framework" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/libcap" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/xattr" show=0 feeds=no actions=yes]] -[[!inline pages="open_issues/valgrind" show=0 feeds=no actions=yes]] +[[!inline pages="community/gsoc/project_ideas/valgrind" show=0 feeds=no actions=yes]] +[[!inline pages="community/gsoc/project_ideas/driver_glue_code" show=0 feeds=no actions=yes]] +[[!inline pages="community/gsoc/project_ideas/dtrace" show=0 feeds=no actions=yes]] diff --git a/open_issues/performance/io_system.mdwn b/community/gsoc/project_ideas/disk_io_performance.mdwn index 4af093ba..ae634709 100644 --- a/open_issues/performance/io_system.mdwn +++ b/community/gsoc/project_ideas/disk_io_performance.mdwn @@ -9,7 +9,7 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -[[!meta title="I/O System"]] +[[!meta title="Disk I/O Performance Tuning"]] [[!tag open_issue_hurd]] @@ -20,8 +20,8 @@ slow hard disk access. The reason for this slowness is lack and/or bad implementation of common optimization techniques, like scheduling reads and writes to minimize head movement; effective block caching; effective reads/writes to partial blocks; -[[reading/writing multiple blocks at once|clustered_page_faults]]; and -[[read-ahead]]. The +[[reading/writing multiple blocks at once|open_issues/performance/io_system/clustered_page_faults]]; and +[[open_issues/performance/io_system/read-ahead]]. The [[ext2_filesystem_server|hurd/translator/ext2fs]] might also need some optimizations at a higher logical level. @@ -31,12 +31,12 @@ requires understanding the data flow through the various layers involved in disk access on the Hurd ([[filesystem|hurd/virtual_file_system]], [[pager|hurd/libpager]], driver), and general experience with optimizing complex systems. That said, the killing feature we are definitely -missing is the [[read-ahead]], and even a very simple implementation would bring +missing is the [[open_issues/performance/io_system/read-ahead]], and even a very simple implementation would bring very big performance speedups. Here are some real testcases: - * [[binutils_ld_64ksec]]; + * [[open_issues/performance/io_system/binutils_ld_64ksec]]; * running the Git testsuite which is mostly I/O bound; diff --git a/community/gsoc/project_ideas/driver_glue_code.mdwn b/community/gsoc/project_ideas/driver_glue_code.mdwn index 9c063e9f..65ea4f0f 100644 --- a/community/gsoc/project_ideas/driver_glue_code.mdwn +++ b/community/gsoc/project_ideas/driver_glue_code.mdwn @@ -1,5 +1,4 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -9,41 +8,64 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -[[!meta title="New Driver Glue Code"]] +[[!meta title="New Driver Framework"]] -Although a driver framework in user space would be desirable, presently the Hurd -uses kernel drivers in the microkernel, -[[GNU_Mach|microkernel/mach/gnumach]]. (And changing this would be far beyond a -GSoC project...) - -The problem is that the drivers in GNU Mach are presently old Linux drivers -(mostly from 2.0.x) accessed through a glue code layer. This is not an ideal -solution, but works quite OK, except that the drivers are very old. The goal of -this project is to redo the glue code, so we can use drivers from current Linux -versions, or from one of the free BSD variants. - -While it would be certainly possible to create custom glue code again, a more -sustainable and probably also easier approach is to use -[[DDE]] instead -- it already -does the hard work of providing an environment where the foreign drivers can -run, and has the additional advantage of being externally maintained. - -This is a doable, but pretty involved project. Previous experience with driver -programming probably is a must. (No Hurd-specific knowledge is required, -though.) +The Hurd presently uses hardware drivers +implemented in the microkernel, [[GNU_Mach|microkernel/mach/gnumach]]. +These drivers are old Linux drivers (mostly from 2.0.x) +accessed through a glue code layer. +This is not an ideal solution, but works quite OK, +except that the drivers are extremely old by now. +Thus we need a new framework, +so we can use drivers from current Linux versions instead, +or perhaps from one of the free BSD variants. This is [[!GNU_Savannah_task 5488]]. [[open issues/user-space device drivers]]. [[open issues/device drivers and io systems]]. +The most promising approach for getting newer drivers seems to be [[DDE]]: +it already does the hard work of providing an environment +where the foreign drivers can run, +and offers the additional benefit of being externally maintained. +DDE also offers the necessary facilities +for running all drivers in separate userspace processes, +which is more desirable than drivers running in the microkernel. -Possible mentors: Samuel Thibault (youpi) +[[Zheng Da|zhengda]] has already done considerable work on this. +The basic framework for using DDE in the Hurd is present, +and network card drivers are already working very well. +However, this work isn't fully integrated in the Hurd yet. +The additional kernel interfaces that were created for this +are still prototypes, and will need to be reworked. +Also, there is no build system for automatically compiling +all Linux network card drivers in one go. + +Other types of drivers are missing so far. +Support for IDE drivers has been partially implemented, +but isn't fully working yet. +To fully replace the old in-kernel drivers, +further infrastructure will be necessary +to make userspace disk drivers usable for the root filesystem. -Exercise: Take a driver for some newer piece of hardware (e.g. Intel e1000 -ethernet) from a recent system, and try to port it to run in the existing -driver framework in GNU Mach. Completing the port might be too involved for the -exercise; but it's pretty likely that you will find something else to improve -in the glue code while working on this... +Some other subsystems are missing or incomplete in DDE itself, +and will require additional work that is not specific to the Hurd implementation. + +The goal of this task is to fix at least one of the mentioned major shortcomings: +rework the kernel interfaces; +provide a streamlined build system for the drivers; +finish IDE support; +or implement support for some other subsystem. +<!-- should probably provide separate task descriptions for each... --> + +This is a doable, but pretty involved project. +Previous experience with driver programming probably is a must. +To be able to work on the framework, +the student will also have to get a good understanding of certain aspects of Hurd, +such as memory management for example. + +Possible mentors: Samuel Thibault (youpi) -*Status*: [[Zheng Da|zhengda]] is working on DDE, and has mostly completed the -initial port. +Exercise: Get one of the not yet integrated Linux network card drivers to work. +(Note: This should be straightforward, +once you have the framework properly built and set up...) diff --git a/community/gsoc/project_ideas/dtrace.mdwn b/community/gsoc/project_ideas/dtrace.mdwn new file mode 100644 index 00000000..6261c03e --- /dev/null +++ b/community/gsoc/project_ideas/dtrace.mdwn @@ -0,0 +1,62 @@ +[[!meta copyright="Copyright © 2008, 2009, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Kernel Instrumentation"]] + +[[!tag open_issue_gnumach]] + +One of the main problems of the current Hurd implementation is very poor +[[open_issues/performance]]. While we have a bunch of ideas what could cause the performance +problems, these are mostly just guesses. Better understanding what really +causes bad performance is necessary to improve the situation. + +For that, we need tools for performance measurements. While all kinds of more +or less specific [[open_issues/profiling]] tools could be conceived, the most promising and +generic approach seems to be a framework for logging certain events in the +running system (both in the microkernel and in the Hurd servers). This would +allow checking how much time is spent in certain modules, how often certain +situations occur, how things interact, etc. It could also prove helpful in +debugging some issues that are otherwise hard to find because of complex +interactions. + +The most popular kernel instrumentation framework is Sun's dtrace, +originally written for Solaris, +but also adopted by some other systems. +However, the GPL-incompatible license means it can't be used in Linux, +and thus Linux developers created their own frameworks instead: +first [[SystemTap]], and now [[LTTng]]. + +In 2008, Andrei Barbu did initial work on kernel probes for the Hurd. +However, not all of his patches got merged, +because some turned out not to be fully functional. +Also, he didn't get around to work on userspace probes, +nor on a nice frontend for writing test scripts employing the probes. + +The goal of this project is to make the instrumentation framework +more usable and complete, +and to better integrate it in the Hurd. +For that, the student will have to work +on some real profiling and/or debugging tasks, +and fix any shortcomings he encounters in the framework. + +This is a pretty involved task. +Previous experience with low-level programming is a must; +and it also requires a good grasp on interactions in complex systems. + +To work on this project, +the student will have to get familiar with GNU Mach. +(The microkernel employed by the Hurd.) +Some understanding of other aspects of the Hurd will also be required, +depending on the exact nature of the profiling/debugging performed. + +Possible mentors: Samuel Thibault (youpi) + +Exercise: Use the existing probes to perform some simple measurement. diff --git a/open_issues/locking.mdwn b/community/gsoc/project_ideas/libdiskfs_locking.mdwn index 11a10524..faac8bd9 100644 --- a/open_issues/locking.mdwn +++ b/community/gsoc/project_ideas/libdiskfs_locking.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -9,14 +9,12 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -[[!tag open_issue_hurd]] +[[!meta title="Fix libdiskfs Locking Issues"]] -Every now and then, new locking issues are discovered in -[[hurd/libdiskfs]] or [[hurd/translator/ext2fs]], for example. Nowadays -these in fact seem to be the most often encountered cause of Hurd crashes -/ lockups. +[[!tag open_issue_hurd]] -One of these could be traced +Nowadays the most often encountered cause of Hurd crashes seems to be lockups +in the [[hurd/translator/ext2fs]] server. One of these could be traced recently, and turned out to be a lock inside [[hurd/libdiskfs]] that was taken and not released in some cases. There is reason to believe that there are more faulty paths causing these lockups. @@ -24,17 +22,24 @@ faulty paths causing these lockups. The task is systematically checking the [[hurd/libdiskfs]] code for this kind of locking issues. To achieve this, some kind of test harness has to be implemented: For example instrumenting the code to check locking correctness constantly at -runtime. Or implementing a [[unit testing]] framework that explicitly checks +runtime. Or implementing a [[open_issues/unit_testing]] framework that explicitly checks locking in various code paths. (The latter could serve as a template for implementing unit tests in other parts of the Hurd codebase...) -(A [[systematic code review|security]] would probably suffice to find the +(A [[systematic code review|open_issues/security]] would probably suffice to find the existing locking issues; but it wouldn't document the work in terms of actual code produced, and thus it's not suitable for a GSoC project...) This task requires experience with debugging locking issues in -[[multithreaded|multithreading]] applications. +[[multithreaded|open_issues/multithreading]] applications. -Tools have been written for automated [[code analysis]]; these can help to +Tools have been written for automated [[open_issues/code_analysis]]; these can help to locate and fix such errors. + +Possible mentors: Samuel Thibault (youpi) + +Exercise: If you could actually track down and fix one of the existing locking +errors before the end of the application process, that would be excellent. This +might be rather tough though, so probably you need to talk to us about an +alternative exercise task... diff --git a/community/gsoc/project_ideas/procfs.mdwn b/community/gsoc/project_ideas/procfs.mdwn new file mode 100644 index 00000000..b4d1dc5f --- /dev/null +++ b/community/gsoc/project_ideas/procfs.mdwn @@ -0,0 +1,51 @@ +[[!meta copyright="Copyright © 2008, 2009, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled +[[GNU Free Documentation License|/fdl]]."]]"""]] + +[[!meta title="procfs"]] + +Although there is no standard (POSIX or other) for the layout of the `/proc` +pseudo-filesystem, it turned out a very useful facility in GNU/Linux and other +systems, and many tools concerned with process management use it. (`ps`, `top`, +`htop`, `gtop`, `killall`, `pkill`, ...) + +Instead of porting all these tools to use [[hurd/libps]] (Hurd's official method for +accessing process information), they could be made to run out of the box, by +implementing a Linux-compatible `/proc` filesystem for the Hurd. + +The goal is to implement all `/proc` functionality needed for the various process +management tools to work. (On Linux, the `/proc` filesystem is used also for +debugging purposes; but this is highly system-specific anyways, so there is +probably no point in trying to duplicate this functionality as well...) + +The [[existing_partially_working_procfs_implementation|hurd/translator/procfs]] +can serve as a starting point, but needs to be largely rewritten. (It should +use [[hurd/libnetfs]] rather than [[hurd/libtrivfs]]; the data format needs to +change to be more Linux-compatible; and it needs adaptation to newer system +interfaces.) + +This project requires learning [[hurd/translator]] programming, and +understanding some of the internals of process management in the Hurd. It +should not be too hard coding-wise; and the task is very nicely defined by the +existing Linux `/proc` interface -- no design considerations necessary. + +**Note**: We already have several applications for this task. + +Possible mentors: Olaf Buddenhagen (antrik) + +Exercise: Add or fix one piece in the existing procfs translator. + +*Status*: Madhusudan.C.S has implemented a new, fully functional [[procfs|madhusudancs]] for +GSoC 2008. He is still working on some outstanding issues. + +--- + +Note that [[jkoenig's `procfs` re-write|hurd/translator/procfs/jkoenig]] should +address all these issues already. diff --git a/community/gsoc/project_ideas/secure_chroot.mdwn b/community/gsoc/project_ideas/secure_chroot.mdwn index 57739861..bfaf330b 100644 --- a/community/gsoc/project_ideas/secure_chroot.mdwn +++ b/community/gsoc/project_ideas/secure_chroot.mdwn @@ -12,7 +12,7 @@ License|/fdl]]."]]"""]] [[!meta title="Secure chroot Implementation"]] As the Hurd attempts to be (almost) fully [[UNIX]]-compatible, it also implements a -`chroot` [[system call]]. However, the current implementation is not really +`chroot()` [[system call]]. However, the current implementation is not really good, as it allows easily escaping the `chroot`, for example by use of [[passive_translators|hurd/translator]]. diff --git a/community/gsoc/project_ideas/testing_framework.mdwn b/community/gsoc/project_ideas/testing_framework.mdwn new file mode 100644 index 00000000..ff9899d9 --- /dev/null +++ b/community/gsoc/project_ideas/testing_framework.mdwn @@ -0,0 +1,55 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled +[[GNU Free Documentation License|/fdl]]."]]"""]] + +[[!meta title="Automated Testing Framework"]] + +Hurd development would benefit greatly from automated tests. +Unit tests should be added for all the major components +(Mach; Hurd servers and libraries). +Also, functional tests can be performed on various levels: +Mach; individual servers; and interaction of several servers. + +(The highest level would actually be testing libc functionality, +which in turn uses the various Hurd mechanisms. +glibc already comes with a test suite -- +though it might be desirabe to add some extra tests +for exercising specific aspects of the Hurd...) + +Our page on [[automated testing|open_issues/unit_testing]] collects some relevant material. + +The Goal of this task is to provide testing frameworks +that allow automatically running tests +as part of the Hurd and Mach build processes. +The student will have to create the necessary infrastrucure, +and a couple of sample tests employing it. +Ideally, all the aspects mentioned above should be covered. +At least some have to be ready for use and upstream merging +before the end of the summer. + +(As a bonus, +in addition to these explicit tests, +it would be helpful to integrate some methods +for testing [[locking validity|libdiskfs_locking]], +performing static code analysis etc.) + +This task probably requires some previous experience +with unit testing of C programs, +as well as dealing with complex build systems. +No in-depth knowledge about any specific parts of the Hurd should be necessary, +but some general understanding of the Hurd architecture +will have to be aquired to complete this project. +This makes it a very good project +to get started on Hurd development :-) + +Possible mentors: ? + +Exercise: Create a program performing some simple test(s) on the Hurd or Mach code. +It doesn't need to be integrated in the build process yet -- +a standalone progrem with it's own Makefile is fine for now. diff --git a/community/gsoc/project_ideas/testing_framework/discussion.mdwn b/community/gsoc/project_ideas/testing_framework/discussion.mdwn new file mode 100644 index 00000000..872d0eb7 --- /dev/null +++ b/community/gsoc/project_ideas/testing_framework/discussion.mdwn @@ -0,0 +1,270 @@ +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +freenode, #hurd channel, 2011-03-05: + + <nixness> what about testing though? + <nixness> like sort of "what's missing? lets write tests for it so that + when someone gets to implementing it, he knows what to do. Repeat" + project + <antrik> you mean creating an automated testing framework? + <antrik> this is actually a task I want to add for this year, if I get + around to it :-) + <nixness> yeah I'd very much want to apply for that one + <nixness> cuz I've never done Kernel hacking before, but I know that with + the right tasks like "test VM functionality", I would be able to write up + the automated tests and hopefully learn more about what breaks/makes the + kernel + <nixness> (and it would make implementing the feature much less hand-wavy + and more correct) + <nixness> antrik, I believe the framework(CUnit right?) exists, but we just + need to write the tests. + <antrik> do you have prior experience implementing automated tests? + <nixness> lots of tests! + <nixness> yes, in Java mostly, but I've played around with CUnit + <antrik> ah, great :-) + <nixness> here's what I know from experience: 1) write basic tests. 2) + write ones that test multiple features 3) stress test [option 4) + benchmark and record to DB] + <youpi> well, what we'd rather need is to fix the issues we already know + from the existing testsuites :) + +[[GSoC project propsal|community/gsoc/project_ideas/testsuites]]. + + <nixness> youpi, true, and I think I should check what's available in way + of tests, but if the tests are "all or nothing" then it becomes really + hard to fix your mistakes + <youpi> they're not all or nothing + <antrik> youpi: the existing testsuites don't test specific system features + <youpi> libc ones do + <youpi> we could also check posixtestsuite which does too + +[[open_issues/open_posix_test_suite]]. + + <antrik> AFAIK libc has very few failing tests + +[[open_issues/glibc_testsuite]]. + + <youpi> err, like twenty? + <youpi> € grep -v '^#' expected-results-i486-gnu-libc | wc -l + <youpi> 67 + <youpi> nope, even more + <antrik> oh, sorry, I confused it with coreutils + <pinotree> plus the binutils ones, i guess + <youpi> yes + +[[open_issues/binutils#weak]]. + + <antrik> anyways, I don't think relying on external testsuites for + regression testing is a good plan + <antrik> also, that doesn't cover unit testing at all + <youpi> why ? + <youpi> sure there can be unit testing at the translator etc. level + <antrik> if we want to implement test-driven development, and/or do serious + refactoring without too much risk, we need a test harness where we can + add specific tests as needed + <youpi> but more than often, the issues are at the libc / etc. level + because of a combination o fthings at the translator level, which unit + testing won't find out + * nixness yewzzz! + <nixness> sure unit testing can find them out. if they're good "unit" tests + <youpi> the problem is that you don't necessarily know what "good" means + <youpi> e.g. for posix correctness + <youpi> since it's not posix + <nixness> but if they're composite clever tests, then you lose that + granularity + <nixness> youpi, is that a blackbox test intended to be run at the very end + to check if you're posix compliant? + <antrik> also, if we have our own test harness, we can run tests + automatically as part of the build process, which is a great plus IMHO + <youpi> nixness: "that" = ? + <nixness> oh nvm, I thought there was a test stuie called "posix + correctness" + <youpi> there's the posixtestsuite yes + <youpi> it's an external one however + <youpi> antrik: being external doesn't mean we can't invoke it + automatically as part of the build process when it's available + <nixness> youpi, but being internal means it can test the inner workings of + certain modules that you are unsure of, and not just the interface + <youpi> sure, that's why I said it'd be useful too + <youpi> but as I said too, most bugs I've seen were not easy to find out at + the unit level + <youpi> but rather at the libc level + <antrik> of course we can integrate external tests if they exist and are + suitable. but that that doesn't preclude adding our own ones too. in + either case, that integration work has to be done too + <youpi> again, I've never said I was against internal testsuite + <antrik> also, the major purpose of a test suite is checking known + behaviour. a low-level test won't directly point to a POSIX violation; + but it will catch any changes in behaviour that would lead to one + <youpi> what I said is that it will be hard to write them tight enough to + find bugs + <youpi> again, the problem is knowing what will lead to a POSIX violation + <youpi> it's long work + <youpi> while libc / posixtestsuite / etc. already do that + <antrik> *any* unexpected change in behaviour is likely to cause bugs + somewher + <youpi> but WHAT is "expected" ? + <youpi> THAT is the issue + <youpi> and libc/posixtessuite do know that + <youpi> at the translator level we don't really + <youpi> see the recent post about link() + +[link(dir,name) should fail with +EPERM](http://lists.gnu.org/archive/html/bug-hurd/2011-03/msg00007.html) + + <youpi> in my memory jkoenig pointed it out for a lot of such calls + <youpi> and that issue is clearly not known at the translator level + <nixness> so you're saying that the tests have to be really really + low-level, and work our way up? + <youpi> I'm saying that the translator level tests will be difficult to + write + <antrik> why isn't it known at the translator level? if it's a translator + (not libc) bug, than obviously the translator must return something wrong + at some point, and that's something we can check + <youpi> because you'll have to know all the details of the combinations + used in libc, to know whether they'll lead to posix issues + <youpi> antrik: sure, but how did we detect that that was unexpected + behavior? + <youpi> because of a glib test + <youpi> at the translator level we didn't know it was an unexpected + behavior + <antrik> gnulib actually + <youpi> and if you had asked me, I wouldn't have known + <antrik> again, we do *not* write a test suite to find existing bugs + <youpi> right, took one for the other + <youpi> doesn't really matter actually + <youpi> antrik: ok, I don't care then + <antrik> we write a test suite to prevent future bugs, or track status of + known bugs + <youpi> (don't care *enough* for now, I mean) + <nixness> hmm, so write something up antrik for GSoC :) and I'll be sure to + apply + <antrik> now that we know some translators return a wrong error status in a + particular situation, we can write a test that checks exactly this error + status. that way we know when it is fixed, and also when it breaks again + <antrik> nixness: great :-) + <nixness> sweet. that kind of thing would also need a db backend + <antrik> nixness: BTW, if you have a good idea, you can send an application + for it even if it's not listed among the proposed tasks... + <antrik> so you don't strictly need a writeup from me to apply for this :-) + <nixness> antrik, I'll keep that in mind, but I'll also be checking your + draft page + <nixness> oh cool :) + <antrik> (and it's a well known fact that those projects which students + proposed themselfs tend to be the most successful ones :-) ) + * nixness draft initiated + <antrik> youpi: BTW, I'm surprised that you didn't mention libc testsuite + before... working up from there is probably a more effective plan than + starting with high-level test suites like Python etc... + <youpi> wasn't it already in the gsoc proposal? + <youpi> bummer + <antrik> nope + +freenode, #hurd channel, 2011-03-06: + + <nixness> how's the hurd coding workflow, typically? + +*nixness* -> *foocraft*. + + <foocraft> we're discussing how TDD can work with Hurd (or general kernel + development) on #osdev + <foocraft> so what I wanted to know, since I haven't dealt much with GNU + Hurd, is how do you guys go about coding, in this case + <tschwinge> Our current workflow scheme is... well... is... + <tschwinge> Someone wants to work on something, or spots a bug, then works + on it, submits a patch, and 0 to 10 years later it is applied. + <tschwinge> Roughly. + <foocraft> hmm so there's no indicator of whether things broke with that + patch + <foocraft> and how low do you think we can get with tests? A friend of mine + was telling me that with kernel dev, you really don't know whether, for + instance, the stack even exists, and a lot of things that I, as a + programmer, can assume while writing code break when it comes to writing + kernel code + <foocraft> Autotest seems promising + +See autotest link given above. + + <foocraft> but in any case, coming up with the testing framework that + doesn't break when the OS itself breaks is hard, it seems + <foocraft> not sure if autotest isolates the mistakes in the os from + finding their way in the validity of the tests themselves + <youpi> it could be interesting to have scripts that automatically start a + sub-hurd to do the tests + +[[hurd/subhurd#unit_testing]]. + + <tschwinge> foocraft: To answer one of your earlier questions: you can do + really low-level testing. Like testing Mach's message passing. A + million times. The questions is whether that makes sense. And / or if + it makes sense to do that as part of a unit testing framework. Or rather + do such things manually once you suspect an error somewhere. + <tschwinge> The reason for the latter may be that Mach IPC is already + heavily tested during normal system operation. + <tschwinge> And yet, there still may be (there are, surely) bugs. + <tschwinge> But I guess that you have to stop at some (arbitrary?) level. + <foocraft> so we'd assume it works, and test from there + <tschwinge> Otherwise you'd be implementing the exact counter-part of what + you're testing. + <tschwinge> Which may be what you want, or may be not. Or it may just not + be feasible. + <foocraft> maybe the testing framework should have dependencies + <foocraft> which we can automate using make, and phony targets that run + tests + <foocraft> so everyone writes a test suite and says that it depends on A + and B working correctly + <foocraft> then it'd go try to run the tests for A etc. + <tschwinge> Hmm, isn't that -- on a high level -- have you have by + different packages? For example, the perl testsuite depends (inherently) + on glibc working properly. A perl program's testsuite depends on perl + working properly. + <foocraft> yeah, but afaik, the ordering is done by hand + +freenode, #hurd channel, 2011-03-07: + + <antrik> actually, I think for most tests it would be better not to use a + subhurd... that leads precisely to the problem that if something is + broken, you might have a hard time running the tests at all :-) + <antrik> foocraft: most of the Hurd code isn't really low-level. you can + use normal debugging and testing methods + <antrik> gnumach of course *does* have some low-level stuff, so if you add + unit tests to gnumach too, you might run into issues :-) + <antrik> tschwinge: I think testing IPC is a good thing. as I already said, + automated testing is *not* to discover existing but unknown bugs, but to + prevent new ones creeping in, and tracking progress on known bugs + <antrik> tschwinge: I think you are confusing unit testing and regression + testing. http://www.bddebian.com/~hurd-web/open_issues/unit_testing/ + talks about unit testing, but a lot (most?) of it is actually about + regression tests... + <tschwinge> antrik: That may certainly be -- I'm not at all an expert in + this, and just generally though that some sort of automated testing is + needed, and thus started collecting ideas. + <tschwinge> antrik: You're of course invited to fix that. + +IRC, freenode, #hurd, 2011-03-08 + +(After discussing the [[open_issues/anatomy_of_a_hurd_system]].) + + <antrik> so that's what your question is actually about? + <foocraft> so what I would imagine is a set of only-this-server tests for + each server, and then we can have fun adding composite tests + <foocraft> thus making debugging the composite scenarios a bit less tricky + <antrik> indeed + <foocraft> and if you were trying to pass a composite test, it would also + help knowing that you still didn't break the server-only test + <antrik> there are so many different things that can be tested... the + summer will only suffice to dip into this really :-) + <foocraft> yeah, I'm designing my proposal to focus on 1) make/use a + testing framework that fits the Hurd case very well 2) write some tests + and docs on how to write good tests + <antrik> well, doesn't have to be *one* framework... unit testing and + regression testing are quite different things, which can be covered by + different frameworks diff --git a/community/gsoc/project_ideas/testsuites.mdwn b/community/gsoc/project_ideas/testsuites.mdwn index 7cb7d103..f5ee2084 100644 --- a/community/gsoc/project_ideas/testsuites.mdwn +++ b/community/gsoc/project_ideas/testsuites.mdwn @@ -1,20 +1,27 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!meta title="Fix Compatibility Problems Exposed by Testsuites"]] A number of software packages come with extensive testsuites. -Some notable ones are libc, Perl, Python, GNU Coreutils, and glib. +Some notable ones are [[glibc|open_issues/glibc_testsuite]], gnulib, Perl, +Python, GNU Coreutils, and glib. While these testsuites were written mostly to track regressions in the respective packages, some of the tests fail on the Hurd in general. +There is also the [[Open POSIX Testsuite|open_issues/open_posix_test_suite]] +which is more of a whole system interface testing suite. + +Then, there is the [[open_issues/File_System_Exerciser]] which we can use to +test our file system servers for conformity. + While in some cases these might point to wrong usage of system interfaces, most of the time such failures are actually caused by shortcomings in Hurd's implementation of these interfaces. These shortcomings are often not obvious in normal use, @@ -42,6 +49,9 @@ While tracking down the various issues, the student will be digging into the inner workings of the Hurd, and thus gradually gaining the knowledge required for Hurd development in general. +A complementary task is adding a proper [[open_issues/unit_testing]] framework +to the GNU Hurd's code base, and related packages. + Possible mentors: Samuel Thibault (youpi) Exercise: Take a stab at one of the testsuite failures, diff --git a/open_issues/valgrind.mdwn b/community/gsoc/project_ideas/valgrind.mdwn index 2b0624d7..e9e94857 100644 --- a/open_issues/valgrind.mdwn +++ b/community/gsoc/project_ideas/valgrind.mdwn @@ -1,4 +1,5 @@ -[[!meta copyright="Copyright © 2009, 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2009, 2010, 2011 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -10,6 +11,8 @@ License|/fdl]]."]]"""]] [[!meta title="Porting Valgrind to the Hurd"]] +[[!tag open_issue_gnumach open_issue_hurd]] + [Valgrind](http://valgrind.org/) is an extremely useful debugging tool for memory errors. (And some other kinds of hard-to-find errors too.) Aside from being useful for program development in general, @@ -32,14 +35,14 @@ Compared to Linux, [[microkernel/Mach]] (the microkernel used by the Hurd) has very few kernel traps. Almost all [[system call]]s are implemented as [[RPC]]s instead -- either handled by Mach itself, or by the various [[Hurd servers|hurd/translator]]. -All RPCs use a pair of `mach_msg` invocations: +All RPCs use a pair of `mach_msg()` invocations: one to send a request message, and one to receive a reply. -However, while all RPCs use the same `mach_msg` trap, +However, while all RPCs use the same `mach_msg()` trap, the actual effect of the call varies greatly depending on which RPC is invoked -- -similar to the `ioctl` call on Linux. +similar to the `ioctl()` call on Linux. Each request thus must be handled individually. -Unlike `ioctl`, +Unlike `ioctl()`, the RPC invocations have explicit type information for the parameters though, which can be retrieved from the message header. By analyzing the parameters of the RPC reply message, diff --git a/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn b/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn index d55527a7..00d09094 100644 --- a/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn +++ b/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn @@ -56,7 +56,9 @@ For that I want to use: ### Tarfs apt-get install zip libz-dev libbz2-dev - hg clone http://bitbucket.org/ArneBab/hurd-tarfs tarfs + git clone git://git.sv.gnu.org/hurd/incubator.git tarfs + cd tarfs/ + git checkout tarfs/master cd tarfs make make install @@ -86,9 +88,12 @@ For that I want to use: cd ../.. mkdir test - settrans -c test2 /usr/local/bin/nsmux test - tar -cf test/intro.tar repos/hurd_intro - ls test2/intro.tar,,tarfs + touch test/hello + settrans -ca test2 /usr/local/bin/nsmux test + # tar -cvf test/intro.tar repos/hurd_intro + cat test2/hello + cat test2/hello,,hello + # Hello, World! ### clisp diff --git a/contributing/web_pages.mdwn b/contributing/web_pages.mdwn index 105d9dd5..351c82bd 100644 --- a/contributing/web_pages.mdwn +++ b/contributing/web_pages.mdwn @@ -234,6 +234,6 @@ download them from. Files to update: - * `/config_edittemplate/*.mdwn` + * `/config_edittemplate/*.mdwn`, `/.templates/autotag.tmpl` * `/contributing/web_pages/news/skeleton.mdwn` * `/copyright.mdwn` diff --git a/faq/which_microkernel.mdwn b/faq/which_microkernel.mdwn new file mode 100644 index 00000000..608e6b3f --- /dev/null +++ b/faq/which_microkernel.mdwn @@ -0,0 +1,15 @@ +[[!meta copyright="Copyright © 2009, 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="What happened with the ports to the L4 / Coyotos / Viengoos +microkernels?"]] + +This story is told on the page about the +[[history/port_to_another_microkernel]]. diff --git a/history.mdwn b/history.mdwn index 8f155b54..0abcbd52 100644 --- a/history.mdwn +++ b/history.mdwn @@ -1,13 +1,13 @@ -[[!meta copyright="Copyright © 1998, 1999, 2001, 2002, 2007, 2008, 2009 Free -Software Foundation, Inc."]] +[[!meta copyright="Copyright © 1998, 1999, 2001, 2002, 2007, 2008, 2009, 2011 +Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!tag stable_URL]] @@ -91,4 +91,4 @@ mailing lists. --- - * [[Port_to_L4]] + * [[Port_to_another_microkernel]] diff --git a/history/port_to_another_microkernel.mdwn b/history/port_to_another_microkernel.mdwn new file mode 100644 index 00000000..b347cf38 --- /dev/null +++ b/history/port_to_another_microkernel.mdwn @@ -0,0 +1,171 @@ +[[!meta copyright="Copyright © 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, +2009, 2010, 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Porting the Hurd to another microkernel"]] + +At first, there was an effort to port the Hurd from the [[Mach +microkernel|microkernel/mach]] to the [[L4_microkernel_family|microkernel/l4]]. +Then the story continued... + +[[!toc levels=2]] + + +# L4 + +## Initial Idea + +Encountering a number of fundamental design issues with the [[Mach +microkernel|microkernel/mach]] (mostly regarding [[resource +management|open_issues/resource_management_problems]]), some of the Hurd +developers began experimenting with using other microkernels for the Hurd +around the turn of the millenium. + +The idea of using L4 as a [[microkernel]] for a Hurd system was initially +voiced in the [[community]] by Okuji Yoshinori, who, for discussing this +purpose, created the [[mailing_lists/l4-hurd]] mailing list in November 2000. + +Over the years, a lot of discussion have been held on this mailing list, which +today is still the right place for [[next-generation Hurd|hurd/ng]] +discussions. + + +## Why? + +Even though that said resource management issues constitute a broad research +topic, there was no hope that the original Mach project would work on these: +[[microkernel/Mach]] wasn't maintained by its original authors anymore. Mach +had served its purpose as a research vehicle, and has been retired by its +stakeholders. + +Thus, switching to a well-maintained current [[microkernel]] was expected to +yield a more solid foundation for a Hurd system than the [[decaying +Mach|microkernel/mach/history]] design and implementation was able to. + +At that time, the [[L4 microkernel family|microkernel/L4]] was one obvious +choice. Being a second-generation microkernel, it was deemed to provide for a +faster system kernel implementation, especially in the time-critical [[IPC]] +paths. Also, as L4 was already implemented for a bunch of different +architectures (x86, Alpha, MIPS; also including SMP support), and the Hurd +itself being rather archtecture-agnostic, it was expected to be able to easily +support more platforms than with the existing system. + + +## Steps and Goals + +At the same time, the idea was -- while mucking with the system's core anyway +-- to improve on some fundamental design issues, too -- like the resource +management problems, for example. + +One goal of porting the Hurd to L4 was to make the Hurd independent of +[[microkernel/Mach]] interfaces, to make it somewhat microkernel-agnostic. + +One idea was to first introduce a Mach-on-L4 emulation layer, to easily get a +usable (though slow) Hurd-using-Mach-interfaces-on-L4 system, and then +gradually move the Hurd servers to use L4 intefaces rather than Mach ones. + +A design upon the lean L4 kernel would finally have made it feasible to move +devices drivers out of the kernel's [[TCB]]. + + +# Implementation + +The project itself then was mostly lead by Marcus Brinkmann and Neal Walfield. +Neal started the original Hurd/L4 port while visiting Karlsruhe university in +2002. He explains: + +> My intention was to adapt the Hurd to exploit L4's concepts and intended +> [[design_pattern]]s; it was not to simply provide a Mach +> [[compatibility_layer]] on top of L4. When I left Karlsruhe, I no longer had +> access to [[microkernel/l4/Pistachio]] as I was unwilling to sign an NDA. +> Although the specification was available, the Karlsruhe group only [released +> their code in May +> 2003](https://lists.ira.uni-karlsruhe.de/pipermail/l4ka/2003-May/000345.html). +> Around this time, Marcus began hacking on Pistachio. He created a relatively +> complete run-time. I didn't really become involved again until the second +> half of 2004, after I complete by Bachelors degree. + +Development of Hurd/L4 was done in the [CVS module +`hurd-l4`](http://savannah.gnu.org/cgi-bin/viewcvs/hurd/hurd-l4/). The `doc` +directory contains a design document that is worth reading for anyone who +wishes to learn more about Hurd/L4. + +Even though there was progress -- see, for example, the [[QEMU image for +L4|hurd/running/qemu/image_for_l4]] -- this port never reached a releasable +state. Simple POSIX programs, such as `banner` could run, but for more complex +system interfaces, a lot more work was needed. + +Eventually, a straight-forward port of the original Hurd's design wasn't deemed +feasible anymore by the developers, partly due to them not cosidering L4 +suitable for implementing a general-purpose operating system on top of it, and +because of deficiencies in the original Hurd's design, which they discovered +along their way. Neal goes on: + +> Before Marcus and I considered [[microkernel/Coyotos]], we had already +> rejected some parts of the Hurd's design. The +> [[open_issues/resource_management_problems]] were +> what prompted me to look at L4. Also, some of the problems with +> [[hurd/translator]]s were already well-known to us. (For a more detailed +> description of the problems we have identified, see our [[hurd/critique]] in the +> 2007 July's SIGOPS OSR. We have also written a forward-looking +> [[hurd/ng/position_paper]].) + +> We visited Jonathan Shapiro at Hopkins in January 2006. This resulted in a +> number of discussions, some quite influential, and not always in a way which +> aligned our position with that of Jonathan's. This was particularly true of +> a number of security issues. + +A lange number of discussion threads can be found in the archives of the +[[mailing_lists/l4-hurd]] mailing list. + +> Hurd-NG, as we originally called it, was an attempt to articulate the system +> that we had come to envision in terms of interfaces and description of the +> system's structure. The new name was selected, if I recall correctly, as it +> clearly wasn't the Hurd nor the Hurd based on L4. + + +## Termination + +As of 2005, development of Hurd/L4 has stopped. + + +# Coyotos + +Following that, an attempt was started to use the kernel of the +[[microkernel/Coyotos]] system. As Coyotos is an object capability system +througout, the microkernel would obviously be more suitable for this purpose; +and it looked pretty promising in the beginning. However, further +investigations found that there are some very fundamental philosophical +differences between the Coyotos and Hurd designs; and thus this this attempt +was also abandonned, around 2006 / 2007. (This time before producing any +actual code.) + + +# Viengoos + +By now (that is, after 2006), there were some new [[microkernel/L4]] variants +available, which added protected [[IPC]] paths and other features necessary for +object capability systems; so it might be possible to implement the Hurd on top +of these. However, by that time the developers concluded that microkernel +design and system design are interconnected in very intricate ways, and thus +trying to use a third-party microkernel will always result in trouble. So Neal +Walfield created the experimental [[microkernel/Viengoos]] kernel instead -- +based on the experience from the previous experiments with L4 and Coyotos -- +for his [[research on resource +management|open_issues/resource_management_problems]]. Currently he works in +another research area though, and thus Viengoos is on hold. + + +# Intermediate Results + +Note that while none of the microkernel work is active now, the previous +experiments already yielded a lot of experience, which will be very useful in +the further development / improvement of the mainline (Mach-based) Hurd +implementation. diff --git a/hurd/faq/which_microkernel.mdwn b/history/port_to_another_microkernel/discussion.mdwn index c5026afa..1a06c8ba 100644 --- a/hurd/faq/which_microkernel.mdwn +++ b/history/port_to_another_microkernel/discussion.mdwn @@ -8,15 +8,21 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -[[!meta title="What happened to the L4 / Coyotos / Viengoos microkernels?"]] +I can't say that I like the merge of FAQ entry an Hurd/L4 page. The stuff taken +from the L4 page creates the impression that Mach is a relic that we urgently +need to get rid of -- an impression I carefully tried to avoid in my own +writeup. Also, it's way too long IMHO. (In fact my own writeup is already on +the fringe of being too wordy I fear.) And I don't think that bare links to +other pages make good FAQ answers in general. -L4 was promising but happened to not be suitable for implementing a general-purpose operating system on top of it. See [[history/port_to_l4]] for the historical details. +IMHO the original FAQ entry should be restored (with some details fixed); while +the Hurd/L4 page can be referenced for further details. -Coyotos is abandoned upstream +-- antrik -Neal Walfield started working on a newly designed kernel called [[viengoos|microkernel/viengoos]]. Unfortunately, he currently lacks time and the projects it paused. +> OK, I understand (and accept) your rationale; [[!taglink +> open_issue_documentation]]. --[[tschwinge]] -In the meanwhile, people are thus continuing with [[microkernel/mach]]. --- diff --git a/history/port_to_l4.mdwn b/history/port_to_l4.mdwn index b58c0d91..3f951a64 100644 --- a/history/port_to_l4.mdwn +++ b/history/port_to_l4.mdwn @@ -1,108 +1,13 @@ -[[!meta copyright="Copyright © 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, -2009, 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -[[!meta title="Porting the Hurd to L4: Hurd/L4"]] +[[!tag stable_URL]] -There was an effort to port the Hurd from [[microkernel/Mach]] to the -[[L4_microkernel_family|microkernel/L4]]. - -The idea of using L4 as a [[microkernel]] for a [[Hurd_system|hurd]] was -initially voiced in the [[Hurd_community|community]] by Okuji Yoshinori, who, -for discussing this purpose, created the [[mailing lists/l4-hurd]] mailing list -in November 2000. - -The project itself then was mostly lead by Marcus Brinkmann and Neal Walfield. -Even though there was progress -- see, for example, the -[[QEMU image for L4|hurd/running/qemu/image for l4]] -- this port never reached a -releasable state. Eventually, a straight-forward port of the original Hurd's -design wasn't deemed feasible anymore by the developers, partly due to them not -cosidering L4 suitable for implementing a general-purpose operating system on -top of it, and because of deficiencies in the original Hurd's design, which -they discovered along their way. Read the [[hurd/critique]] and a -[[hurd/ng/position paper]]. - -By now, the development of Hurd/L4 has stopped. However, Neal Walfield moved -on to working on a newly designed kernel called [[microkernel/viengoos]]. - -Over the years, a lot of discussion have been held on the -[[mailing lists/l4-hurd]] mailing list, which today is still the right place -for [[next-generation Hurd|hurd/ng]] discussions. - -Development of Hurd/L4 was done in the `hurd-l4` module of the Hurd CVS -repository. The `doc` directory contains a design document that is worth -reading for anyone who wishes to learn more about Hurd/L4. - - -One goal of porting the Hurd to L4 was to make the Hurd independend of Mach -interfaces, to make it somewhat microkernel-agnostic. - -Mach wasn't maintained by its original authors anymore, so switching to a -well-maintained current [[microkernel]] was expected to yield a more solid -foundation for a Hurd system than the decaying Mach design and implementation -was able to. - -L4 being a second-generation [[microkernel]] was deemed to provide for a faster -system kernel implementation, especially in the time-critical [[IPC]] paths. -Also, as L4 was already implemented for a bunch of different architectures -(IA32, Alpha, MIPS; SMP), and the Hurd itself being rather archtecture-unaware, -it was expected to be able to easily support more platforms than with the -existing system. - -A design upon the lean L4 kernel would finally have moved devices drivers out -of the kernel's [[TCB]]. - - -One idea was to first introduce a Mach-on-L4 emulation layer, to easily get a -usable (though slow) Hurd-using-Mach-interfaces-on-L4 system, and then -gradually move the Hurd servers to use L4 intefaces rather than Mach ones. - - -Neal Walfield started the original Hurd/L4 port while at Karlsruhe in 2002. He -explains: - -> My intention was to adapt the Hurd to exploit L4's concepts and intended -> [[design_pattern]]s; it was not to simply provide a Mach -> [[compatibility_layer]] on top of L4. When I left Karlsruhe, I no longer had -> access to [[microkernel/l4/Pistachio]] as I was unwilling to sign an NDA. -> Although the specification was available, the Karlsruhe group only [released -> their code in May -> 2003](https://lists.ira.uni-karlsruhe.de/pipermail/l4ka/2003-May/000345.html). -> Around this time, Marcus began hacking on Pistachio. He created a relatively -> complete run-time. I didn't really become involved again until the second -> half of 2004, after I complete by Bachelors degree. - -> Before Marcus and I considered [[microkernel/Coyotos]], we had already -> rejected some parts of the Hurd's design. The -> [[open issues/resource management problems]] were -> what prompted me to look at L4. Also, some of the problems with -> [[hurd/translator]]s were already well-known to us. (For a more detailed -> description of the problems we have identified, see our [[hurd/critique]] in the -> 2007 July's SIGOPS OSR. We have also written a forward-looking -> [[hurd/ng/position paper]].) - -> We visited Jonathan Shapiro at Hopkins in January 2006. This resulted in a -> number of discussions, some quite influential, and not always in a way which -> aligned our position with that of Jonathan's. This was particularly true of -> a number of security issues. - -A lange number of discussion threads can be found in the archives of the -[[mailing lists/l4-hurd]] mailing list. - -> Hurd-NG, as we originally called it, was an attempt to articulate the system -> that we had come to envision in terms of interfaces and description of the -> system's structure. The new name was selected, if I recall correctly, as it -> clearly wasn't the Hurd nor the Hurd based on L4. - - -The source code is still available in [CVS module -`hurd-l4`](http://savannah.gnu.org/cgi-bin/viewcvs/hurd/hurd-l4/) (note that -this repository has in the beginning also been used for Neal's -[[microkernel/Viengoos]]). +[[!meta redir=port_to_another_microkernel]] diff --git a/hurd-l4.mdwn b/hurd-l4.mdwn index 579c1190..afc8f8f3 100644 --- a/hurd-l4.mdwn +++ b/hurd-l4.mdwn @@ -1,13 +1,13 @@ -[[!meta copyright="Copyright © 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2009, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!tag stable_URL]] -[[!meta redir=history/port_to_l4]] +[[!meta redir=history/port_to_another_microkernel]] @@ -1,13 +1,13 @@ [[!meta copyright="Copyright © 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, -2009, 2010 Free Software Foundation, Inc."]] +2009, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] The GNU Hurd is under active development. Because of that, there is no *stable* version. We distribute the Hurd sources only through [[Git|source_repositories]] at present. @@ -31,7 +31,7 @@ in the *unstable* branch of the Debian archive. * [[What_Is_the_GNU_Hurd]] - A Brief Description * [[Advantages]]. And [[challenges]]. * [[History]] - * [[history/Port_to_L4]] + * [[history/Port_to_another_microkernel]] * [[Logo]] * [[Status]] * [[KnownHurdLimits]] diff --git a/hurd/ng.mdwn b/hurd/ng.mdwn index de33949d..481386a4 100644 --- a/hurd/ng.mdwn +++ b/hurd/ng.mdwn @@ -13,7 +13,7 @@ This section explains the motivations behind the new design: * [[Issues_with_L4_Pistachio]] * [[Limitations_of_the_original_Hurd_design]] - * History of the [[history/port_to_L4]] + * History of the [[history/port_to_another_microkernel]] # Work already done diff --git a/hurd/running/live_cd.mdwn b/hurd/running/live_cd.mdwn index 42d85c5b..c9360594 100644 --- a/hurd/running/live_cd.mdwn +++ b/hurd/running/live_cd.mdwn @@ -1,4 +1,6 @@ -You can download a gzipped iso of a Hurd Live CD at +[[Arch Hurd|hurd/running/arch_hurd/]] offers Hurd LiveCDs at <http://www.archhurd.org/gethurd.php>. + +Also you can download a gzipped iso of a Hurd Live CD at <http://www.superunprivileged.org/hurd/live-cd/>. The Superunpriveleged crew also offers a tiny Hurd Live CD that is under 10 diff --git a/hurd/running/qemu.mdwn b/hurd/running/qemu.mdwn index c5b2cf1c..141ab2b1 100644 --- a/hurd/running/qemu.mdwn +++ b/hurd/running/qemu.mdwn @@ -27,6 +27,12 @@ Also you can use qemu to easily try one of our [[Hurd_LiveCDs|hurd/running/live_ # What is Needed to create a QEMU image +## Debian Installer + +Instructions for creating a qemu image from the install CDs from debian installer can be found in the README alongside the d-i Hurd images: <http://people.debian.org/~sthibault/hurd-i386/installer/cdimage/> + +## Old method + 1. First thing is to install [[/QEMU]]. 2. A [[grub]] boot disk for the floppy disk image needed for booting. The [0\.97 version](ftp://alpha.gnu.org/gnu/grub/grub-0.97-i386-pc.ext2fs) works fine. I downloaded it and renamed to `floppy.img`. Alternatively, the Debian grub-disk package (up till version 0.97-28) is fine as well. 3. You will need a [Debian/Hurd installation CD](http://www.debian.org/ports/hurd/hurd-cd). K16 works fine. diff --git a/hurd/subhurd.mdwn b/hurd/subhurd.mdwn index cb4a40a8..b8e595d3 100644 --- a/hurd/subhurd.mdwn +++ b/hurd/subhurd.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2007, 2008, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2007, 2008, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -21,6 +21,8 @@ possible to use [[debugging/gdb/noninvasive_debugging]], but this is less flexible.) Vice versa, it is also possible to use a subhurd to debug the *main* Hurd system, for example, the latter's root file system. +[[!toc levels=2]] + # Howto @@ -139,3 +141,47 @@ translator|translator/ext2fs]]'s [[PID]], but this is no problem, as currently [[PID]]s are visible across subhurd boundaries. (It is a [[!taglink open_issue_hurd]] whether this is the right thing to do in [[open_issues/virtualization]] contexts, but that's how it currently is.) + + +<a name="unit_testing"></a> +## Unit Testing + +freenode, #hurd channel, 2011-03-06: + +From [[open_issues/unit_testing]]. + + <youpi> it could be interesting to have scripts that automatically start a + sub-hurd to do the tests + <youpi> though that'd catch sub-hurd issues :) + <foocraft> so a sub-hurd is a hurd that I can run on something that I know + works(like linux)? + <foocraft> Virtual machine I would think + <foocraft> and over a network connection it would submit results back to + the host :p + * foocraft brain damage + <youpi> sub-hurd is a bit like chroot + <youpi> except that it's more complete + <foocraft> oh okay + <youpi> i.e. almost everything gets replaced with what you want, except the + micro-kernel + <youpi> that way you can even test the exec server for instance, without + risks of damaging the host OS + <foocraft> and we know the micro-kernel works correctly, right youpi? + <youpi> well, at least it's small enough that most bugs are not there + <foocraft> 1) all tests run in subhurd 2) output results in a place in the + subhurd 3) tester in the host checks the result and pretty-prints it 4) + rinse & repeat + <youpi> the output can actually be redirected iirc + <youpi> since you give the sub-hurd a "console" + <foocraft> youpi, yup yeah, so now it's more like chroot if that's the case + <youpi> it really looks like chroot, yes + <foocraft> but again, there's this subset of tests that we need to have + that ensures that even the tester running on the subhurd is valid, and it + didn't break because of a bug in the subhurd + <tschwinge> As long as you do in-system testing, you'll always (have to) + rely on some functionality provided by the host system. + <foocraft> the worst thing that could happen with unit testing is false + results that lead someone to try to fix something that isn't broken :p + <tschwinge> Yes. + <youpi> usually one tries to repeat the test by hand in a normal + environment @@ -161,7 +161,6 @@ a:hover /* Placement. */ .sidebar { - width: auto; margin-left: 20px; } @@ -19,7 +19,7 @@ License|/fdl]]."]]"""]] # Related - * [[open_issues/dtrace]] + * [[community/gsoc/project_ideas/dtrace]] * [[SystemTap]] diff --git a/microkernel/coyotos.mdwn b/microkernel/coyotos.mdwn index 5ecea688..fec023ba 100644 --- a/microkernel/coyotos.mdwn +++ b/microkernel/coyotos.mdwn @@ -1,5 +1,5 @@ -[[!meta copyright="Copyright © 2006, 2007, 2008, 2010 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2006, 2007, 2008, 2010, 2011 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -28,3 +28,6 @@ design. The coyotos microkernel specification can be found [here](http://www.coyotos.org/docs/ukernel/spec.html). + +There once was the idea of a GNU/Hurd [[port using the Coyotos +microkernel|history/port_to_another_microkernel]], but this didn't come live. diff --git a/microkernel/l4.mdwn b/microkernel/l4.mdwn index 45929842..7af5e6fc 100644 --- a/microkernel/l4.mdwn +++ b/microkernel/l4.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2004, 2006, 2007, 2008, 2010 Free Software +[[!meta copyright="Copyright © 2004, 2006, 2007, 2008, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -20,7 +20,8 @@ on formally verifying an L4 microkernel. * {{$sel4}} -There was a GNU/Hurd [[history/port_to_L4]], which is now stalled. +There was a GNU/Hurd [[port to L4|history/port_to_another_microkernel]], which +is now stalled. [[!ymlfront data=""" diff --git a/news/2010.mdwn b/news/2010.mdwn index f6a86052..2ba85266 100644 --- a/news/2010.mdwn +++ b/news/2010.mdwn @@ -122,4 +122,9 @@ interested in contributing to Hurd development in general, please see talk to us at <bug-hurd@gnu.org> or the `#hurd` IRC channel on freenode. +--- + +French article by Manuel Menal, [*Gnu : L'année 2010 du +Hurd*](http://linuxfr.org/news/lann%C3%A9e-2010-du-hurd). + """]] diff --git a/news/2011-03-26.mdwn b/news/2011-03-26.mdwn new file mode 100644 index 00000000..588f5fcf --- /dev/null +++ b/news/2011-03-26.mdwn @@ -0,0 +1,15 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta date="2011-03-26 14:20 UTC"]] + +The **Google Summer of Code 2011** is on! If you're a student, consider +applying for a GNU Hurd project -- details to be found on our +*[[community/GSoC]] page*. diff --git a/open_issues/anatomy_of_a_hurd_system.mdwn b/open_issues/anatomy_of_a_hurd_system.mdwn new file mode 100644 index 00000000..e1d5c9d8 --- /dev/null +++ b/open_issues/anatomy_of_a_hurd_system.mdwn @@ -0,0 +1,73 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!taglink open_issue_documentation]] + +A bunch of this should also be covered in other (introductionary) material, +like Bushnell's Hurd paper. All this should be unfied and streamlined. + +IRC, freenode, #hurd, 2011-03-08 + + <foocraft> I've a question on what are the "units" in the hurd project, if + you were to divide them into units if they aren't, and what are the + dependency relations between those units(roughly, nothing too pedantic + for now) + <antrik> there is GNU Mach (the microkernel); there are the server + libraries in the Hurd package; there are the actual servers in the same; + and there is the POSIX implementation layer in glibc + <antrik> relations are a bit tricky + <antrik> Mach is the base layer which implements IPC and memory management + <foocraft> hmm I'll probably allocate time for dependency graph generation, + in the worst case + <antrik> on top of this, the Hurd servers, using the server libraries, + implement various aspects of the system functionality + <antrik> client programs use libc calls to use the servers + <antrik> (servers also use libc to communicate with other servers and/or + Mach though) + <foocraft> so every server depends solely on mach, and no other server? + <foocraft> s/mach/mach and/or libc/ + <antrik> I think these things should be pretty clear one you are somewhat + familiar with the Hurd architecture... nothing really tricky there + <antrik> no + <antrik> servers often depend on other servers for certain functionality + +--- + +IRC, freenode, #hurd, 2011-03-12 + + <dEhiN> when mach first starts up, does it have some basic i/o or fs + functionality built into it to start up the initial hurd translators? + <antrik> I/O is presently completely in Mach + <antrik> filesystems are in userspace + <antrik> the root filesystem and exec server are loaded by grub + <dEhiN> o I see + <dEhiN> so in order to start hurd, you would have to start mach and + simultaneously start the root filesystem and exec server? + <antrik> not exactly + <antrik> GRUB loads all three, and then starts Mach. Mach in turn starts + the servers according to the multiboot information passed from GRUB + <dEhiN> ok, so does GRUB load them into ram? + <dEhiN> I'm trying to figure out in my mind how hurd is initially started + up from a low-level pov + <antrik> yes, as I said, GRUB loads them + <dEhiN> ok, thanks antrik...I'm new to the idea of microkernels, but a + veteran of monolithic kernels + <dEhiN> although I just learned that windows nt is a hybrid kernel which I + never knew! + <rm> note there's a /hurd/ext2fs.static + <rm> I belive that's what is used initially... right? + <antrik> yes + <antrik> loading the shared libraries in addition to the actual server + would be unweildy + <antrik> so the root FS server is linked statically instead + <dEhiN> what does the root FS server do? + <antrik> well, it serves the root FS ;-) + <antrik> it also does some bootstrapping work during startup, to bring the + rest of the system up diff --git a/open_issues/code_analysis.mdwn b/open_issues/code_analysis.mdwn index ad59f962..21e09089 100644 --- a/open_issues/code_analysis.mdwn +++ b/open_issues/code_analysis.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -40,7 +40,7 @@ analysis|performance]], [[formal_verification]], as well as general * <http://blog.llvm.org/2010/04/whats-wrong-with-this-code.html> - * [[Valgrind]] + * [[community/gsoc/project_ideas/Valgrind]] * [Smatch](http://smatch.sourceforge.net/) diff --git a/open_issues/crash_server.mdwn b/open_issues/crash_server.mdwn index d97f5458..7ed4afbf 100644 --- a/open_issues/crash_server.mdwn +++ b/open_issues/crash_server.mdwn @@ -1,4 +1,5 @@ -[[!meta copyright="Copyright © 2009, 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2009, 2010, 2011 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -187,3 +188,8 @@ one... /home/tschwinge/tmp/gnumach/gnumach-1-branch-Xen-branch.build/../gnumach-1-branch-Xen-branch/kern/ipc_kobject.c:76 mach_msg_trap /home/tschwinge/tmp/gnumach/gnumach-1-branch-Xen-branch.build/../gnumach-1-branch-Xen-branch/ipc/mach_msg.c:1367 + +--- + +If someone is working in this area, they may want to have a look at +[[GDB_gcore]], and port <http://code.google.com/p/google-coredumper/>, too. diff --git a/open_issues/debian_cross_toolchain.mdwn b/open_issues/debian_cross_toolchain.mdwn new file mode 100644 index 00000000..e0665466 --- /dev/null +++ b/open_issues/debian_cross_toolchain.mdwn @@ -0,0 +1,15 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +Have a look at the Debian *Cross Toolchain* project, +<https://alioth.debian.org/projects/crosstoolchain/>, +<http://wiki.debian.org/ToolChain/Cross>. diff --git a/open_issues/debugging.mdwn b/open_issues/debugging.mdwn index e66a086f..e5fbf7a0 100644 --- a/open_issues/debugging.mdwn +++ b/open_issues/debugging.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -42,7 +42,10 @@ We have debugging infrastructure. For example: Continues: <http://lwn.net/Articles/414264/>, which introduces <http://dmtcp.sourceforge.net/>. - * [[locking]] + * [[crash_server}}, [[GDB_gcore]], + <http://code.google.com/p/google-coredumper/> + + * [[community/gsoc/project_ideas/libdiskfs_locking]] * <http://lwn.net/Articles/415728/>, or <http://lwn.net/Articles/415471/> -- just two examples; there's a lot of such stuff for Linux. diff --git a/open_issues/dtrace.mdwn b/open_issues/dtrace.mdwn deleted file mode 100644 index cbac28fb..00000000 --- a/open_issues/dtrace.mdwn +++ /dev/null @@ -1,47 +0,0 @@ -[[!meta copyright="Copyright © 2008, 2009, 2011 Free Software Foundation, -Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -One of the main problems of the current Hurd implementation is very poor -[[performance]]. While we have a bunch of ideas what could cause the performance -problems, these are mostly just guesses. Better understanding what really -causes bad performance is necessary to improve the situation. - -For that, we need tools for performance measurements. While all kinds of more -or less specific [[profiling]] tools could be conceived, the most promising and -generic approach seems to be a framework for logging certain events in the -running system (both in the microkernel and in the Hurd servers). This would -allow checking how much time is spent in certain modules, how often certain -situations occur, how things interact, etc. It could also prove helpful in -debugging some issues that are otherwise hard to find because of complex -interactions. - -The most popular framework for that is Sun's dtrace; but there might be others. -The student has to evaluate the existing options, deciding which makes most -sense for the Hurd; and implement that one. (Apple's implementation of dtrace -in their Mach-based kernel might be helpful here...) - -This project requires ability to evaluate possible solutions, and experience -with integrating existing components as well as low-level programming. - -Possible mentors: Samuel Thibault (youpi) - -Related: [[profiling]], [[LTTng]], [[SystemTap]] - -Exercise: In lack of a good exercise directly related to this task, just pick -one of the kernel-related or generally low-level tasks from the bug/task -trackers on savannah, and make a go at it. You might not be able to finish the -task in a limited amount of time, but you should at least be able to make a -detailed analysis of the issue. - -*Status*: Andei Barbu was working on -[SystemTap](http://csclub.uwaterloo.ca/~abarbu/hurd/) for GSoC 2008, but it -turned out too Linux-specific. He implemented kernel probes, but there is no -nice frontend yet. diff --git a/open_issues/ext2fs_page_cache_swapping_leak.mdwn b/open_issues/ext2fs_page_cache_swapping_leak.mdwn new file mode 100644 index 00000000..0ace5cd3 --- /dev/null +++ b/open_issues/ext2fs_page_cache_swapping_leak.mdwn @@ -0,0 +1,23 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +IRC, OFTC, #debian-hurd, 2011-03-24 + + <youpi> I still believe we have an ext2fs page cache swapping leak, however + <youpi> as the 1.8GiB swap was full, yet the ld process was only 1.5GiB big + <pinotree> a leak at swapping time, you mean? + <youpi> I mean the ext2fs page cache being swapped out instead of simply + dropped + <pinotree> ah + <pinotree> so the swap tends to accumulate unuseful stuff, i see + <youpi> yes + <youpi> the disk content, basicallyt :) diff --git a/open_issues/sudo_date_crash.mdwn b/open_issues/file_system_exerciser.mdwn index 53303abc..4277e5e7 100644 --- a/open_issues/sudo_date_crash.mdwn +++ b/open_issues/file_system_exerciser.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -8,9 +8,8 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -[[!tag open_issue_gnumach]] +[[!tag open_issue_hurd]] -IRC, unknown channel, unknown date. +Test our file system implementations with the File System Exerciser. - <grey_gandalf> I did a sudo date... - <grey_gandalf> and the machine hangs + * <http://codemonkey.org.uk/projects/fsx/> diff --git a/open_issues/gdb_gcore.mdwn b/open_issues/gdb_gcore.mdwn index 7d4980f1..69211ac0 100644 --- a/open_issues/gdb_gcore.mdwn +++ b/open_issues/gdb_gcore.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2009, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -21,3 +21,6 @@ GDB's `gcore` command doesn't work / needs to be implemented / ported in GDB: /media/data/home/tschwinge/core.cA0ICY:2: Error in sourced command file: Undefined command: "gcore". Try "help". gcore: failed to create core.8371 + +If someone is working in this area, they may want to port +<http://code.google.com/p/google-coredumper/>, too. diff --git a/open_issues/gdb_noninvasive_mode_new_threads.mdwn b/open_issues/gdb_noninvasive_mode_new_threads.mdwn new file mode 100644 index 00000000..9b3992f4 --- /dev/null +++ b/open_issues/gdb_noninvasive_mode_new_threads.mdwn @@ -0,0 +1,15 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_gdb]] + +Debugging a translator. `gdb binary`. `set noninvasive on`. `attach [PID]`. +Translator does some work. GDB doesn't notice new threads. `detach`. `attach +[PID]` -- now new threads are visible. diff --git a/open_issues/gdb_thread_ids.mdwn b/open_issues/gdb_thread_ids.mdwn index c31a9967..c04a10ee 100644 --- a/open_issues/gdb_thread_ids.mdwn +++ b/open_issues/gdb_thread_ids.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -6,8 +6,8 @@ id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!meta title="GDB: thread ids"]] @@ -23,3 +23,9 @@ GNU GDB's Pedro Alves: Also see [[thread numbering of ps and GDB]]. + +--- + +`attach` to a multi-threaded process. See threads 1 to 5. `detach`. `attach` +again -- thread numbers continue where they stopped last time: now they're +threads 6 to 10. diff --git a/open_issues/multithreading.mdwn b/open_issues/multithreading.mdwn index 39203333..addc29c3 100644 --- a/open_issues/multithreading.mdwn +++ b/open_issues/multithreading.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -10,7 +10,21 @@ License|/fdl]]."]]"""]] [[!tag open_issue_hurd]] -Hurd servers / VFS libraries are multithreaded, roughly using one thread per +Hurd servers / VFS libraries are multithreaded. + + +# Implementation + + * well-known threading libraries + + * [[hurd/libthreads]] + + * [[hurd/libpthread]] + + +# Design + +Roughly using one thread per incoming request. This is not the best approach: it doesn't really make sense to scale the number of worker threads with the number of incoming requests, but instead they should be scaled according to the backends' characteristics. @@ -18,7 +32,9 @@ instead they should be scaled according to the backends' characteristics. The [[hurd/Critique]] should have some more on this. -Alternative approaches: +# Alternative approaches: + + * <http://www.concurrencykit.org/> * Continuation-passing style diff --git a/open_issues/nightly_builds.mdwn b/open_issues/nightly_builds.mdwn index 506697bb..5d1257fb 100644 --- a/open_issues/nightly_builds.mdwn +++ b/open_issues/nightly_builds.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -15,6 +15,8 @@ Resources: * [[toolchain/cross-gnu]] + * [[Debian_Cross_Toolchain]] + * As reported in the [[news/2010-05-31]] news, there's Hydra doing nightly builds / Nix packages. diff --git a/open_issues/nightly_builds_deb_packages.mdwn b/open_issues/nightly_builds_deb_packages.mdwn index 9f5e2373..11fc4c79 100644 --- a/open_issues/nightly_builds_deb_packages.mdwn +++ b/open_issues/nightly_builds_deb_packages.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -24,4 +24,8 @@ There is infrastructure available to test whole OS installations. --- +[[Debian_Cross_Toolchain]] for cross-building? + +--- + See also [[nightly_builds]]. diff --git a/open_issues/performance.mdwn b/open_issues/performance.mdwn index 9b3701b3..eb9f3f8a 100644 --- a/open_issues/performance.mdwn +++ b/open_issues/performance.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -18,7 +18,7 @@ In [[microkernel]]-based systems, there is generally a considerable [[RPC]] overhead. In a multi-server system, it is non-trivial to implement a high-performance -[[I/O System|io_system]]. +[[I/O System|community/gsoc/project_ideas/disk_io_performance]]. When providing [[faq/POSIX_compatibility]] (and similar interfaces) in an environemnt that doesn't natively implement these interfaces, there may be a diff --git a/open_issues/performance/fork.mdwn b/open_issues/performance/fork.mdwn index 2748be53..5ceb6455 100644 --- a/open_issues/performance/fork.mdwn +++ b/open_issues/performance/fork.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -14,3 +14,24 @@ Our [[`fork` implementation|glibc/fork]] is nontrivial. To do: hard numbers. [[Microbenchmarks]]? + + +# Windows / Cygwin + + * <http://www.google.com/search?q=cygwin+fork> + + * <http://www.redhat.com/support/wpapers/cygnus/cygnus_cygwin/architecture.html> + + In particular, *5.6. Process Creation*. + + * <http://archive.gamedev.net/community/forums/topic.asp?topic_id=360290> + + * <http://cygwin.com/cgi-bin/cvsweb.cgi/src/winsup/cygwin/how-cygheap-works.txt?cvsroot=src> + + > Cygwin has recently adopted something called the "cygwin heap". This is + > an internal heap that is inherited by forked/execed children. It + > consists of process specific information that should be inherited. So + > things like the file descriptor table, the current working directory, and + > the chroot value live there. + + * <http://www.perlmonks.org/?node_id=588994> diff --git a/open_issues/performance/io_system/binutils_ld_64ksec.mdwn b/open_issues/performance/io_system/binutils_ld_64ksec.mdwn index b59a87a7..79c2300f 100644 --- a/open_issues/performance/io_system/binutils_ld_64ksec.mdwn +++ b/open_issues/performance/io_system/binutils_ld_64ksec.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -10,7 +10,8 @@ License|/fdl]]."]]"""]] [[!tag open_issue_hurd]] -This one may be considered as a testcase for I/O system optimization. +This one may be considered as a testcase for [[I/O system +optimization|community/gsoc/project_ideas/disk_io_performance]]. It is taken from the [[binutils testsuite|binutils]], `ld/ld-elf/sec64k.exp`, where this diff --git a/open_issues/performance/io_system/clustered_page_faults.mdwn b/open_issues/performance/io_system/clustered_page_faults.mdwn index 3a187523..37433e06 100644 --- a/open_issues/performance/io_system/clustered_page_faults.mdwn +++ b/open_issues/performance/io_system/clustered_page_faults.mdwn @@ -10,6 +10,8 @@ License|/fdl]]."]]"""]] [[!tag open_issue_gnumach open_issue_hurd]] +[[community/gsoc/project_ideas/disk_io_performance]]. + IRC, freenode, #hurd, 2011-02-16 <braunr> exceptfor the kernel, everything in an address space is diff --git a/open_issues/performance/io_system/read-ahead.mdwn b/open_issues/performance/io_system/read-ahead.mdwn index 3ee30b5d..b6851edd 100644 --- a/open_issues/performance/io_system/read-ahead.mdwn +++ b/open_issues/performance/io_system/read-ahead.mdwn @@ -10,6 +10,8 @@ License|/fdl]]."]]"""]] [[!tag open_issue_gnumach open_issue_hurd]] +[[community/gsoc/project_ideas/disk_io_performance]] + IRC, #hurd, freenode, 2011-02-13: <etenil> youpi: Would libdiskfs/diskfs.h be in the right place to make diff --git a/open_issues/perlmagick.mdwn b/open_issues/perlmagick.mdwn index 1daac62b..8a57a8fd 100644 --- a/open_issues/perlmagick.mdwn +++ b/open_issues/perlmagick.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -57,6 +57,49 @@ Etc. +/usr/lib/gcc/i486-gnu/4.4.2/include/omp.h: +# State as of 2011-03-06 + +freenode, #hurd channel, 2011-03-06: + + <pinotree> tschwinge: (speaking on working perl, how did it en with that + "(glibc) double free" crash with perl?) + <pinotree> *end + <tschwinge> I think I remember I suspected it's a libgomp (!) issue in the + end. I have not yet continued working on that. + <pinotree> libogmp? looks like you know more than me, then :) + <youpi> tschwinge: oh, I'm interested + <youpi> I know a bit about libgomp :) + <tschwinge> I bisected this down to where Imagemagick added -fgomp (or + whatever it is). And then the perl library (Imagemagick.pm?) which loads + the imagemagick.so segfaulted. + <tschwinge> ImageMagick did this change in the middle of a x.x.x.something + release.. + <tschwinge> My next step would have been to test whether libgomp works at + all for us. + <youpi> ./usr/sbin/debootstrap:DEBOOTSTRAP_CHECKSUM_FIELD="SHA$SHA_SIZE" + <youpi> erf + <youpi> so they switched to another checksum + <youpi> but we don't have that one on all of our packages :) + <youpi> tschwinge: + <youpi> buildd@bach:~$ OMP_NUM_THREADS=2 ./test + <youpi> I'm 0x1 + <youpi> I'm 0x3 + <youpi> libgomp works at least a bit + <tschwinge> OK. + <pinotree> i guess we should hope the working bits don't stop at that point + ;) + <tschwinge> If open_issues/perlmagick is to be believed a diff of 6.4.1-1 + and 6.4.1-2 should tell what exactly was changed. + <tschwinge> Oh! + <tschwinge> I even have it on the page already! ;-) + <tschwinge> -fopenmp + <youpi> I've tried the pragmas that imagemagick uses + <youpi> they work + <tschwinge> Might be the issue fixed itself? + <youpi> I don't know, it's the latest libc here + <youpi> (and latest hurd, to be uploaded) + + # Other [[!debbug 551017]] diff --git a/open_issues/pfinet_vs_system_time_changes.mdwn b/open_issues/pfinet_vs_system_time_changes.mdwn new file mode 100644 index 00000000..714c8784 --- /dev/null +++ b/open_issues/pfinet_vs_system_time_changes.mdwn @@ -0,0 +1,42 @@ +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +IRC, unknown channel, unknown date. + + <grey_gandalf> I did a sudo date... + <grey_gandalf> and the machine hangs + +This was very likely a misdiagnosis: + +IRC, freenode, #hurd, 2011-03-25 + + <tschwinge> antrik: I suspect it'S some timing stuff in pfinet that perhaps + uses absolute time, and somehow wildely gets confused? + <antrik> tschwinge: BTW, pfinet doesn't actually die I think -- it just + drops open connections... + <antrik> perhaps it thinks they timed out + <tschwinge> antrik: Isn't the translator restarted instead? + <antrik> don't think so + <antrik> when pfinet actually dies, I also loose the NFS mounts, which + doesn't happen in this case + <antrik> hehe "... and the machine hangs" + <antrik> he didn't bother to check that the machine is perfectly fine, only + the SSH connection got dropped + <tschwinge> Ah, I see. So it'S perhaps indeed simply closes TCP + connections that have been without data for ``too long''? + <antrik> yeah, that's my guess + <antrik> my clock is speeding, so ntpdate sets it in the past + <antrik> perhaps there is some math that concludes the connection have been + inactive for -200 seconds, which (unsigned) is more than any timeout :-) + <tschwinge> (The other way round, you might likely get some integer + wrap-around, and thus the same result.) + <tschwinge> Yes. diff --git a/open_issues/profiling.mdwn b/open_issues/profiling.mdwn index e04fb08a..7e3c7350 100644 --- a/open_issues/profiling.mdwn +++ b/open_issues/profiling.mdwn @@ -17,7 +17,7 @@ done for [[performance analysis|performance]] reasons. Should be working, but some issues have been reported, regarding GCC spec files. Should be possible to fix (if not yet done) easily. - * [[dtrace]] + * [[community/gsoc/project_ideas/dtrace]] Have a look at this, integrate it into the main trees. diff --git a/open_issues/rpc_to_self_with_rendez-vous_leading_to_duplicate_port_destroy.mdwn b/open_issues/rpc_to_self_with_rendez-vous_leading_to_duplicate_port_destroy.mdwn new file mode 100644 index 00000000..9db92250 --- /dev/null +++ b/open_issues/rpc_to_self_with_rendez-vous_leading_to_duplicate_port_destroy.mdwn @@ -0,0 +1,163 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +[RPC to self with rendez-vous leading to duplicate port +destroy](http://lists.gnu.org/archive/html/bug-hurd/2011-03/msg00045.html) + +IRC, freenode, #hurd, 2011-03-14 + + <antrik> youpi: I wonder, why does the root FS call diskfs_S_dir_lookup() + at all?... + <youpi> errr, because a client asked for it? + <youpi> (problem with RPCs is you can't easily know where they come from :) + ) + <youpi> (especially when it's the root fs...) + <antrik> ah, it's about a client request... didn't see that + <youpi> well, I just said "is called", yes + <antrik> I do not really understand though why it tries to reauthenticate + against itself... + <antrik> I fear my memory of the lookup mechanism grew a bit dim + <youpi> see the source + <youpi> it's about a translated entry + <antrik> (and I never fully understood some aspects anyways...) + <youpi> it needs to start the translated entry as another user, possibly + <antrik> yes, but a translated entry normally would be served by *another* + process?... + <youpi> sure, but ext2fs has to prepare it + <youpi> thus reauthenticate to prepare the correct set of rights + <antrik> prepare what? + <youpi> rights + <youpi> so the process is not root, doesn't have / opened as root, etc. + <antrik> rights for what? + <youpi> err, about everything + <antrik> IIRC the reauthentication is done by the parent FS on the port to + the *translated* node + <antrik> and the translated node should be a different process?... + <youpi> that's not what I read in the source + <youpi> fshelp_fetch_root + <youpi> ports[INIT_PORT_CRDIR] = reauth (getcrdir ()); + <youpi> here, getcrdir() returns ext2fs itself + <antrik> well, perhaps the issue is that I have no idea what + fshelp_fetch_root() does, nor why it is called here... + <youpi> it notably starts the translator that dir_lookup is looking at, if + needed + <youpi> possibly as a different user, thus reauthentication of CRDIR + <antrik> so this is about a port that is passed to the translator being + started? + <youpi> no + <youpi> well, depends on what you mean by "port" + <youpi> it's about reauthenticating a port to be passed to the translator + being started + <youpi> and for that a rendez-vous port is needed for the reauthentication + <youpi> and that's the one at stake + <antrik> yeah, I meant the port that is reauthenticated + <antrik> what is CRDIR? + <youpi> current root dir ... + <antrik> so the parent translator passes it's own root dir to the child + translator; and the issue is that for the root FS the root dir points to + the root FS itself... + <youpi> yes + <antrik> OK, that makes sense + <youpi> (but that's only one example, rgrep mach_port_destroy hurd/ show + other potential issues) + <antrik> well, that's actually what I wanted to mention next... why is the + rendez-vous port destroyed, instead of just deallocating the port right + and letting reference counting to it's thing?... + <antrik> do its thing + <youpi> "just to make sure" I guess + <antrik> it's pretty obvious that this will cause trouble for any RPC + referencing itself... + <youpi> well, follow-up with that on the list + <youpi> with roland/tb in CC + <youpi> only they would know any real reason for destroy + <youpi> btw, if you knew how we could make _hurd_select()'s raw __mach_msg + call be interruptible by signals, that'll permit to fix sudo + <youpi> (damn, I need sleep, my tenses are all wrong) + <antrik> BTW, does this cause any actual trouble?... + <antrik> I don't know much about interruption... cfhammer might have a + better idea, he look into that stuff quite a bit AIUI + <antrik> looked + <antrik> (hehe, it's not only your tenses... guess there's something in the + ether ;-) ) + <youpi> it makes sudo, mailq, etc. fail sometimes + <antrik> I mean the rendez-vous thing + <youpi> that's it, yes + <youpi> sudo etc. fail at least due to this + <antrik> so these are two different problems that both affect sudo? + <antrik> (rendez-vous and interruption I mean) + <youpi> yes + <youpi> with my patch the buildds have much fewer issues, but still some + <youpi> (my interrupt-related patch) + <youpi> I'm installing a s/destroy/deallocate/ version of ext2fs on the + buildds, we'll see how it behaves + <youpi> (it fixes my testcase at least) + <antrik> interrupt-related patch? + <antrik> only thing interrupt-related I remember was the reauthentication + race... + <youpi> that's what I mean + <antrik> well, cfhammer investigated this is quite some depth, explaining + quite well why the race is only mitigated but still exists... problem is + that we didn't know how to fix it properly + <antrik> because nobody seems to understand the cancellation code, except + perhaps for Roland and Thomas + <antrik> (and I'm not even entirely sure about them :-) ) + <antrik> I think his findings and our conclusions are documented on the + ML... + <youpi> by "much fewer issues", I mean that some of the symptoms have + disappeared, others haven't + <antrik> BTW, couldn't the rendez-vous thing be worked around by simply + ignoring the errors from the failing deallocate?... + <youpi> no, failing deallocate are actually dangerous + <antrik> why? + <youpi> since the name might have been reused for something else in the + meanwhile + <youpi> that's the whole point of the warning I had added in the kernel + itself + <antrik> I see + <youpi> such things really deserve tracking, since they can have any kind + of consequence + <antrik> does Mach try to reuse names quickly, rather than only after + wrapping around?... + <youpi> it seems to + <antrik> OK, then this is a serious problem indeed + <youpi> (note: I rarely divine issues when there aren't actual frequent + symptoms :) ) + <antrik> well, the problem with the warning is that it only shows in the + cases that do *not* cause a problem... so it's hard to associate them + with any specific issues + <youpi> well, most of the time the port is not reused quickly enough + <youpi> so in most case it shows up more often than causing problem + +IRC, freenode, #hurd, 2011-03-14 + + <youpi> ok, mach_port_deallocate actually can't be used + <youpi> since mach_reply_port() returns a receive right, not a send right + * youpi guesses he will really have to manage to understand all that port + stuff completely + <antrik> oh, right + <antrik> youpi: hm... now I'm confused though. if one client holds a + receive right, the other client (or in this case the same process) should + have a send or send-once right -- these should *not* share the same name + in my understanding + <antrik> destroying the receive right should turn the send right into a + dead name + <antrik> so unless I'm missing something, the destroy shouldn't be a + problem, and there must be something else going wrong + <antrik> hm... actually I'm probably wrong + <antrik> yeah, definitely wrong. receive rights and "ordinary" send rights + share the name. only send-once rights are special + <antrik> I wonder whether the problem could be worked around by using a + send-once right... + <antrik> mach_port_mod_refs(mach_task_self(), name, + MACH_PORT_RIGHT_RECEIVE, -1) can be used to deallocate only the receive + right + <antrik> oh, you already figured that out :-) diff --git a/open_issues/sync_but_still_unclean_filesystem.mdwn b/open_issues/sync_but_still_unclean_filesystem.mdwn index f1fbb4e0..83c7951e 100644 --- a/open_issues/sync_but_still_unclean_filesystem.mdwn +++ b/open_issues/sync_but_still_unclean_filesystem.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -10,9 +10,19 @@ License|/fdl]]."]]"""]] [[!tag open_issue_gnumach open_issue_hurd]] +Also filed as [[!GNU_Savannah_bug 29292]]. + \#hurd, 2010, end of May / beginning of June [runnign sync, but sill unclean filesystem at next boot] - <slpz> guillem: when libpager syncs an object, it sends an m_o_lock_request and waits (if the synchronous argument was specified) for a m_o_lock_completed. But m_o_lock_completed only means that dirty pages have been sent to the translator, and this one still needs to write them to the backing storage - <slpz> guillem: there's no problem if sync() returns before actually writting the changes to disk, but this also happens when shutting down the translator - <slpz> guillem: in theory, locking mechanisms in libpager should prevent this from happening by keeping track of write operations, but this seems to fail in some situations + <slpz> guillem: when libpager syncs an object, it sends an m_o_lock_request + and waits (if the synchronous argument was specified) for a + m_o_lock_completed. But m_o_lock_completed only means that dirty pages + have been sent to the translator, and this one still needs to write them + to the backing storage + <slpz> guillem: there's no problem if sync() returns before actually + writting the changes to disk, but this also happens when shutting down + the translator + <slpz> guillem: in theory, locking mechanisms in libpager should prevent + this from happening by keeping track of write operations, but this seems + to fail in some situations diff --git a/open_issues/unit_testing.mdwn b/open_issues/unit_testing.mdwn index d7821fd3..1378be85 100644 --- a/open_issues/unit_testing.mdwn +++ b/open_issues/unit_testing.mdwn @@ -8,6 +8,11 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] +This task may be suitable for [[community/GSoC]]: +[[community/gsoc/project_ideas/testing_framework]] + +--- + A collection of thoughts with respect to unit testing. We definitely want to add unit test suites to our code base. @@ -54,6 +59,8 @@ abandoned). Developers*](http://lwn.net/Articles/412302/) by Steven Rostedt, 2010-10-28. [v2](http://lwn.net/Articles/414064/), 2010-11-08. + * <http://autotest.kernel.org/wiki/WhitePaper> + # Related @@ -70,3 +77,10 @@ abandoned). * [LaBrea](https://github.com/dustin/labrea/wiki), or similar tools can be used for modelling certain aspects of system behavior (long response times, for example). + + +# Discussion + +See the [[GSoC project idea|community/gsoc/project_ideas/testing_framework]]'s +[[discussion +subpage|community/gsoc/project_ideas/testing_framework/discussion]]. diff --git a/sidebar.mdwn b/sidebar.mdwn index 501431e1..6dd2a35d 100644 --- a/sidebar.mdwn +++ b/sidebar.mdwn @@ -11,6 +11,17 @@ is included in the section entitled Welcome to... [[!img /logo/boxes-redrawn.png link=/logo]] ... the GNU Hurd! +--- + +[[!template id=highlight text="""**Breaking News** + +--- + +The **Google Summer of Code 2011** is on! If you're a student, consider +applying for a GNU Hurd project -- details to be found on our +*[[community/GSoC]] page*."""]] + +--- * **[[Home|/index]]** * **[[Community]]** * **[[Documentation]]** diff --git a/systemtap.mdwn b/systemtap.mdwn index abd1961e..ba64c0d4 100644 --- a/systemtap.mdwn +++ b/systemtap.mdwn @@ -19,7 +19,7 @@ License|/fdl]]."]]"""]] # Related - * [[open_issues/dtrace]] + * [[community/gsoc/project_ideas/dtrace]] * [[LTTng]] diff --git a/templates/highlight.mdwn b/templates/highlight.mdwn new file mode 100644 index 00000000..c8da1377 --- /dev/null +++ b/templates/highlight.mdwn @@ -0,0 +1,3 @@ +<div class="infobox"> +<TMPL_VAR text> +</div> diff --git a/topgit.mdwn b/topgit.mdwn index b71038ec..fb374337 100644 --- a/topgit.mdwn +++ b/topgit.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -30,7 +30,8 @@ development branches, for example [[source_repositories/binutils]] or # Running it on GNU/Hurd Nothing special to that, technically, *only* that our [[I/O system's (non-) -performance|open_issues/performance/io_system]] will render this unbearably +performance|community/gsoc/project_ideas/disk_io_performance]] will render this +unbearably slow for anything but simple test cases. So don't try to run it on the [[GCC]] or [[glibc]] repositories. Talk to [[tschwinge]] about how he's using it on a GNU/Linux machine and push the resulting trees to GNU/Hurd systems. |