diff options
Diffstat (limited to 'community/gsoc/project_ideas')
40 files changed, 1736 insertions, 211 deletions
diff --git a/community/gsoc/project_ideas/debian_installer.mdwn b/community/gsoc/project_ideas/debian_installer.mdwn index ca10a61e..37dcc72d 100644 --- a/community/gsoc/project_ideas/debian_installer.mdwn +++ b/community/gsoc/project_ideas/debian_installer.mdwn @@ -10,6 +10,11 @@ is included in the section entitled [[!meta title="Port the Debian Installer to the Hurd"]] +[!] Jérémie Koenig has been working on this as a [[Google Summer of Code +2010|2010]] project. + +--- + The primary means of distributing the Hurd is through Debian GNU/Hurd. However, the installation CDs presently use an ancient, non-native installer. The situation could be much improved by making sure that the newer *Debian @@ -21,6 +26,21 @@ Some preliminary work has been done, see The goal is to have the Debian Installer fully working on the Hurd. It requires relatively little Hurd-specific knowledge. +A lot of the "non-Linux support" part of the project has already been done thanks to a previous GSoC, so at least no ground reason should bar the project. A lot of the required udebs are already in Debian or are pending upload, so that building an image and booting it does already work. A preliminary list of what remains is + + * Add initrd support to GNU Mach, unless youpi does it before :) This should not be very complicated by re-using the iopl driver code. + * hurdify genext2fs to handle 4096 block size by default (see bug #562999) and support translator entries. + * Port busybox. This needs to be synchronized with kfreebsd people, who have probably already done some work, but that seemingly still hasn't been merged. In the meanwhile, youpi has a version with most of it disabled so a d-i image can actually be built. + * Port keyboard-setup to configure the xkb driver of the Hurd console + +As a starting point to get a grasp at how the debian installer is built, students might wish to look at the current Debian installer source and build process on Linux: + + * svn co svn://svn.debian.org/d-i/trunk/installer/ + * cd installer/build + * make build_monolithic + +The same can be done on hurd-i386 but a few non-uploaded packages are needed, see http://people.debian.org/~sthibault/hurd-i386/README-d-i + Possible mentors: Samuel Thibault (youpi) -Exercise: Try to get one piece of the installer running on Hurd. +Exercise: Fix a couple of Hurd issues in busybox. diff --git a/community/gsoc/project_ideas/disk_io_performance.mdwn b/community/gsoc/project_ideas/disk_io_performance.mdwn index bb047308..ae634709 100644 --- a/community/gsoc/project_ideas/disk_io_performance.mdwn +++ b/community/gsoc/project_ideas/disk_io_performance.mdwn @@ -1,34 +1,48 @@ -[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!meta title="Disk I/O Performance Tuning"]] +[[!tag open_issue_hurd]] + The most obvious reason for the Hurd feeling slow compared to mainstream -systems like GNU/Linux, is very slow harddisk access. +systems like GNU/Linux, is a low I/O system performance, in particular very +slow hard disk access. The reason for this slowness is lack and/or bad implementation of common -optimisation techniques, like scheduling reads and writes to minimalize head +optimization techniques, like scheduling reads and writes to minimize head movement; effective block caching; effective reads/writes to partial blocks; -reading/writing multiple blocks at once; and read-ahead. The +[[reading/writing multiple blocks at once|open_issues/performance/io_system/clustered_page_faults]]; and +[[open_issues/performance/io_system/read-ahead]]. The [[ext2_filesystem_server|hurd/translator/ext2fs]] might also need some -optimisations at a higher logical level. +optimizations at a higher logical level. The goal of this project is to analyze the current situation, and implement/fix -various optimisations, to achieve significantly better disk performance. It +various optimizations, to achieve significantly better disk performance. It requires understanding the data flow through the various layers involved in -disk acces on the Hurd ([[filesystem|hurd/virtual_file_system]], +disk access on the Hurd ([[filesystem|hurd/virtual_file_system]], [[pager|hurd/libpager]], driver), and general experience with -optimising complex systems. That said, the killing feature we are definitely -missing is the read-ahead, and even a very simple implementation would bring +optimizing complex systems. That said, the killing feature we are definitely +missing is the [[open_issues/performance/io_system/read-ahead]], and even a very simple implementation would bring very big performance speedups. +Here are some real testcases: + + * [[open_issues/performance/io_system/binutils_ld_64ksec]]; + + * running the Git testsuite which is mostly I/O bound; + + * use [[TopGit]] on a non-toy repository. + + Possible mentors: Samuel Thibault (youpi) Exercise: Look through all the code involved in disk I/O, and try something diff --git a/community/gsoc/project_ideas/download_backends.mdwn b/community/gsoc/project_ideas/download_backends.mdwn index 749597a6..f794e814 100644 --- a/community/gsoc/project_ideas/download_backends.mdwn +++ b/community/gsoc/project_ideas/download_backends.mdwn @@ -19,7 +19,7 @@ Download protocols like FTP, HTTP, BitTorrent etc. are very good candidates for this kind of modularization: a program could simply use the download functionality by accessing FTP, HTTP etc. translators. -There is already an ftpfs traslator in the Hurd tree, as well as an [httpfs +There is already an ftpfs translator in the Hurd tree, as well as an [httpfs translator on hurdextras](http://www.nongnu.org/hurdextras/#httpfs); however, these are only suitable for very simple use cases: they just provide the actual file contents downloaded from the URL, but no additional status information @@ -35,7 +35,7 @@ The goal of this project is to design a suitable interface, implement it for at least one download protocol, and adapt apt-get (or some other program) to use this as a backend. -This task requires some design skills and some knowlegde of internet protocols, +This task requires some design skills and some knowledge of internet protocols, to create a suitable interface. Translator programming knowledge will have to be obtained while implementing it. diff --git a/community/gsoc/project_ideas/driver_glue_code.mdwn b/community/gsoc/project_ideas/driver_glue_code.mdwn index 04efe202..8581c7cb 100644 --- a/community/gsoc/project_ideas/driver_glue_code.mdwn +++ b/community/gsoc/project_ideas/driver_glue_code.mdwn @@ -1,42 +1,73 @@ -[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -[[!meta title="New Driver Glue Code"]] +[[!meta title="New Driver Framework"]] -Although a driver framework in userspace would be desirable, presently the Hurd -uses kernel drivers in the microkernel, -[[GNU_Mach|microkernel/mach/gnumach]]. (And changing this would be far beyond a -GSoC project...) +[[!tag stable_URL]] -The problem is that the drivers in GNU Mach are presently old Linux drivers -(mostly from 2.0.x) accessed through a glue code layer. This is not an ideal -solution, but works quite OK, except that the drivers are very old. The goal of -this project is to redo the glue code, so we can use drivers from current Linux -versions, or from one of the free BSD variants. +The Hurd presently uses hardware drivers +implemented in the microkernel, [[GNU_Mach|microkernel/mach/gnumach]]. +These drivers are old Linux drivers (mostly from 2.0.x) +accessed through a glue code layer. +This is not an ideal solution, but works quite OK, +except that the drivers are extremely old by now. +Thus we need a new framework, +so we can use drivers from current Linux versions instead, +or perhaps from one of the free BSD variants. -While it would be certainly possible to create custom glue code again, a more -sustainable and probably also easier approch is to use -[ddekit](http://demo.tudos.org/dsweeper_tutorial.html) instead -- it already -does the hard work of providing an environment where the foreign drivers can -run, and has the additional advantage of being externally maintained. +This is [[!GNU_Savannah_task 5488]]. +[[open issues/user-space device drivers]]. +[[open issues/device drivers and io systems]]. -This is a doable, but pretty involved project. Previous experience with driver -programming probably is a must. (No Hurd-specific knowledge is required, -though.) +The most promising approach for getting newer drivers seems to be [[DDE]]: +it already does the hard work of providing an environment +where the foreign drivers can run, +and offers the additional benefit of being externally maintained. +DDE also offers the necessary facilities +for running all drivers in separate userspace processes, +which is more desirable than drivers running in the microkernel. -This is [[!GNU_Savannah_task 5488]]. +[[Zheng Da|zhengda]] has already done considerable work on this. +The basic framework for using DDE in the Hurd is present, +and network card drivers are already working very well. +However, this work isn't fully integrated in the Hurd yet. +The additional kernel interfaces that were created for this +are still prototypes, and will need to be reworked. +Also, there is no build system for automatically compiling +all Linux network card drivers in one go. + +Other types of drivers are missing so far. +Support for IDE drivers has been partially implemented, +but isn't fully working yet. +To fully replace the old in-kernel drivers, +further infrastructure will be necessary +to make userspace disk drivers usable for the root filesystem. + +Some other subsystems are missing or incomplete in DDE itself, +and will require additional work that is not specific to the Hurd implementation. + +The goal of this task is to fix at least one of the mentioned major shortcomings: +rework the kernel interfaces; +provide a streamlined build system for the drivers; +finish IDE support; +or implement support for some other subsystem. +<!-- should probably provide separate task descriptions for each... --> + +This is a doable, but pretty involved project. +Previous experience with driver programming probably is a must. +To be able to work on the framework, +the student will also have to get a good understanding of certain aspects of Hurd, +such as memory management for example. -Possible mentors: Samuel Thibault (youpi) +Possible mentors: Zheng Da, Samuel Thibault (youpi) -Exercise: Take a driver for some newer piece of hardware (e.g. Intel e1000 -ethernet) from a recent system, and try to port it to run in the existing -driver framework in GNU Mach. Completing the port might be too involved for the -exercise; but it's pretty likely that you will find something else to improve -in the glue code while working on this... +Exercise: Get one of the not yet integrated Linux network card drivers to work. +(Note: This should be straightforward, +once you have the framework properly built and set up...) diff --git a/community/gsoc/project_ideas/dtrace.mdwn b/community/gsoc/project_ideas/dtrace.mdwn index 93c2a5f3..6261c03e 100644 --- a/community/gsoc/project_ideas/dtrace.mdwn +++ b/community/gsoc/project_ideas/dtrace.mdwn @@ -1,22 +1,25 @@ -[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2011 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -[[!meta title="dtrace Support"]] +[[!meta title="Kernel Instrumentation"]] + +[[!tag open_issue_gnumach]] One of the main problems of the current Hurd implementation is very poor -performance. While we have a bunch of ideas what could cause the performance +[[open_issues/performance]]. While we have a bunch of ideas what could cause the performance problems, these are mostly just guesses. Better understanding what really causes bad performance is necessary to improve the situation. For that, we need tools for performance measurements. While all kinds of more -or less specific profiling tools could be convieved, the most promising and +or less specific [[open_issues/profiling]] tools could be conceived, the most promising and generic approach seems to be a framework for logging certain events in the running system (both in the microkernel and in the Hurd servers). This would allow checking how much time is spent in certain modules, how often certain @@ -24,23 +27,36 @@ situations occur, how things interact, etc. It could also prove helpful in debugging some issues that are otherwise hard to find because of complex interactions. -The most popular framework for that is Sun's dtrace; but there might be others. -The student has to evaluate the existing options, deciding which makes most -sense for the Hurd; and implement that one. (Apple's implementation of dtrace -in their Mach-based kernel might be helpful here...) - -This project requires ability to evaluate possible solutions, and experience -with integrating existing components as well as low-level programming. +The most popular kernel instrumentation framework is Sun's dtrace, +originally written for Solaris, +but also adopted by some other systems. +However, the GPL-incompatible license means it can't be used in Linux, +and thus Linux developers created their own frameworks instead: +first [[SystemTap]], and now [[LTTng]]. + +In 2008, Andrei Barbu did initial work on kernel probes for the Hurd. +However, not all of his patches got merged, +because some turned out not to be fully functional. +Also, he didn't get around to work on userspace probes, +nor on a nice frontend for writing test scripts employing the probes. + +The goal of this project is to make the instrumentation framework +more usable and complete, +and to better integrate it in the Hurd. +For that, the student will have to work +on some real profiling and/or debugging tasks, +and fix any shortcomings he encounters in the framework. + +This is a pretty involved task. +Previous experience with low-level programming is a must; +and it also requires a good grasp on interactions in complex systems. + +To work on this project, +the student will have to get familiar with GNU Mach. +(The microkernel employed by the Hurd.) +Some understanding of other aspects of the Hurd will also be required, +depending on the exact nature of the profiling/debugging performed. Possible mentors: Samuel Thibault (youpi) -Exercise: In lack of a good exercise directly related to this taks, just pick -one of the kernel-related or generally low-level tasks from the bug/task -trackers on savannah, and make a go at it. You might not be able to finish the -task in a limited amount of time, but you should at least be able to make a -detailed analysis of the issue. - -*Status*: Andei Barbu was working on -[SystemTap](http://csclub.uwaterloo.ca/~abarbu/hurd/) for GSoC 2008, but it -turned out too Linux-specific. He implemented kernel probes, but there is no -nice frontend yet. +Exercise: Use the existing probes to perform some simple measurement. diff --git a/community/gsoc/project_ideas/file_locking.mdwn b/community/gsoc/project_ideas/file_locking.mdwn index b6b393f9..206d4d7d 100644 --- a/community/gsoc/project_ideas/file_locking.mdwn +++ b/community/gsoc/project_ideas/file_locking.mdwn @@ -1,27 +1,31 @@ -[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2012 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -[[!meta title="Fix File Locking"]] +[[!meta title="Fix and Complete File Locking Support"]] -Over the years, [[UNIX]] has aquired a host of different file locking mechanisms. +Over the years, [[UNIX]] has acquired a host of different file locking mechanisms. Some of them work on the Hurd, while others are buggy or only partially implemented. This breaks many applications. The goal is to make all file locking mechanisms work properly. This requires finding all existing shortcomings (through systematic testing and/or checking for known issues in the bug tracker and mailing list archives), and fixing -them. +them. The biggest missing feature is record locking, i.e. the lockf variant, +which needs a complete implementation. This task will require digging into parts of the code to understand how file locking works on the Hurd. Only general programming skills are required. +A preliminary patch is [[!GNU_Savannah_patch 332 desc="available"]]. + Possible mentors: Samuel Thibault (youpi) Exercise: Find one of the existing issues, either by looking at the task/bug diff --git a/community/gsoc/project_ideas/gcc_asan.mdwn b/community/gsoc/project_ideas/gcc_asan.mdwn new file mode 100644 index 00000000..229c46ec --- /dev/null +++ b/community/gsoc/project_ideas/gcc_asan.mdwn @@ -0,0 +1,21 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Port GCC's AddressSanitizer to the Hurd"]] + +[[!tag open_issue_gcc]] + +See the entry on the [[open_issues/code_analysis]] page. + +See also the [[valgrind]] task. + +A follow-up project is porting GCC's ThreadSanitizer. + +Possible mentors: Thomas Schwinge (tschwinge) diff --git a/community/gsoc/project_ideas/gccgo.mdwn b/community/gsoc/project_ideas/gccgo.mdwn new file mode 100644 index 00000000..54b20754 --- /dev/null +++ b/community/gsoc/project_ideas/gccgo.mdwn @@ -0,0 +1,34 @@ +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Porting Google Go (GCC: gccgo)"]] + +The goal of this project is to make the [Google Go programming +language](http://golang.org/) available on GNU/Hurd in its [[GCC]] *gccgo* +implementation. + +Presumably less work will be needed on the language's frontend itself, but +rather on the supporting libraries. + +Apart from a solid knowledge of the [[POSIX]] API, working knowledge of the +Google Go programming language is a must. Some Hurd knowledge will have to be +acquired while working on the project. + +Designing and implementing [[language_bindings]] is a follow-up project. + +Possible mentors: Ian Lance Taylor: gccgo bits, [[Thomas Schwinge +(tschwinge)|tschwinge]]: Hurd bits. + +Exercise: Fix one of the problems preventing *gccgo* from working on the Hurd. + +--- + +[[Open Issue page|open_issues/gccgo]]. [Entry in the GCC +wiki](http://gcc.gnu.org/wiki/SummerOfCode#gccgo_hurd). diff --git a/community/gsoc/project_ideas/gnat.mdwn b/community/gsoc/project_ideas/gnat.mdwn index b7f2ea62..ba34cc9c 100644 --- a/community/gsoc/project_ideas/gnat.mdwn +++ b/community/gsoc/project_ideas/gnat.mdwn @@ -1,24 +1,32 @@ -[[!meta copyright="Copyright © 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2009, 2011, 2012 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -[[!meta title="Porting GNAT"]] +[[!meta title="Porting GNAT (GCC)"]] -The GNU Ada Translator (GNAT) isn't available for the Hurd so far. There are -also a number of other Debian packages depending on GNAT, and thus not -buildable on the Hurd. +An initial port of the GNU Ada Translator (GNAT) is available for the Hurd. The goal of this project is getting GNAT fully working in Debian GNU/Hurd. It -requires implementing some explicitely system-specific stuff in GNAT, and maybe -fixing a few other problems. Good knowledge of Ada is a must; some Hurd -knowledge will have to be aquired while working on the project. +requires implementing some explicitly system-specific stuff in GNAT (mostly in +its runtime libraries), and for that also address a number of issues in Hurd +and other libraries. Knowledge of Ada is a must; some Hurd +knowledge will have to be acquired while working on the project. -Possible mentors: Samuel Thibault (youpi) +Designing and implementing [[language_bindings]] is a follow-up project. -Exercise: Fix one of the problems preventing GNAT from working on the Hurd. +Possible mentors: [[Samuel Thibault (youpi)|samuelthibault]], [[Thomas Schwinge +(tschwinge)|tschwinge]]. + +Exercise: Fix one of the known issues of GNAT on the Hurd. + +--- + +[[Open Issue page|open_issues/gnat]]. [Entry in the GCC +wiki](http://gcc.gnu.org/wiki/SummerOfCode#gnat_hurd). diff --git a/community/gsoc/project_ideas/gnumach_cleanup.mdwn b/community/gsoc/project_ideas/gnumach_cleanup.mdwn index e75c9d3e..4aef0d1b 100644 --- a/community/gsoc/project_ideas/gnumach_cleanup.mdwn +++ b/community/gsoc/project_ideas/gnumach_cleanup.mdwn @@ -11,7 +11,7 @@ is included in the section entitled [[!meta title="GNU Mach Code Cleanup"]] Although there are some attempts to move to a more modern microkernel -alltogether, the current Hurd implementation is based on +altogether, the current Hurd implementation is based on [[GNU_Mach|microkernel/mach/gnumach]], which is only a slightly modified variant of the original CMU [[microkernel/Mach]]. @@ -19,7 +19,7 @@ Unfortunately, Mach was created about two decades ago, and is in turn based on even older BSD code. Parts of the BSD kernel -- file systems, [[UNIX]] [[mechanism]]s like processes and signals, etc. -- were ripped out (to be implemented in [[userspace_servers|hurd/translator]] instead); while other mechanisms were -added to allow implementing stuff in userspace. +added to allow implementing stuff in user space. ([[Pager_interface|microkernel/mach/external_pager_mechanism]], [[microkernel/mach/IPC]], etc.) diff --git a/community/gsoc/project_ideas/language_bindings.mdwn b/community/gsoc/project_ideas/language_bindings.mdwn index bf75c805..d9a426be 100644 --- a/community/gsoc/project_ideas/language_bindings.mdwn +++ b/community/gsoc/project_ideas/language_bindings.mdwn @@ -1,15 +1,19 @@ -[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011, 2012 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!meta title="Bindings to Other Programming Languages"]] +<!-- See also open_issues/gccgo, open_issues/gnat, open_issues/perl, and +open_issues/python. --> + The main idea of the Hurd design is giving users the ability to easily modify/extend the system's functionality ([[extensible_system|extensibility]]). This is done by creating [[filesystem_translators|hurd/translator]] and other @@ -19,7 +23,7 @@ However, in practice this is not as easy as it should, because creating translators and other servers is quite involved -- the interfaces for doing that are not exactly simple, and available only for C programs. Being able to easily create simple translators in RAD languages is highly desirable, to -really be able to reap the advantages of the Hurd architecture. +really be able to reap the [[advantages]] of the Hurd architecture. Originally Lisp was meant to be the second system language besides C in the GNU system; but that doesn't mean we are bound to Lisp. Bindings for any popular @@ -30,20 +34,24 @@ Several approaches are possible when creating such bindings. One way is simply to provide wrappers to all the available C libraries ([[hurd/libtrivfs]], [[hurd/libnetfs]] etc.). While this is easy (it requires relatively little consideration), it may not be the optimal solution. It is preferable to hook in at a lower level, thus -being able te create interfaces that are specially adapted to make good use of +being able to create interfaces that are specially adapted to make good use of the features available in the respective language. -These more specialised bindings could hook in at some of the lower level +These more specialized bindings could hook in at some of the lower level library interfaces ([[hurd/libports]], [[hurd/glibc]], etc.); use the [[microkernel/mach/MIG]]-provided [[microkernel/mach/RPC]] stubs directly; or even create native stubs directly from the interface definitions. The -[[lisp_bindings_created_by_Flavio_Cruz|flaviocruz]] in last year's GSoC mostly -use the latter approach, and can serve as a good example. +[[lisp_bindings_created_by_Flavio_Cruz|flaviocruz]] as his [[2008 GSoC +project|2008]] mostly use the latter approach, and can serve as a good example. +In his [[2011 GSoC project|2011]], Jérémie Koenig designed and began +implementing an object-oriented interface; see his [[Java status +page|user/jkoenig/java]] for details. There is another possible reason for preferring lower-level bindings: Presently, the Hurd server libraries use the cthreads threading library, which -predates the pthread standard prevalent today. There is a pthread library for -the Hurd as well, but it's not possible to use both cthreads and pthreads in +predates the pthread standard prevalent today. There is a +[[pthread library for the Hurd|libpthread]] +as well, but it's not possible to use both cthreads and pthreads in the same executable. Thus, until [[porting_the_Hurd_libraries_to_pthreads|community/gsoc/project_ideas/pthreads]] is finished, implementing bindings for any language that uses pthreads (in the @@ -64,4 +72,4 @@ There was also some previous work on [Perl bindings](http://www.nongnu.org/hurdextras/#pith), which might serve as a reference if you want to work on Perl. -Possible mentors: Anatoly A. Kazantsev (jim-crow) for Python +Possible mentors: Anatoly A. Kazantsev (anatoly) for Python diff --git a/community/gsoc/project_ideas/lexical_dot-dot.mdwn b/community/gsoc/project_ideas/lexical_dot-dot.mdwn index f0b8db7c..e0dabc01 100644 --- a/community/gsoc/project_ideas/lexical_dot-dot.mdwn +++ b/community/gsoc/project_ideas/lexical_dot-dot.mdwn @@ -22,7 +22,7 @@ resolution. Some application already use lexical resolution internally for that reason. It is generally agreed that many problems could be avoided if the standard filesystem lookup calls used lexical resolution as well. The compatibility -problems probably would be negligable. +problems probably would be negligible. The goal of this project is to modify the filename lookup mechanism in the Hurd to use lexical resolution, and to check that the system is still fully @@ -31,7 +31,7 @@ mechanism. See also [[!GNU_Savannah_bug 17133]]. -Possible mentors: ? +Possible mentors: Carl Fredrik Hammar (cfhammar) Exercise: This project requires changes to the name lookup mechanism in the Hurd-related glibc parts, as well as the Hurd servers. Thus, the exercise task diff --git a/community/gsoc/project_ideas/libcap.mdwn b/community/gsoc/project_ideas/libcap.mdwn index 10ca508e..18c49c48 100644 --- a/community/gsoc/project_ideas/libcap.mdwn +++ b/community/gsoc/project_ideas/libcap.mdwn @@ -1,12 +1,12 @@ -[[!meta copyright="Copyright © 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2009, 2010 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!meta title="Implementing libcap"]] @@ -14,7 +14,7 @@ libcap is a library providing the API to access POSIX capabilities. These allow giving various kinds of specific privileges to individual users, without giving them full root permissions. -Although the Hurd design should faciliate implementing such features in a quite +Although the Hurd design should facilitate implementing such features in a quite natural fashion, there is no support for POSIX capabilities yet. As a consequence, libcap is not available on the Hurd, and thus various packages using it can not be easily built in Debian GNU/Hurd. @@ -31,6 +31,10 @@ Some knowledge of POSIX capabilities will need to be obtained, and for the latter part also some knowledge about the Hurd architecture. This project is probably doable without previous experience with either, though. +David Hedberg applied for this project in 2010, +and though he didn't go through with it, +he fleshed out many [[details]]. + Possible mentors: Samuel Thibault (youpi) Exercise: Make libcap compile on Debian GNU/Hurd. It doesn't need to actually diff --git a/community/gsoc/project_ideas/libcap/details.mdwn b/community/gsoc/project_ideas/libcap/details.mdwn new file mode 100644 index 00000000..85695978 --- /dev/null +++ b/community/gsoc/project_ideas/libcap/details.mdwn @@ -0,0 +1,766 @@ +[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Details on implementing libcap"]] + + +This is the proposal submitted by David Hedberg for GSoC 2010 (who opted +to go with another mentoring organization), adapted from an initial +proposal and several amendments into a single document, but the idea +is to further adapt it to a more neutral design document over time. + + +# The proposal + +### Quick description of POSIX capabilities + +The original suggestion can be found [[here|libcap]]. +POSIX capabilities never actually entered the POSIX standard but was +left as a draft. Linux has nevertheless implemented this draft, and +there are reasons for doing the same in the Hurd - a more fine grained +control of rights leading to increased security being one of them. + +POSIX capabilities are give on a per-process basis, and basically allows +splitting up those rights usually granted to root into smaller and more +specific rights. Examples of capabilities are CAP_CHOWN and CAP_KILL, +overriding certain restrictions on chown and allowing the process to +kill processes with different UID's, respectively. + +Each process is given three sets with capabilities: the Permitted set +(P), the Effective set (E) and the Inheritable set (I). The effective +set contains the capabilities that are currently active. The permitted +set contains the capabilities that the process has the right to use. +The inheritable set contains the capabilities that can be inherited +by children to the process. A process can drop capabilities from its +permitted set, but not set them. The effective set and the inheritable +set can be changed freely as long as they stay subsets of the permitted +set. + +Capabilities can also be set on executables. When executed, the resulting +process is given the capabilities both as defined by the parent process +and by the capabilities set on the file (formula below), resulting in +what might be explained as a fine-grained setuid. Implementing this +requires support for *xattr* or similar. + +Some applications that are currently using capabilities are samba, ntp, +vsftp, pure-ftpd, proftpd, squid, asterisk and dovecot. + + +### A quick description of capabilities in Linux + +Each process has a three bit fields representing each of the three +sets (P, E and I). Each bit field is currently built up of two (32 +bit) integers to be able to hold the 33 currently defined capabilities +(see linux/capability.h). Each process further has a bounding set which +bounds the permitted set. Two [[system call]]s handles the setting and getting +of capabilities; *capset* and *capget*. Some related functionality +can also be controlled by calling *prctl*: the right to read/drop the +bounding capabilities (PR_CAPBSET_READ/PR_CAPBSET_DROP) and whether +or not the process should keep its capabilities when a change is made +to the threads UID's (PR_SET_KEEPCAPS/PR_GET_KEEPCAPS). User space +applications are expected(recommended?) to use libcap to take advantage +of the functionality provided. Some applications also use libcap-ng +which is "intended to make programming with POSIX capabilities much +easier than the traditional libcap library". Most applications seem +to be using the original libcap, however. + + +## Implementing libcap + +The exercise for this assignment was to get the libcap used in +Linux to compile under the Hurd. This was accomplished using the +latest git version of libcap from (..), corresponding to libcap +2.19. The changes were simple and amounted to simply removing the +dependency on some Linux-specific headers and creating stubs for +capset, capget and prctl described above. This suggests that porting +this library to the Hurd once the proper functionality is in place +could be relatively simple. The patch is available +[here](https://alioth.debian.org/tracker/index.php?func=detail&aid=312442&group_id=30628&atid=410472 "Alioth"). +One could also consider implementing the three missing functions in the +Hurd (or Hurd glibc) which would allow the usage of the Linux libcap +without modifications. As the Linux libcap maintainer might or might +not be interested in making libcap multi platform, this approach might +be preferable. + + +## Implementing POSIX capabilities in the Hurd + +As I am still reading up on how things fit together in the Hurd this is +very likely to contain some misunderstandings and be at least partly off +the mark. I intend to have grasped this by the time I start working on +it however, if I were to be given the chance. Below are two possible +approaches as I have understood them after some reading and discussions +on #hurd@freenode. + + +### The basics, Approach 1: Special UID's + +Let each capability be represented by a specific UID. One could imagine +reserving a range of the possible uid_t's for this purpose. The euids +vector in the authhandle struct could perhaps hold the effective set while +the auids vector could hold the permitted set as these seem to roughly +correspond to eachother in intent. This leaves the Inheritable set. +One solution could be to store the inheritable set in the agids vector, +but that does not seem to be a very natural nor nice solution. One could +extend the authhandle struct with an additional vector, but one would then +need to also extend the auth interface with RPC's to be able to modify +and access it. Another possibility is to store all the capabilities +in the same idvec and use separate defines for the the different sets +(CAP_P_CHMOD, CAP_E_CHMOD, CAP_I_CHMOD). This does not seem like a +good solution. + +Storing each capability in its own uid_t might also arguably be somewhat +wasteful, although this is probably of secondary concern (if at all). +One could also imagine that legacy applications might be confused, +although I am not sure I can see any obvious problems. What happens +when a process have only capability id's? + + +### The basics, Approach 2: Extend the auth interface + +This approach would expand the auth interface and extend +the auth server with another set of RPC's for capabilities +(*auth_getcaps*, *auth_makecaps*, *auth_user_authenticate* and +*auth_server_authenticate*), mirroring those currently declared for id's. +This would obviously require changes to more parts of the Hurd for +processes to be able to take advantage of capabilities, but as the logic +behind handling capabilities and that behind handling user id's might +not be completely comparable, this might make for a cleaner solution. +It would also remove the problem of having to sensibly map all the +three capability sets onto semantically differing sets of user/group +ids, something that might be even more important if we were to also +implement something like the bounding sets used in Linux or perhaps +other related functionality. We are also not limited to having to store +the capabilities as id vectors, although doing so would perhaps still +make sense. + + +### The individual capabilities + +Implementing the individual capabilities will probably have to be thought +of on a case-by-case basis. Taking chown (in *libdiskfs*) as an example, +the exact approach would differ slightly depending on how the approach +taken to implement capabilities. If the first approach were to be taken, +the UID's (and thus the capabilities) of the caller would already be +available to the function through the *iouser* struct contained in the +*protid* struct given as the first argument. Implementing capabilities +would then simply be a matter of checking for the special UID's. If the +second approach were to be taken, one would need to perhaps expand the +iouser struct to contain information about the capabilities. + +Just like Linux has defined many Linux-specific capabilities - some of +which could certainly be useful also applied to the Hurd - one could +also imagine extending the POSIX capability system also for Hurd specific +purposes. + + +## Some applications using capabilities + +#### Samba 3 + +Uses CAP_MKNOD and CAP_LEASE in smbd to only keep the necessary abilities. +Documentation mentions CAP_LINUX_IMMUTABLE as a way to protect files +from being deleted. Can also use a couple of IRIX specific capabilities +(CAP_NETWORK_MGT and CAP_DEVICE_MGT) as alternatives to the Linux-specific +ones if running on IRIX. + + +#### ntpd + +Checks if capabilities are supported, more precisely if CAP_SYS_TIME, +CAP_SETUID, CAP_SETGID, CAP_SYS_CHROOT and CAP_NET_BIND_SERVICE are +supported. If they are supported, it uses prctl with PR_SET_KEEPCAPS +to keep privileges across setuid() and then drops root. After done with +CAP_SETUID, CAP_SETGID, CAP_SYS_CHROOT it drops every capability except +CAP_SYS_TIME and, if needed, CAP_NET_BIND_SERVICE. + + +#### vsftpd + +Uses CAP_CHOWN, CAP_NET_BIND_SERVICE when using the "one process" +security model (typically disabled by default). + + +#### proftpd-basic + +Provides support for capabilities from mod_cap. Uses CAP_USE_CHOWN, +CAP_USE_DAC_OVERRIDE, CAP_USE_DAC_READ_SEARCH and CAP_USE_AUDIT_WRITE. +Distribution contains README.capabilities with some explanations. +Also ships with their own libcap for some reason, based on libcap 1.10. + + +#### dovecot + +Keeps CAP_CHOWN, CAP_SYS_CHROOT, CAP_SETUID, CAP_SETGID, +CAP_NET_BIND_SERVICE, CAP_DAC_OVERRIDE for proper operations, drops +the rest. + + +#### bind9 + +Reasons for each capability are clearly noted in comments in update.c +in linux_initialprivs() and linux_minprivs(). initialprivs drops all +capabilities and proceeds to set CAP_NET_BIND_SERVICE, CAP_SYS_CHROOT, +CAP_SETUID, CAP_SETGID, CAP_DAC_READ_SEARCH and CAP_CHOWN. minprivs only +sets CAP_NET_BIND_SERVICE and CAP_SYS_RESOURCE. + + +#### pulseaudio + +Mentions CAP_NICE (CAP_SYS_NICE), but does not appear to be using it +(anymore?). Seems to use libcap to drop caps, however. + + +#### pinentry + +Checks if CAP_IPC_LOCK is available and "uses it" to gain only the +ability to lock memory when needed. + + +#### zsh + +Comes with a module "caps" which contains "[b]uiltins for manipulating +POSIX.1e (POSIX.6) capability (privilege) sets." Most useful here is the +"cap" builtin, which makes it possible to change the shell's process +capability sets. This might be useful for testing. + + +#### inetutils (ping,traceroute) + +Does not use capabilities explicitly, but is nevertheless a useful +example of how file capabilities could be used. ping and traceroute +are currently installed suid root since they need to be able to open +raw sockets. With file capabilities, this could be accomplished by +instead setting the capability CAP_NET_RAW on the two executables, +thus giving the utilities almost only the specific rights they need. + + +## The capabilities + +The above might give some hint as to what capabilities should be +prioritized. One assumption I have made is that the goal of this project +is to implement, as far as possible, the same functionality as what is +present in Linux. No effort has (so far) been made to look into possible +applications specific to the Hurd. + +A few of the above mentioned applications also explicitly uses +PR_SET_KEEPCAPS (through prctl()) to specify that capabilities should +be passed on to children. This means that the implementation in the +Hurd should take this into account. + +I have below done a preliminary classification of the capabilities +as defined in Linux capability.h into four "classes": Network, File +management, "glibc -> mach" and Other. There are also some capabilities +that either depend on functionality not implemented or are too Linux +specific. I have not described each capability in detail as looking +at the comments in capability.h and reading in capabilities(7) are +better sources. + + +### The Networking Class + +These would mostly affect pfinet. The general picture seem to be that +pfinet currently uses a boolean (int) isroot in struct sock_user to keep +track of the credentials of the caller. This would need to be expanded +somehow to keep track of the separate capabilities. + +CAP_NET_BIND_SERVICE: Allow binding to TCP/UDP sockets below 1024 +CAP_NET_RAW: Allow use of RAW and PACKET sockets. +CAP_NET_BROADCAST: "Allow broadcasting, listen to multicast" +CAP_NET_ADMIN: This seem to be a bit of a "catch-all" for network-related +administration. + + +### The Files Management Class + +The description of CAP_CHOWN in the original proposal should apply to +(most of?) these. That is, modify the iouser struct. At least libdiskfs +should be modified, but the same or similar modifications might need to +be made to several servers (libnetfs..? The actual servers implementing +the filesystem?) + +CAP_CHOWN: "Make arbitrary changes to file UIDs and GIDs" +CAP_FOWNER: allow chmod, utime, .. for files not owned. +CAP_FSETID: Don't clear setuid/setgid when a file is modified. +CAP_DAC_OVERRIDE and +CAP_DAC_READ_SEARCH: Bypasses file/directory read/write/execute permission +checks. ( hurdexec.c, file-access.c, .. ? ) +CAP_MKNOD: allows usage of "the privileged aspects of mknod()". Does this +one make sense in the Hurd? + + +### The (glibc -> gnumach) Class + +These seem to be implemented in glibc by direct calls to gnumach. +If they should be implemented, maybe a proxy in the Hurd is needed? + +CAP_SYS_TIME: manipulate the system clock, set real-time clock. +CAP_IPC_LOCK: mlock, mlockall, mmap, shmctl +CAP_KILL: No permission checks for killing processes +CAP_SYS_NICE: setpriority/getpriority for arbitrary processes. + + +### The Rest Class + +CAP_SYS_CHROOT: Allows usage of chroot(). +It's either really simple (not needed in the case of the Hurd) or really +difficult. Needs some figuring out. One of the calls that should be +high-priority. +CAP_SYS_ADMIN: General administration rights. Seemingly sprinkled out +into just about everything. Quick grep through the Linux sources gives +440 hits in .c-files. +CAP_SYS_BOOT: Allow use of reboot(). +glibc calls init:startup_reboot.. +CAP_SETGID: Allows usage of setgid,setgroups and "forged gids on socket +credentials passing" +CAP_SETUID: Allows usage of set*uid and "forged pids on socket credentials +passing" +CAP_SYS_TTY_CONFIG: vhangup, some other places. vhangup() is a stub in +the Hurd, but maybe the console server is otherwise affected? +CAP_SYS_RESOURCE: Override various limits. (quota, reserved space etc. +on ext2, interrupt frequencies, consoles,...). According to "The Critique" +mach performs no resource accounting so some of these might be moot to +implement, while others still apply. +CAP_SYS_RAWIO Allow raw input/output. Sprinkled in many places, +device drivers among others. Many of these will probably be difficult +to implement. +CAP_SETPCAP: This one has (or had?) two different usages in Linux: +If file capabilities are not supported it gives the right to grant +or remove capabilities from the permitted set of arbitrary processes. +If file capabilities are supported, it allows for removing capabilities +from the bounding set of the current process. As the Hurd implementation +won't have file capabilities initially it might make sense to implement +this if possible. If bounding sets are implemented this should probably +be the way provided to modify them. + + +### Unimplementable + +*(At this point in time, as far as I can determine)* + +CAP_LINUX_IMMUTABLE: depends on chattr etc. working. +CAP_SETFCAP: depends on xattr's +CAP_SYS_PACCT: acct() missing in the Hurd. +CAP_SYS_MODULE, CAP_SYS_PTRACE, CAP_MAC_OVERRIDE, CAP_MAC_ADMIN, +CAP_AUDIT_WRITE, CAP_AUDIT_CONTROL, CAP_LEASE + + +## Priority when implementing + +The two most used capabilities as unscientifically judged from +the selection of applications above are CAP_NET_BIND_SERVICE and +CAP_CHOWN, suggesting that implementing the "network class" and the +"file management" class of capabilities as classified above might be a +good start. These also, to me, seem to be easier classes to implement. +CAP_NET_ADMIN might need some extra work. + +Second most common were CAP_SYS_CHROOT, CAP_SETGID and CAP_SETUID. I am +not completely clear on how these should be handled. + +Assuming those are out of the way, CAP_IPC_LOCK, CAP_SYS_TIME, CAP_KILL +and CAP_SYS_NICE might be a good choice to tackle if possible. This +might, if I have understood things correctly, involve writing a proxy +Hurd server for these calls in mach. + +CAP_SYS_ADMIN, CAP_SYS_RESOURCE and CAP_SYS_RAWIO. These contains a bit +of "everything" (ADMIN being the worse one), meaning that experience +and infrastructure gained from implementing the previous capabilities +might come in handy. CAP_SYS_RAWIO might be difficult; it can be found +inside many drivers in the Linux source. + + +## Additional general details + +[This article](http://www.ibm.com/developerworks/library/l-posixcap.html) +contains a good general description of how capabilities in Linux +works. As there will be no file capabilities in the Hurd initially, +an approach emulating the behavior Linux exhibits when SECURE_NOROOT +and SECURE_NO_SETUID_FIXUP are *not* set seems to be a good start. +This is called the "root-user-is-privileged" model in the article, +and somewhat simplified it means that processes started by root, or +setuid-root, is given all capabilities no matter what capabilities the +parent might or might not hold at the time of execution. Quoting verbatim +from the article: + +> * When SECURE_NOROOT is not set, then when a process executes a file, +> the new capability sets may be calculated as though the file had some +> file capability sets set fully populated. In particular: +> +> * The file inheritable and permitted sets will be full on if the +> process's real or effective uid is 0 (root) or the file is setuid +> root. +> +> * The file effective set will be full on if the process's effective +> uid is root or the file is setuid root. +> +> +> * When SECURE_NO_SETUID_FIXUP is not set, then when a process switches +> its real or effective uids to or from 0, capability sets are further +> shifted around: +> +> * If a process switches its effective uid from 0 to non-0, then its +> effective capability set is cleared. +> +> * If a process switches its real, effective, or saved uids from at +> least one being 0 to all being non-zero, then both the permitted +> and effective capabilities are cleared. +> +> * If a process sets its effective uid from non-zero to 0, then the +> effective capabilities are set equal to the permitted capabilities. + +The capabilities of the resulting process are determined by the following +formulas, again taken from the article, with p for Process and f for file: + +> pI' = pI +> pP' = fP | (fI & pI) +> pE' = pP' & fE + +The security under the above described model, being what at least some +of the applications I listed in my last comment employs, is basically +the following (also detailed somewhat in the same article): + +* Execute process as root (or setuid) to gain all capabilities. + +* Use the prctl [[system call]] to enable keepcaps for the process + (same(?) effect as enabling SECURE_NO_SETUID_FIXUP for the process). + keepcaps should be off by default. + +* setuid to a non-root user, and by doing so losing the possibility to + regain capabilities by simply starting a new process. + +* Drop all the capabilities except those few actually needed. + + +## Infrastructure details - Special UIDs approach + +The auth server must somehow keep track of three sets of capabilities. +I suggest keeping these three sets in three additional idvec's in the +authhandle struct, and will for the purpose of this description name +these pcaps (permitted), ecaps (effective) and icaps (inheritable). +This will simplify keeping track of the internal logic somewhat. +In addition to this, there is a need to keep track of the "keepcaps" flag +as described above. I suggest representing this with an int keepcaps +in the same struct. + +1. Expand authhandle struct with three additional idvecs and one integer. +Fix static functions handling the struct, such as destroy_authhandle. + +2. Implement the necessary logic in auth_makeauth to handle capabilities. + +Problems: +Assume that all capabilities are mapped onto uids, given names on the form +uid_<capability>, for example uid_cap_net_raw. Assume that the presence +of an uid in euids suggest that the capability is in the effective set +of the process, that the presence of this uid in auids suggests that it +is in the permitted set of the process, and that the presence of this +uid in aguids suggest that it is in the inheritable set of the process. +That they are internally stored in separate idvec's can be ignored as +an implementation detail. + +* The UID's have as it is different meanings depending on where in the + array they are positioned, and certain clients seem to rely on this. + The first UID in euids is the effective uid, and the first and second + UIDs in auids are the real and saved UIDS respectively. At least + some users of makeauth would need to made aware of capabilities, + for example setuid in glibc. + +* Setting/getting the keepcaps-flag is also a bit of a problem. To avoid + changing the auth interface yet another special UID could be used + for this purpose, although that seems to be really stretching it. + The cleaner solution would probably be to expand the interface with + something along the lines of auth_setkeepcaps/auth_getkeepcaps. + This interface would only be used by prctl. + +Another problem with this approach is that it seems a bit difficult +to oversee the affects that using other RPC's like fsys_getroot and +io_restrict_auth might have on capabilities. + + +## Infrastructure details - "extend-interfaces" approach + +This approach has started to seem like the better way to me, as the +usage of capabilities becomes more explicit through the entire "chain", +perhaps making it somewhat more easy to understand all the interactions. + +I suggest something like the following new interface methods: + +*** + + +### The auth interface + + routine auth_getauth_caps ( + handle: auth_t; + out euids: idarray_t; + out auids: idarray_t; + out egids: idarray_t; + out agids: idarray_t; + out ecaps: idarray_t; + out pcaps: idarray_t; + out icaps: idarray_t); + + routine auth_makeauth_caps ( + handle: auth_t; + other_handles: portarray_t; + euids: idarray_t; + auids: idarray_t; + egids: idarray_t; + agids: idarray_t; + ecaps: idarray_t; + pcaps: idarray_t; + icaps: idarray_t; + flags: int; /* keepcaps.. ? */ + out newhandle: mach_port_make_send_t); + + routine auth_server_authenticate_caps ( + handle: auth_t; + sreplyport reply: mach_port_poly_t; + rendezvous: mach_port_send_t; + newport: mach_port_poly_t; + out euids: idarray_t; + out auids: idarray_t; + out egids: idarray_t; + out agids: idarray_t; + out ecaps: idarray_t; + out pcaps: idarray_t; + out icaps: idarray_t); + + +### The io interface + + routine io_restrict_auth_caps ( + io_object: io_t; + RPT + out new_object: mach_port_send_t; + uids: idarray_t SCP; + gids: idarray_t SCP; + ecaps: idarray_t SCP); + + +### The fsys interface + + routine fsys_getroot_caps ( + fsys: fsys_t; + RPT + #ifdef FSYS_GETROOT_UREPLY + ureplyport ureply: mig_reply_port_t; + #endif + dotdot_node: mach_port_send_t; + gen_uids: idarray_t; + gen_gids: idarray_t; + out ecaps: idarray_t; + out pcaps: idarray_t; + out icaps: idarray_t; + flags: int; + out do_retry: retry_type; + out retry_name: string_t; + out file: mach_port_send_t); + +*** + +These are meant to be able to replace the old methods with +capability-aware methods, instead of merely complementing them. +The replacing work could then be made a more gradual process. Steps: + +* Extend authhandle with the same data members as in the UID-case. + +* Implement new _caps-functions according to described interface + extensions above, refactor code a bit to share common uid-handling + logic. Both makeauth's should drop all capabilities if switching from + uid 0 without having keepcaps. For example, keepcaps should be unset + by default. + +* Fix glibc. Extend hurd_id_data in hurd/id.h to store capabilities, + switch to capability aware functions where necessary. + +* io-reauthenticate. Fix implementations to use + auth_server_authenticate_caps instead. For this we also need somewhere + to save the caps, so it ties in with for example the extension of + iouser as mentioned in the details. + +* fsys_getroot. Implement fsys_getroot_caps in libdiskfs, trans, + libtreefs, libtrivs, libnetfs. Fix users of function in libdiskfs, + libfshelp, settrans, libtreefs, clookup. + +* io_restrict_auth. Implement io_restrict_auth_caps in libdiskfs, + libtreefs, libtrivfs, libnetfs, boot. Fix users in utils(symlink, + firmlink), libtrivs, term, clookup + +Among the problems might be that there are a lot of arguments that +needs to be passed around, and that it seems somewhat ugly to end up +with function names referencing caps in particular. + +Below some more about the specific capabilities. This should in large +be common to the two approaches above. + + +## Actually handing out the capabilities to process + +This seems like a good job for the file_exec route in the fs interface. +Quoting from the comments above the definition in fs.defs: "*Necessary +initialization, including authentication changes associated with set[ug]id +execution must be handled by the filesystem*". The capability-granting +functionality should to be added in at least the implementations in +libdiskfs and libnetfs as far as I can determine, and should be "easy +enough" once the infrastructure for implementing the file-related +capabilities (CAP_CHOWN etc.) are in place. This also seem to make +sense considering the future possibility for file capabilities. + + +## Some implementation details of individual capabilities. + +### Net-related capabilities. + +This turned out to be a bit less work than I had thought, as the +imported Linux code already seem to contain all the necessary checks. +What remains to do to implement all of these capabilities is mostly a +matter of putting some infrastructure in place. + +* In struct sock_user (pfinet.h), change isroot for idvec + caps. Alternatively, add idvec caps. + +* Change the function make_sock_user in socket.c to take an idvec caps + as a parameter and properly set the given caps to the corresponding + idvec in sock_user. + +* Fix users of make_sock_user: S_io_reauthenticate, S_io_restrict_auth, + S_socket_create, S_socket_accept. This should be doable with the + current infrastructure. For example, S_socket_create currently + sets isroot in the new sock_user from the corresponding member in + the trivfs_protid struct. This does not present a problem however, + as the latter struct also provides access to an iouser (iohelp.h) + from which the needed uids of the user should be available. + +* Fix up parts of source from Linux, modify task_struct add idvec, + modify prepare_current to take the caps idvec instead of isroot. + Re-implement the existing function capable(int cap) to actually check + for the capability passed as an argument instead of just return isroot. + +* Change a few isroot's at 3 other places in the code to check for + capabilities. Since these places all have access to isroot and thus by + implication the sock_user, they also have access to the new capability + information; no restructuring necessary. + + +### File-related capabilities + +While there are a lot of servers in the Hurd, I am not sure all of these +capabilities actually make sense to implement in all of them. + + +#### CAP_CHOWN + +Implementing this in libdiskfs should take care of it where it makes +sense. Servers using libdiskfs uses iouser from libiohelp to hold +user credentials. As noted above, this struct is already capable of +holding our capabilities as uid's or is otherwise extended to contain +the necessary idvecs if using the second general approach. Adding a +check along the lines of *idvec_contains(uid_cap_chown)* in function +diskfs_S_file_chown (file-chown.c) should be all that's needed. + +In libnetfs, netfs_attempt_chown is declared as a function that the +server using the library must implement. Any checks for chown rights +are however most likely performed on the server side, suggesting that +there is nothing we can do here to implement CAP_CHOWN. Even if we do +need to add something, an iouser containing the necessary information +to implement the checks in a manner analogous to that in libdiskfs seems +to be passed to each important function. + + +#### CAP_DAC_* + +These might actually make sense to implement in more servers, and the +logic seems somewhat involved. Need to add the necessary checks to +at least file_check_access, file_exec in libdiskfs. file_exec also in +libnetfs, probably. Possibly changes also in other places. + +The main difficulties overall seem to lie in getting the infrastructure +properly in place rather than implementing most of the individual +capabilities, and I have updated the schedule a bit in an attempt to +reflect this. + + +## Schedule updating + +The more I look into this the less time it seems likely to take to +do the work. Below is my best estimation at the moment, and I have +basically adjusted everything to what I think is more likely estimations. +If this is correct I would be more or less around midterm. I haven't +gone completely to the individual level as that doesn't seem to make +sense, but what is clustered together are either related capabilities +or a collection of capabilities classified roughly with regards to how +much I know about the area and how many different rights they control. +It's not my intention to "slack off" or anything, so if this estimation +were to be correct I could perhaps take a look at the xattrs-patch, +or spend the rest of my time fixing PATH_MAX-issues. Then again, maybe +there is some great difficulty hidden somewhere. + + +#### Some justifications: + +Dummy libcap, more or less done. +*1 day* (making sure it "fails gracefully" shouldn't really take longer than this) + +Application for testing, the beginnings of a fairly extensive "test suit" on Linux. +*2 days* + +Basic infrastructure. +*5 days*, depends on the chosen approach, but it is probably wise to +reserve at least a bunch of days for this. + +Implementations of prctl/capset/capget in libshouldbeinlibc, +or a port of libcap to the Hurd in any other way. +*5 days* + +CAP_NET_BIND_SERVICE, CAP_NET_RAW, CAP_NET_BROADCAST, CAP_NET_ADMIN +*4 days*, as noted above this should be easy, but it might uncover bugs +in the newly implemented infrastructure for example. + +CAP_CHOWN,CAP_FOWNER,CAP_FSETID +*2 days*, I think these only needs to be done in libdiskfs + +CAP_DAC_OVERRIDE,CAP_DAC_READ_SEARCH +*4 days*, these might need changes to various servers + +CAP_SYS_TIME,CAP_IPC_LOCK,CAP_KILL +CAP_SYS_NICE,CAP_SYS_CHROOT,CAP_SYS_BOOT +*2 weeks*, these are varied and I'm not that sure exactly how each should +be tackled so some research is needed. + +CAP_SETGID,CAP_SETUID,CAP_SYS_TTY_CONFIG +*4 days* + +CAP_SYS_ADMIN,CAP_SYS_RESOURCE,CAP_SYS_RAWIO +*2 weeks*, these too are pretty varied and some might need some individual +researching + +CAP_SETPCAP +*1 day* + + +## Schedule + +24/5 Start coding +25/5 Dummy libcap ready for use. +27/5 The beginnings of a "test suite", written on Linux. + 1/6 Basic infrastructure in place + 6/6 Dummy libcap extended with real functionality to make use of + implemented capability and infrastructure, or the Hurd adapted for + compatibility with Linux libcap. +10/6 The "network class" of capabilities implemented. +12/6 CAP_CHOWN, CAP_FOWNER, CAP_FSETID +16/6 CAP_DAC_OVERRIDE, CAP_DAC_READ_SEARCH +30/6 CAP_SYS_TIME, CAP_IPC_LOCK, CAP_KILL, CAP_SYS_NICE, + CAP_SYS_CHROOT, CAP_SYS_BOOT + 4/7 CAP_SETGID,CAP_SETUID,CAP_SYS_TTY_CONFIG +12/7 "Mentors and students can begin submitting mid-term evaluations" +16/7 GSoC Mid-term evaluations deadline. +18/7 CAP_SYS_ADMIN, CAP_SYS_RESOURCE, CAP_SYS_RAWIO +19/7 CAP_SETPCAP diff --git a/community/gsoc/project_ideas/libdiskfs_locking.mdwn b/community/gsoc/project_ideas/libdiskfs_locking.mdwn index 2a08b387..faac8bd9 100644 --- a/community/gsoc/project_ideas/libdiskfs_locking.mdwn +++ b/community/gsoc/project_ideas/libdiskfs_locking.mdwn @@ -1,15 +1,18 @@ -[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!meta title="Fix libdiskfs Locking Issues"]] +[[!tag open_issue_hurd]] + Nowadays the most often encountered cause of Hurd crashes seems to be lockups in the [[hurd/translator/ext2fs]] server. One of these could be traced recently, and turned out to be a lock inside [[hurd/libdiskfs]] that was taken @@ -18,17 +21,21 @@ faulty paths causing these lockups. The task is systematically checking the [[hurd/libdiskfs]] code for this kind of locking issues. To achieve this, some kind of test harness has to be implemented: For -exmple instrumenting the code to check locking correctness constantly at -runtime. Or implementing a unit testing framework that explicitely checks +example instrumenting the code to check locking correctness constantly at +runtime. Or implementing a [[open_issues/unit_testing]] framework that explicitly checks locking in various code paths. (The latter could serve as a template for -implementing unit checks in other parts of the Hurd codebase...) +implementing unit tests in other parts of the Hurd codebase...) -(A systematic code review would probably suffice to find the existing locking +(A [[systematic code review|open_issues/security]] would probably suffice to find the +existing locking issues; but it wouldn't document the work in terms of actual code produced, and thus it's not suitable for a GSoC project...) -This task requires experience with debugging locking issues in multithreaded -applications. +This task requires experience with debugging locking issues in +[[multithreaded|open_issues/multithreading]] applications. + +Tools have been written for automated [[open_issues/code_analysis]]; these can help to +locate and fix such errors. Possible mentors: Samuel Thibault (youpi) diff --git a/community/gsoc/project_ideas/libgtop.mdwn b/community/gsoc/project_ideas/libgtop.mdwn index ef1b2bcb..41897a1f 100644 --- a/community/gsoc/project_ideas/libgtop.mdwn +++ b/community/gsoc/project_ideas/libgtop.mdwn @@ -1,4 +1,5 @@ -[[!meta copyright="Copyright © 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2009, 2010, 2012 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -10,12 +11,16 @@ is included in the section entitled [[!meta title="Porting libgtop"]] +/!\ On 2012-05-05 Andjos reported (commit +web.git:8061106f2d1f15fa9a54947bc45d4cba68d89bba) that this task has already +been completed. + libgtop is a library used by many applications (especially GNOME applications) to abstract the system-specific methods for obtaining information about the current state of the system -- processes running, system load etc. -A [[Linux-compatible_procfs|madhusudancs]] implementation has been created -during GSoC 2008, and should cover a large part of the functionality of +A Linux-compatible [[hurd/translator/procfs]] is available +and should cover a large part of the functionality of libgtop. However, not all necessary information is exported via /proc (even on Linux); there are some bits still missing in the Hurd procfs implementation; and there are a couple of bugs that need to be fixed to make it fully usable. @@ -24,9 +29,9 @@ The goal of this project is a fully functional libgtop in Debian GNU/Hurd. Some application(s) using it also need to be ported, e.g. gnome-system-monitor. Some bits of this work are easy, others need some digging into Hurd internals. -This task doesn't require any specific previous knowlegde (besides of general +This task doesn't require any specific previous knowledge (besides of general C/UNIX programming skills of course); but during the course of the project, -some knowlegde about Hurd internals will have to be obtained, along with a bit +some knowledge about Hurd internals will have to be obtained, along with a bit of Debian stuff. Possible mentors: Samuel Thibault (youpi) diff --git a/community/gsoc/project_ideas/maxpath.mdwn b/community/gsoc/project_ideas/maxpath.mdwn index 5be8917f..4a1314c2 100644 --- a/community/gsoc/project_ideas/maxpath.mdwn +++ b/community/gsoc/project_ideas/maxpath.mdwn @@ -13,7 +13,7 @@ is included in the section entitled POSIX describes some constants (or rather macros) like PATH_MAX/MAXPATHLEN and similar, which may be defined by the system to indicate certain limits. Many people overlook the *may* though: Systems only should define them if they -actually have such fixed limits. The Hurd, following the GNU Coding Standards, +actually have such fixed limits (see [limits.h](http://pubs.opengroup.org/onlinepubs/009695399/basedefs/limits.h.html)). The Hurd, following the GNU Coding Standards, tries to avoid this kind of arbitrary limits, and consequently doesn't define the macros. diff --git a/community/gsoc/project_ideas/mtab.mdwn b/community/gsoc/project_ideas/mtab.mdwn index 045533e6..a60a8038 100644 --- a/community/gsoc/project_ideas/mtab.mdwn +++ b/community/gsoc/project_ideas/mtab.mdwn @@ -22,7 +22,7 @@ is no central place keeping track of mounts. As a consequence, there is currently no easy way to obtain a listing of all mounted file systems. This also means that commands like `df` can only work on -explicitely specified mountpoints, instead of displaying the usual listing. +explicitly specified mountpoints, instead of displaying the usual listing. One possible solution to this would be for the translator startup mechanism to update the `mtab` on any `mount`/`unmount`, like in traditional systems. @@ -31,7 +31,7 @@ with passive translators, i.e., translators that are not presently running, but set up to be started automatically whenever the node is accessed? Probably these should be counted among the mounted filesystems; but how to handle the `mtab` updates for a translator that is not started yet? Generally, being -centralized and event-based, this is a pretty unelegant, non-hurdish solution. +centralized and event-based, this is a pretty inelegant, non-hurdish solution. A more promising approach is to have `mtab` exported by a special translator, which gathers the necessary information on demand. This could work by @@ -64,10 +64,10 @@ other filtering rules might be desirable. After taking decisions on the outstanding design questions, the student will implement both the actual [[mtab_translator|hurd/translator/mtabfs]], and the -necessery interface(s) for gathering the data. It requires getting a good +necessary interface(s) for gathering the data. It requires getting a good understanding of the translator mechanism and Hurd interfaces in general. -Possible mentors: Olaf Buddenhagen (antrik) +Possible mentors: Olaf Buddenhagen (antrik), Carl Fredrik Hammar (cfhammar) Exercise: Make some improvement to any of the existing Hurd translators. Especially those in [hurdextras](http://www.nongnu.org/hurdextras/) are often diff --git a/community/gsoc/project_ideas/namespace-based_translator_selection.mdwn b/community/gsoc/project_ideas/namespace-based_translator_selection.mdwn index d40d73ac..f668b6f2 100644 --- a/community/gsoc/project_ideas/namespace-based_translator_selection.mdwn +++ b/community/gsoc/project_ideas/namespace-based_translator_selection.mdwn @@ -1,14 +1,21 @@ -[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2012 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -[[!meta title="Namspace-based Translator Selection"]] +[[!meta title="Namespace-based Translator Selection"]] + +[!] [[Sergiu Ivanov|scolobb]] has been working *voluntarily* on this task an +inofficial GSoC 2008 participant. Not all the desired functionality is in +place yet, though. + +--- The main idea behind the Hurd is to make (almost) all system functionality user-modifiable ([[extensible_system|extensibility]]). This includes a @@ -26,7 +33,7 @@ file, and presents a virtual file with the uncompressed contents. Or the other way around. Or a translator that presents an [[XML_file_as_a_directory_tree|hurd/translator/xmlfs]]. Or an mbox as a set of individual files for each mail ([[hurd/translator/mboxfs]]); or ever further -breaking it down into headers, body, attachements... +breaking it down into headers, body, attachments... This gets even more powerful when translators are used as building blocks for larger applications: A mail reader for example doesn't need backends for @@ -40,7 +47,7 @@ There are a few problems with the way translators are set, though. For one, once a translator is set on a node, you always see the translated content. If you need the untranslated contents again, to do a backup for example, you first need to remove the translator again. Also, having to set a translator -explicitely before accessing the contents is pretty cumbersome, making this +explicitly before accessing the contents is pretty cumbersome, making this feature almost useless. A possible solution is implementing a mechanism for selecting translators @@ -61,7 +68,7 @@ place of the original ones. In the long run it's probably desirable to have the mechanism implemented in the standard name lookup mechanism, so it will be available globally, and avoid -the overhead of a proxy; but for the beginnig the proxy solution is much more +the overhead of a proxy; but for the beginning the proxy solution is much more flexible. The goal of this project is implementing a prototype proxy; perhaps also a @@ -75,8 +82,3 @@ Possible mentors: Olaf Buddenhagen (antrik) Exercise: Try to make some modification to the existing unionfs and/or firmlink translators. (More specific suggestions welcome... :-) ) - -*Status*: Sergiu Ivanov has been working *voluntarily* on -[[namespace-based_translator_selection|scolobb]], as an inofficial GSoC 2008 -participant! Not all the desired functionality is in place yet; work is -ongoing. diff --git a/community/gsoc/project_ideas/namespace-based_translator_selection/discussion.mdwn b/community/gsoc/project_ideas/namespace-based_translator_selection/discussion.mdwn new file mode 100644 index 00000000..befd680a --- /dev/null +++ b/community/gsoc/project_ideas/namespace-based_translator_selection/discussion.mdwn @@ -0,0 +1,64 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + + +# IRC, freenode, #hurd, 2012-04-22 + + <youpi> btw, I was wondering, when working on namespace mangling, did they + think about automatitioning ? + <youpi> autopartitioning, I meant + <youpi> i.e. with a foo.img file, open foo.img,,part1 + <braunr> what are you referring to with namespace mangling + <youpi> and voila + <youpi> I don't remember the exact term they used + <braunr> you mean there is a hurd library that parses names and can direct + to different services depending on part of the name ? + <youpi> namespace-based_translator_selection + <youpi> yes + <braunr> i thought it only handled directories + <braunr> well, the classical path representation + * civodul finds it ugly + <youpi> civodul: because of potential conflict, and the not-too-nice ",," + part? + <youpi> actually I wonder whether using directory access would be nicer + <youpi> i.e. you have a foo.gz, just open foo.gz/gunzip to get the unzipped + content + <youpi> and for foo.img.gz, open foo.img.gz/gunzip/part/1 + <civodul> youpi: because of the interpretation of special chars in file + names + <civodul> users should be free to use any character they like in file names + <civodul> foo.gz/gunzip looks nicer to me + <youpi> ok, so we agree + <youpi> that said, the user could choose the separator + <youpi> the namespace can be not run by root for everybody, but just for + your shell, run by yourself + <antrik> civodul: the user can't use any character anyways... '/' and '\0' + are reserved :-P + <civodul> antrik: '/' isn't quite reserved on the Hurd :-) + <civodul> you could implement dir_lookup such that it does something + special about it + <civodul> (server-side) + <antrik> civodul: as for overloading '/', although I haven't thought it + through entirely, I guess that would work for nodes that present as files + normally. however, it would *not* work for directory nodes + <antrik> which would be quite a serious limitation IMHO + <antrik> I can think of various kinds of useful directory translators + <antrik> what's more, one of the main use cases I originally had in mind is + a policy filter + <antrik> you could pass a directory name with a appropriate filter applied + to tar for example, so it wouldn't try to follow any translators + <antrik> I don't see why taking an obscure prefix like ,, would be much of + a problem in practice anyways + <antrik> (also, it doesn't strictly prevent the user from having such file + names... you just need to escape it if accessing such files through the + namespace multiplexer. though admittedly that would need some special + handling in *some* programs to work properly) diff --git a/community/gsoc/project_ideas/nfs.mdwn b/community/gsoc/project_ideas/nfs.mdwn index 683287f7..d4980279 100644 --- a/community/gsoc/project_ideas/nfs.mdwn +++ b/community/gsoc/project_ideas/nfs.mdwn @@ -1,22 +1,23 @@ -[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2012 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!meta title="Improved NFS Implementation"]] The Hurd has both NFS server and client implementations, which work, but not -very well: File locking doesn't work properly (at least in conjuction with a +very well: File locking doesn't work properly (at least in conjunction with a GNU/Linux server), and performance is extremely poor. Part of the problems could be owed to the fact that only NFSv2 is supported so far. (Note though that locking on the Hurd is problematic in general, not only in -conjuction with NFS -- see the [[file_locking]] task.) +conjunction with NFS -- see the [[file_locking]] task.) This project encompasses implementing NFSv3 support, fixing bugs and performance problems -- the goal is to have good NFS support. The work done in @@ -31,6 +32,9 @@ has been done for a former GSoC application -- it might give you some pointers. But don't take any of the statements made there for granted -- check the facts yourself! +A bigger subtask is the [[libnetfs: `io_map`|open_issues/libnetfs_io_map]] +issue. + This task, [[!GNU_Savannah_task 5497]], has no special prerequisites besides general programming skills, and an interest in file systems and network protocols. diff --git a/community/gsoc/project_ideas/package_manager.mdwn b/community/gsoc/project_ideas/package_manager.mdwn index 43e53f7c..23304f6b 100644 --- a/community/gsoc/project_ideas/package_manager.mdwn +++ b/community/gsoc/project_ideas/package_manager.mdwn @@ -37,7 +37,7 @@ came about. There are no global databases of any kind. (Some things might require caching for better performance, but this must happen transparently.) The core of this approach is formed by [[hurd/translator/stowfs]], which -creates a traditional unix directory structure from all the files in the +creates a traditional Unix directory structure from all the files in the individual package directories. But this only handles the lowest level of package management. Additional mechanisms are necessary to handle stuff like dependencies on other packages. diff --git a/community/gsoc/project_ideas/perl.mdwn b/community/gsoc/project_ideas/perl.mdwn deleted file mode 100644 index bfe1968e..00000000 --- a/community/gsoc/project_ideas/perl.mdwn +++ /dev/null @@ -1,30 +0,0 @@ -[[!meta copyright="Copyright © 2009 Free Software Foundation, Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] - -[[!meta title="Improving Perl Support"]] - -Perl is available on the Hurd, but there are quite a lot of test suite -failures. These could be caused by problems in the system-specific -implementation bits of Perl, and/or shortcomings in the actual system -functionality which Perl depends on. - -The goal of this project is to fix all of these problems if possible, or at -least some of them. Some issues might require digging quite deep into Hurd -internals, while others are probably easy to fix. - -Note that while some Perl knowledge is probably necessary to understand what -the test suite failures are about, the actual work necessary to fix these -issues is mostly C programming -- in the implementation of Perl and/or the -Hurd. - -Possible mentors: Samuel Thibault (youpi) - -Exercise: Make some improvement to Perl support on the Hurd, e.g. fixing one of -the known test suite failures. diff --git a/community/gsoc/project_ideas/perl_python.mdwn b/community/gsoc/project_ideas/perl_python.mdwn new file mode 100644 index 00000000..a51393f1 --- /dev/null +++ b/community/gsoc/project_ideas/perl_python.mdwn @@ -0,0 +1,41 @@ +[[!meta copyright="Copyright © 2009, 2010, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Improving Perl or Python Support"]] + +<!-- See also open_issues/perl and open_issues/python. --> + +Perl and Python are available on the Hurd, but there are quite a lot of test suite +failures. These could be caused by problems in the system-specific +implementation bits of Perl/Python, and/or shortcomings in the actual system +functionality which Perl/Python depends on. + +The student applying for this project can pick either Perl or Python, +whichever he is more comfortable with. +(Perl is higher priority though; and there are more failures too.) + +The goal then is to fix all of the problems with the chosen language if possible, or at +least some of them. Some issues might require digging quite deep into Hurd +internals, while others are probably easy to fix. + +Note that while some Perl/Python knowledge is probably necessary to understand what +the test suite failures are about, the actual work necessary to fix these +issues is mostly C programming -- in the implementation of Perl/Python and/or the +Hurd. + +Possible mentors: Samuel Thibault (youpi) + +Exercise: Take a stab at one of the testsuite failures, +and write a minimal testcase exposing the underlying problem. +Actually fixing it would be a bonus of course -- +but as it's hard to predict which issues will be easy and which will be tricky, +we will already be satisfied if the student makes a good effort. +(We hope to see some discussion of the problems in this case though :-) ) diff --git a/community/gsoc/project_ideas/procfs.mdwn b/community/gsoc/project_ideas/procfs.mdwn index 85eec43c..ac5fa6d8 100644 --- a/community/gsoc/project_ideas/procfs.mdwn +++ b/community/gsoc/project_ideas/procfs.mdwn @@ -1,15 +1,25 @@ -[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2011, 2012 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!meta title="procfs"]] +[!] Madhusudan.C.S has implemented a new, fully functional +[[procfs|madhusudancs]] as a [[GSoC 2008 project|2008]]. + +[!] This was not the end of the story: [[jkoenig's +`procfs`|hurd/translator/procfs]] is yet another re-written and +improved version. + +--- + Although there is no standard (POSIX or other) for the layout of the `/proc` pseudo-filesystem, it turned out a very useful facility in GNU/Linux and other systems, and many tools concerned with process management use it. (`ps`, `top`, @@ -33,13 +43,10 @@ interfaces.) This project requires learning [[hurd/translator]] programming, and understanding some of the internals of process management in the Hurd. It should not be too hard coding-wise; and the task is very nicely defined by the -exising Linux `/proc` interface -- no design considerations necessary. +existing Linux `/proc` interface -- no design considerations necessary. **Note**: We already have several applications for this task. Possible mentors: Olaf Buddenhagen (antrik) Exercise: Add or fix one piece in the existing procfs translator. - -*Status*: Madhusudan.C.S has implemented a new, fully functional [[procfs|madhusudancs]] for -GSoC 2008. He is still working on some outstanding issues. diff --git a/community/gsoc/project_ideas/pthreads.mdwn b/community/gsoc/project_ideas/pthreads.mdwn index 9c703d36..2270c774 100644 --- a/community/gsoc/project_ideas/pthreads.mdwn +++ b/community/gsoc/project_ideas/pthreads.mdwn @@ -1,4 +1,5 @@ -[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -10,15 +11,18 @@ is included in the section entitled [[!meta title="Convert Hurd Libraries and Servers to pthreads"]] +[[!tag open_issue_libpthread]] + The Hurd was originally created at a time when the [pthreads standard](http://www.opengroup.org/onlinepubs/009695399/basedefs/pthread.h.html) didn't exist yet. Thus all Hurd servers and libraries are using the old [[cthreads|hurd/libcthreads]] package that came with [[microkernel/Mach]], -which is not compatible with [[pthreads|hurd/libpthread]]. +which is not compatible with pthreads. Not only does that mean that people hacking on Hurd internals have to deal with a non-standard thread package, which nobody is familiar with. Although a -pthreads implementation for the Hurd was created in the meantime, it's not +[[pthreads implementation for the Hurd|libpthread]] +was created in the meantime, it's not possible to use both cthreads and pthreads in the same program. Consequently, pthreads can't presently be used in any Hurd servers -- including translators. diff --git a/community/gsoc/project_ideas/secure_chroot.mdwn b/community/gsoc/project_ideas/secure_chroot.mdwn index a433e8d1..bfaf330b 100644 --- a/community/gsoc/project_ideas/secure_chroot.mdwn +++ b/community/gsoc/project_ideas/secure_chroot.mdwn @@ -1,26 +1,27 @@ -[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!meta title="Secure chroot Implementation"]] As the Hurd attempts to be (almost) fully [[UNIX]]-compatible, it also implements a -`chroot()` system call. However, the current implementation is not really +`chroot()` [[system call]]. However, the current implementation is not really good, as it allows easily escaping the `chroot`, for example by use of [[passive_translators|hurd/translator]]. Many solutions have been suggested for this problem -- ranging from simple -workaround changing the behaviour of passive translators in a `chroot`; -changing the context in which passive translators are exectuted; changing the +workaround changing the behavior of passive translators in a `chroot`; +changing the context in which passive translators are executed; changing the interpretation of filenames in a chroot; to reworking the whole passive -translator mechanism. Some involving a completely different approch to -`chroot` implementation, using a proxy instead of a special system call in the +translator mechanism. Some involving a completely different approach to +`chroot` implementation, using a proxy instead of a special [[system call]] in the filesystem servers. See <http://tri-ceps.blogspot.com/2007/07/theory-of-filesystem-relativity.html> @@ -38,7 +39,7 @@ new mechanisms. (Translators.) More important than the actual code is the documentation of what he did: he must be able to defend why he chose a certain approach, and explain why he believes this approach really secure. -Possible mentors: ? +Possible mentors: Carl Fredrik Hammar (cfhammar) Exercise: It's hard to come up with a relevant exercise, as there are so many possible solutions... Probably best to make an improvement to one of the diff --git a/community/gsoc/project_ideas/server_overriding.mdwn b/community/gsoc/project_ideas/server_overriding.mdwn index 42edf287..c35d88de 100644 --- a/community/gsoc/project_ideas/server_overriding.mdwn +++ b/community/gsoc/project_ideas/server_overriding.mdwn @@ -14,7 +14,7 @@ The main idea of the Hurd is that every user can influence almost all system functionality ([[extensible_system|extensibility]]), by running private Hurd servers that replace or proxy the global default implementations. -However, running such a cumstomized subenvironment presently is not easy, +However, running such a customized subenvironment presently is not easy, because there is no standard mechanism to easily replace an individual standard server, keeping everything else. (Presently there is only the [[hurd/subhurd]] method, which creates a completely new system instance with a completely @@ -40,7 +40,7 @@ mechanism would have to check an override table on each lookup, and apply the desired replacement whenever a match is found. Another approach would be server-side overrides. Again there are various -variants. The actual servers themself could provide a mechanism to redirect to +variants. The actual servers themselves could provide a mechanism to redirect to other servers on request. (3) Or we could use some more generic server-side namespace overrides: Either all filesystem servers could provide a mechanism to modify the namespace they export to certain clients (4), or proxies could be @@ -68,7 +68,7 @@ discussing this topic, from a previous year's GSoC application -- see <http://lists.gnu.org/archive/html/bug-hurd/2007-06/msg00082.html>, <http://lists.gnu.org/archive/html/bug-hurd/2008-03/msg00039.html>. -Possible mentors: Olaf Buddenhagen (antrik) +Possible mentors: Olaf Buddenhagen (antrik), Carl Fredrik Hammar (cfhammar) Exercise: Come up with a glibc patch that allows overriding one specific standard server using method (1). diff --git a/community/gsoc/project_ideas/smp.mdwn b/community/gsoc/project_ideas/smp.mdwn new file mode 100644 index 00000000..e17c2ccf --- /dev/null +++ b/community/gsoc/project_ideas/smp.mdwn @@ -0,0 +1,16 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="SMP"]] + + +# IRC, freenode, #hurd, 2012-09-30 + + <braunr> i expect smp to be our next gsoc project diff --git a/community/gsoc/project_ideas/sound.mdwn b/community/gsoc/project_ideas/sound.mdwn index b92f76da..8411831b 100644 --- a/community/gsoc/project_ideas/sound.mdwn +++ b/community/gsoc/project_ideas/sound.mdwn @@ -36,7 +36,7 @@ Possible mentors: Samuel Thibault (youpi) Exercise: This project requires kernel (driver framework) hacking as well as some Hurd server hacking; so the exercise should involve either of these, or even both. You could for example port some newer driver to run in the existing -framework (see the [[device_driver|driver_glue_code]] project descrption), or +framework (see the [[device_driver|driver_glue_code]] project description), or try to make some fix(es) to the [unfinished random device implementation](http://savannah.gnu.org/patch/?6088) created by Michael Casadevall. diff --git a/community/gsoc/project_ideas/tcp_ip_stack.mdwn b/community/gsoc/project_ideas/tcp_ip_stack.mdwn index b56bff51..331336ac 100644 --- a/community/gsoc/project_ideas/tcp_ip_stack.mdwn +++ b/community/gsoc/project_ideas/tcp_ip_stack.mdwn @@ -21,7 +21,7 @@ more flexibly. Rather than just having the standard socket interface, plus some lower-level hooks for special needs, there are explicit (perhaps filesystem-based) interfaces at all the individual levels; special application can just directly access the desired layer. All kinds of packet filtering, -routing, tunneling etc. can be easily achieved by stacking compononts in the +routing, tunneling etc. can be easily achieved by stacking components in the desired constellation. Implementing a complete modular network stack is not feasible as a GSoC @@ -34,7 +34,7 @@ make such a split later on feasible. This is [[!GNU_Savannah_task 5469]]. -Possible mentors: ? +Possible mentors: zhengda Exercise: You could try making some improvement to the existing pfinet implementation; or you could work towards running some existing userspace diff --git a/community/gsoc/project_ideas/testing_framework.mdwn b/community/gsoc/project_ideas/testing_framework.mdwn new file mode 100644 index 00000000..ff9899d9 --- /dev/null +++ b/community/gsoc/project_ideas/testing_framework.mdwn @@ -0,0 +1,55 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled +[[GNU Free Documentation License|/fdl]]."]]"""]] + +[[!meta title="Automated Testing Framework"]] + +Hurd development would benefit greatly from automated tests. +Unit tests should be added for all the major components +(Mach; Hurd servers and libraries). +Also, functional tests can be performed on various levels: +Mach; individual servers; and interaction of several servers. + +(The highest level would actually be testing libc functionality, +which in turn uses the various Hurd mechanisms. +glibc already comes with a test suite -- +though it might be desirabe to add some extra tests +for exercising specific aspects of the Hurd...) + +Our page on [[automated testing|open_issues/unit_testing]] collects some relevant material. + +The Goal of this task is to provide testing frameworks +that allow automatically running tests +as part of the Hurd and Mach build processes. +The student will have to create the necessary infrastrucure, +and a couple of sample tests employing it. +Ideally, all the aspects mentioned above should be covered. +At least some have to be ready for use and upstream merging +before the end of the summer. + +(As a bonus, +in addition to these explicit tests, +it would be helpful to integrate some methods +for testing [[locking validity|libdiskfs_locking]], +performing static code analysis etc.) + +This task probably requires some previous experience +with unit testing of C programs, +as well as dealing with complex build systems. +No in-depth knowledge about any specific parts of the Hurd should be necessary, +but some general understanding of the Hurd architecture +will have to be aquired to complete this project. +This makes it a very good project +to get started on Hurd development :-) + +Possible mentors: ? + +Exercise: Create a program performing some simple test(s) on the Hurd or Mach code. +It doesn't need to be integrated in the build process yet -- +a standalone progrem with it's own Makefile is fine for now. diff --git a/community/gsoc/project_ideas/testing_framework/discussion.mdwn b/community/gsoc/project_ideas/testing_framework/discussion.mdwn new file mode 100644 index 00000000..b01d13c3 --- /dev/null +++ b/community/gsoc/project_ideas/testing_framework/discussion.mdwn @@ -0,0 +1,272 @@ +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation]] + +freenode, #hurd channel, 2011-03-05: + + <nixness> what about testing though? + <nixness> like sort of "what's missing? lets write tests for it so that + when someone gets to implementing it, he knows what to do. Repeat" + project + <antrik> you mean creating an automated testing framework? + <antrik> this is actually a task I want to add for this year, if I get + around to it :-) + <nixness> yeah I'd very much want to apply for that one + <nixness> cuz I've never done Kernel hacking before, but I know that with + the right tasks like "test VM functionality", I would be able to write up + the automated tests and hopefully learn more about what breaks/makes the + kernel + <nixness> (and it would make implementing the feature much less hand-wavy + and more correct) + <nixness> antrik, I believe the framework(CUnit right?) exists, but we just + need to write the tests. + <antrik> do you have prior experience implementing automated tests? + <nixness> lots of tests! + <nixness> yes, in Java mostly, but I've played around with CUnit + <antrik> ah, great :-) + <nixness> here's what I know from experience: 1) write basic tests. 2) + write ones that test multiple features 3) stress test [option 4) + benchmark and record to DB] + <youpi> well, what we'd rather need is to fix the issues we already know + from the existing testsuites :) + +[[GSoC project propsal|community/gsoc/project_ideas/testsuites]]. + + <nixness> youpi, true, and I think I should check what's available in way + of tests, but if the tests are "all or nothing" then it becomes really + hard to fix your mistakes + <youpi> they're not all or nothing + <antrik> youpi: the existing testsuites don't test specific system features + <youpi> libc ones do + <youpi> we could also check posixtestsuite which does too + +[[open_issues/open_posix_test_suite]]. + + <antrik> AFAIK libc has very few failing tests + +[[open_issues/glibc]]. + + <youpi> err, like twenty? + <youpi> € grep -v '^#' expected-results-i486-gnu-libc | wc -l + <youpi> 67 + <youpi> nope, even more + <antrik> oh, sorry, I confused it with coreutils + <pinotree> plus the binutils ones, i guess + <youpi> yes + +[[open_issues/binutils#weak]]. + + <antrik> anyways, I don't think relying on external testsuites for + regression testing is a good plan + <antrik> also, that doesn't cover unit testing at all + <youpi> why ? + <youpi> sure there can be unit testing at the translator etc. level + <antrik> if we want to implement test-driven development, and/or do serious + refactoring without too much risk, we need a test harness where we can + add specific tests as needed + <youpi> but more than often, the issues are at the libc / etc. level + because of a combination o fthings at the translator level, which unit + testing won't find out + * nixness yewzzz! + <nixness> sure unit testing can find them out. if they're good "unit" tests + <youpi> the problem is that you don't necessarily know what "good" means + <youpi> e.g. for posix correctness + <youpi> since it's not posix + <nixness> but if they're composite clever tests, then you lose that + granularity + <nixness> youpi, is that a blackbox test intended to be run at the very end + to check if you're posix compliant? + <antrik> also, if we have our own test harness, we can run tests + automatically as part of the build process, which is a great plus IMHO + <youpi> nixness: "that" = ? + <nixness> oh nvm, I thought there was a test stuie called "posix + correctness" + <youpi> there's the posixtestsuite yes + <youpi> it's an external one however + <youpi> antrik: being external doesn't mean we can't invoke it + automatically as part of the build process when it's available + <nixness> youpi, but being internal means it can test the inner workings of + certain modules that you are unsure of, and not just the interface + <youpi> sure, that's why I said it'd be useful too + <youpi> but as I said too, most bugs I've seen were not easy to find out at + the unit level + <youpi> but rather at the libc level + <antrik> of course we can integrate external tests if they exist and are + suitable. but that that doesn't preclude adding our own ones too. in + either case, that integration work has to be done too + <youpi> again, I've never said I was against internal testsuite + <antrik> also, the major purpose of a test suite is checking known + behaviour. a low-level test won't directly point to a POSIX violation; + but it will catch any changes in behaviour that would lead to one + <youpi> what I said is that it will be hard to write them tight enough to + find bugs + <youpi> again, the problem is knowing what will lead to a POSIX violation + <youpi> it's long work + <youpi> while libc / posixtestsuite / etc. already do that + <antrik> *any* unexpected change in behaviour is likely to cause bugs + somewher + <youpi> but WHAT is "expected" ? + <youpi> THAT is the issue + <youpi> and libc/posixtessuite do know that + <youpi> at the translator level we don't really + <youpi> see the recent post about link() + +[link(dir,name) should fail with +EPERM](http://lists.gnu.org/archive/html/bug-hurd/2011-03/msg00007.html) + + <youpi> in my memory jkoenig pointed it out for a lot of such calls + <youpi> and that issue is clearly not known at the translator level + <nixness> so you're saying that the tests have to be really really + low-level, and work our way up? + <youpi> I'm saying that the translator level tests will be difficult to + write + <antrik> why isn't it known at the translator level? if it's a translator + (not libc) bug, than obviously the translator must return something wrong + at some point, and that's something we can check + <youpi> because you'll have to know all the details of the combinations + used in libc, to know whether they'll lead to posix issues + <youpi> antrik: sure, but how did we detect that that was unexpected + behavior? + <youpi> because of a glib test + <youpi> at the translator level we didn't know it was an unexpected + behavior + <antrik> gnulib actually + <youpi> and if you had asked me, I wouldn't have known + <antrik> again, we do *not* write a test suite to find existing bugs + <youpi> right, took one for the other + <youpi> doesn't really matter actually + <youpi> antrik: ok, I don't care then + <antrik> we write a test suite to prevent future bugs, or track status of + known bugs + <youpi> (don't care *enough* for now, I mean) + <nixness> hmm, so write something up antrik for GSoC :) and I'll be sure to + apply + <antrik> now that we know some translators return a wrong error status in a + particular situation, we can write a test that checks exactly this error + status. that way we know when it is fixed, and also when it breaks again + <antrik> nixness: great :-) + <nixness> sweet. that kind of thing would also need a db backend + <antrik> nixness: BTW, if you have a good idea, you can send an application + for it even if it's not listed among the proposed tasks... + <antrik> so you don't strictly need a writeup from me to apply for this :-) + <nixness> antrik, I'll keep that in mind, but I'll also be checking your + draft page + <nixness> oh cool :) + <antrik> (and it's a well known fact that those projects which students + proposed themselfs tend to be the most successful ones :-) ) + * nixness draft initiated + <antrik> youpi: BTW, I'm surprised that you didn't mention libc testsuite + before... working up from there is probably a more effective plan than + starting with high-level test suites like Python etc... + <youpi> wasn't it already in the gsoc proposal? + <youpi> bummer + <antrik> nope + +freenode, #hurd channel, 2011-03-06: + + <nixness> how's the hurd coding workflow, typically? + +*nixness* -> *foocraft*. + + <foocraft> we're discussing how TDD can work with Hurd (or general kernel + development) on #osdev + <foocraft> so what I wanted to know, since I haven't dealt much with GNU + Hurd, is how do you guys go about coding, in this case + <tschwinge> Our current workflow scheme is... well... is... + <tschwinge> Someone wants to work on something, or spots a bug, then works + on it, submits a patch, and 0 to 10 years later it is applied. + <tschwinge> Roughly. + <foocraft> hmm so there's no indicator of whether things broke with that + patch + <foocraft> and how low do you think we can get with tests? A friend of mine + was telling me that with kernel dev, you really don't know whether, for + instance, the stack even exists, and a lot of things that I, as a + programmer, can assume while writing code break when it comes to writing + kernel code + <foocraft> Autotest seems promising + +See autotest link given above. + + <foocraft> but in any case, coming up with the testing framework that + doesn't break when the OS itself breaks is hard, it seems + <foocraft> not sure if autotest isolates the mistakes in the os from + finding their way in the validity of the tests themselves + <youpi> it could be interesting to have scripts that automatically start a + sub-hurd to do the tests + +[[hurd/subhurd#unit_testing]]. + + <tschwinge> foocraft: To answer one of your earlier questions: you can do + really low-level testing. Like testing Mach's message passing. A + million times. The questions is whether that makes sense. And / or if + it makes sense to do that as part of a unit testing framework. Or rather + do such things manually once you suspect an error somewhere. + <tschwinge> The reason for the latter may be that Mach IPC is already + heavily tested during normal system operation. + <tschwinge> And yet, there still may be (there are, surely) bugs. + <tschwinge> But I guess that you have to stop at some (arbitrary?) level. + <foocraft> so we'd assume it works, and test from there + <tschwinge> Otherwise you'd be implementing the exact counter-part of what + you're testing. + <tschwinge> Which may be what you want, or may be not. Or it may just not + be feasible. + <foocraft> maybe the testing framework should have dependencies + <foocraft> which we can automate using make, and phony targets that run + tests + <foocraft> so everyone writes a test suite and says that it depends on A + and B working correctly + <foocraft> then it'd go try to run the tests for A etc. + <tschwinge> Hmm, isn't that -- on a high level -- have you have by + different packages? For example, the perl testsuite depends (inherently) + on glibc working properly. A perl program's testsuite depends on perl + working properly. + <foocraft> yeah, but afaik, the ordering is done by hand + +freenode, #hurd channel, 2011-03-07: + + <antrik> actually, I think for most tests it would be better not to use a + subhurd... that leads precisely to the problem that if something is + broken, you might have a hard time running the tests at all :-) + <antrik> foocraft: most of the Hurd code isn't really low-level. you can + use normal debugging and testing methods + <antrik> gnumach of course *does* have some low-level stuff, so if you add + unit tests to gnumach too, you might run into issues :-) + <antrik> tschwinge: I think testing IPC is a good thing. as I already said, + automated testing is *not* to discover existing but unknown bugs, but to + prevent new ones creeping in, and tracking progress on known bugs + <antrik> tschwinge: I think you are confusing unit testing and regression + testing. http://www.bddebian.com/~hurd-web/open_issues/unit_testing/ + talks about unit testing, but a lot (most?) of it is actually about + regression tests... + <tschwinge> antrik: That may certainly be -- I'm not at all an expert in + this, and just generally though that some sort of automated testing is + needed, and thus started collecting ideas. + <tschwinge> antrik: You're of course invited to fix that. + +IRC, freenode, #hurd, 2011-03-08 + +(After discussing the [[open_issues/anatomy_of_a_hurd_system]].) + + <antrik> so that's what your question is actually about? + <foocraft> so what I would imagine is a set of only-this-server tests for + each server, and then we can have fun adding composite tests + <foocraft> thus making debugging the composite scenarios a bit less tricky + <antrik> indeed + <foocraft> and if you were trying to pass a composite test, it would also + help knowing that you still didn't break the server-only test + <antrik> there are so many different things that can be tested... the + summer will only suffice to dip into this really :-) + <foocraft> yeah, I'm designing my proposal to focus on 1) make/use a + testing framework that fits the Hurd case very well 2) write some tests + and docs on how to write good tests + <antrik> well, doesn't have to be *one* framework... unit testing and + regression testing are quite different things, which can be covered by + different frameworks diff --git a/community/gsoc/project_ideas/testsuites.mdwn b/community/gsoc/project_ideas/testsuites.mdwn new file mode 100644 index 00000000..9ca6fe3e --- /dev/null +++ b/community/gsoc/project_ideas/testsuites.mdwn @@ -0,0 +1,62 @@ +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Fix Compatibility Problems Exposed by Testsuites"]] + +A number of software packages come with extensive testsuites. +Some notable ones are [[open_issues/glibc]], gnulib, Perl, +Python, GNU Coreutils, and glib. +While these testsuites were written mostly to track regressions in the respective packages, +some of the tests fail on the Hurd in general. + +There is also the [[Open POSIX Testsuite|open_issues/open_posix_test_suite]] +which is more of a whole system interface testing suite. + +Then, there is the [[open_issues/File_System_Exerciser]] which we can use to +test our file system servers for conformity. + +While in some cases these might point to wrong usage of system interfaces, +most of the time such failures are actually caused by shortcomings in Hurd's implementation of these interfaces. +These shortcomings are often not obvious in normal use, +but can be directly or indirectly responsible for all kinds of failures. +The testsuites help in isolating such problems, +so they can be tracked down and fixed. + +This task thus consists in running some of the mentioned testsuites +(and/or any other ones that come to mind), +and looking into the causes of failures. +The goal is to analyze all failures in one or more of the listed testsuites, +to find out what shortcomings in the Hurd implementation cause them (if any), +and to fix at least some of these shortcomings. + +Note that this task somewhat overlaps with the [[Perl/Python task|perl_python]] listed above. +Here the focus however is not on improving the support for any particular program, +but on fixing general problems in the Hurd. + +This is a very flexible task: +while less experienced students should be able to tackle at least a few of the easier problems, +other issues will be challenging even for experienced hackers. +No specific previous knowledge is required for this task; +only fairly decent C programming skills. +While tracking down the various issues, +the student will be digging into the inner workings of the Hurd, +and thus gradually gaining the knowledge required for Hurd development in general. + +A complementary task is adding a proper [[open_issues/unit_testing]] framework +to the GNU Hurd's code base, and related packages. + +Possible mentors: Samuel Thibault (youpi) + +Exercise: Take a stab at one of the testsuite failures, +and write a minimal testcase exposing the underlying problem. +Actually fixing it would be a bonus of course -- +but as it's hard to predict which issues will be easy and which will be tricky, +we will already be satisfied if the student makes a good effort. +(We hope to see some discussion of the problems in this case though :-) ) diff --git a/community/gsoc/project_ideas/tmpfs.mdwn b/community/gsoc/project_ideas/tmpfs.mdwn index 69adef0f..c38c6da8 100644 --- a/community/gsoc/project_ideas/tmpfs.mdwn +++ b/community/gsoc/project_ideas/tmpfs.mdwn @@ -1,21 +1,27 @@ -[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2012 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!meta title="Fix tmpfs"]] +[!] [[Maksym_Planeta]] has been making good progress here; status is tracked at +[[here|hurd/translator/tmpfs/discussion]]. + +--- + In some situations it is desirable to have a file system that is not backed by actual disk storage, but only by anonymous memory, i.e. lives in the RAM (and possibly swap space). A simplistic way to implement such a memory filesystem is literally creating a -ramdisk, i.e. simply allocating a big chunck of RAM (called a memory store in +ramdisk, i.e. simply allocating a big chunk of RAM (called a memory store in Hurd terminology), and create a normal filesystem like ext2 on that. However, this is not very efficient, and not very convenient either (the filesystem needs to be recreated each time the ramdisk is invoked). A nicer solution is @@ -37,7 +43,7 @@ implementation. It requires digging into some parts of the Hurd, including the [[pager_interface|hurd/libpager]] and [[hurd/translator]] programming. This task probably doesn't require any design work, only good debugging skills. -Possible mentors: ? +Possible mentors: Carl Fredrik Hammar (cfhammar) Exercise: Take a look at tmpfs and try to fix one of the existing issues. Some of them are probably not too tricky; or you might discover something else you diff --git a/community/gsoc/project_ideas/unionfs_boot.mdwn b/community/gsoc/project_ideas/unionfs_boot.mdwn index a801290f..d9f1a9e1 100644 --- a/community/gsoc/project_ideas/unionfs_boot.mdwn +++ b/community/gsoc/project_ideas/unionfs_boot.mdwn @@ -11,7 +11,7 @@ is included in the section entitled [[!meta title="Allow Using unionfs Early at Boot"]] In [[UNIX]] systems, traditionally most software is installed in a common directory -hierachy, where files from various packages live beside each other, grouped by +hierarchy, where files from various packages live beside each other, grouped by function: user-invokable executables in `/bin`, system-wide configuration files in `/etc`, architecture specific static files in `/lib`, variable data in `/var`, and so on. To allow clean installation, deinstallation, and upgrade of @@ -42,4 +42,4 @@ Completing this task will require gaining a very good understanding of the Hurd boot process and other parts of the design. It requires some design skills also to come up with a working mechanism. -Possible mentors: ? +Possible mentors: Carl Fredrik Hammar (cfhammar) diff --git a/community/gsoc/project_ideas/valgrind.mdwn b/community/gsoc/project_ideas/valgrind.mdwn new file mode 100644 index 00000000..e9e94857 --- /dev/null +++ b/community/gsoc/project_ideas/valgrind.mdwn @@ -0,0 +1,83 @@ +[[!meta copyright="Copyright © 2009, 2010, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Porting Valgrind to the Hurd"]] + +[[!tag open_issue_gnumach open_issue_hurd]] + +[Valgrind](http://valgrind.org/) is an extremely useful debugging tool for memory errors. +(And some other kinds of hard-to-find errors too.) +Aside from being useful for program development in general, +a Hurd port will help finding out why certain programs segfault on the Hurd, +although they work on Linux. +Even more importantly, it will help finding bugs in the Hurd servers themselfs. + +To keep track of memory use, +Valgrind however needs to know how each [[system call]] affects the validity of memory regions. +This knowledge is highly kernel-specific, +and thus Valgrind needs to be explicitely ported for every system. + +Such a port involves two major steps: +making Valgrind understand how kernel traps work in general on the system in question; +and how all the individual kernel calls affect memory. +The latter step is where most of the work is, +as the behaviour of each single [[system call]] needs to be described. + +Compared to Linux, +[[microkernel/Mach]] (the microkernel used by the Hurd) has very few kernel traps. +Almost all [[system call]]s are implemented as [[RPC]]s instead -- +either handled by Mach itself, or by the various [[Hurd servers|hurd/translator]]. +All RPCs use a pair of `mach_msg()` invocations: +one to send a request message, and one to receive a reply. +However, while all RPCs use the same `mach_msg()` trap, +the actual effect of the call varies greatly depending on which RPC is invoked -- +similar to the `ioctl()` call on Linux. +Each request thus must be handled individually. + +Unlike `ioctl()`, +the RPC invocations have explicit type information for the parameters though, +which can be retrieved from the message header. +By analyzing the parameters of the RPC reply message, +Valgrind can know exactly which memory regions are affected by that call, +even without specific knowledge of the RPC in question. +Thus implementing a general parser for the reply messages +will already give Valgrind a fairly good approximation of memory validity -- +without having to specify the exact semantic of each RPC by hand. + +While this should make Valgrind quite usable on the Hurd already, it's not perfect: +some RPCs might return a buffer that is only partially filled with valid data; +or some reply parameters might be optional, +and only contain valid data under certain conditions. +Such specific semantics can't be deduced from the message headers alone. +Thus for a complete port, +it will still be necessary to go through the list of all known RPCs, +and implement special handling in Valgrind for those RPCs that need it. + +The goal of this task is at minimum to make Valgrind grok Mach traps, +and to implement the generic RPC handler. +Ideally, specific handling for RPCs needing it should also be implemented. + +Completing this project will require digging into Valgrind's handling of [[system call]]s, +and into Hurd RPCs. +It is not an easy task, but a fairly predictable one -- +there shouldn't be any unexpected difficulties, +and no major design work is necessary. +It doesn't require any specific previous knowledge: +only good programming skills in general. +On the other hand, +the student will obtain a good understanding of Hurd RPCs while working on this task, +and thus perfect qualifications for Hurd development in general :-) + +Possible mentors: Samuel Thibault (youpi) + +Exercise: As a starter, +students can try to teach valgrind a couple of Linux ioctls, +as this will make them learn how to use the read/write primitives of valgrind. diff --git a/community/gsoc/project_ideas/virtualization.mdwn b/community/gsoc/project_ideas/virtualization.mdwn index c7403f70..bd67718b 100644 --- a/community/gsoc/project_ideas/virtualization.mdwn +++ b/community/gsoc/project_ideas/virtualization.mdwn @@ -25,18 +25,18 @@ desired constellations. The goal is to create a set of powerful tools for managing at least one desirable virtualization scenario. One possible starting point could be the [[hurd/subhurd]]/[[hurd/neighborhurd]] mechanism, which allows a second almost totally -independant instance of the Hurd in parallel to the main one. +independent instance of the Hurd in parallel to the main one. While subhurd allow creating a complete second system instance, with an own set of Hurd servers and [[UNIX]] daemons and all, there are also situations where it is -desirable to have a smaller subenvironment, living withing the main system and +desirable to have a smaller subenvironment, living within the main system and using most of its facilities -- similar to a chroot environment. A simple way to create such a subenvironment with a single command would be very helpful. It might be possible to implement (perhaps as a prototype) a wrapper using existing tools (chroot and [[hurd/translator/unionfs]]); or it might require more specific tools, -like some kind of unionfs-like filesytem proxy that mirrors other parts of the -filesystem, but allows overriding individual locations, in conjuction with +like some kind of unionfs-like filesystem proxy that mirrors other parts of the +filesystem, but allows overriding individual locations, in conjunction with either chroot or some similar mechanism to create a subenvironment with a different root filesystem. @@ -73,4 +73,4 @@ Completing this project will require gaining a very good understanding of the Hurd architecture and spirit. Previous experience with other virtualization solutions would be very helpful. -Possible mentors: Olaf Buddenhagen (antrik) +Possible mentors: Olaf Buddenhagen (antrik), Carl Fredrik Hammar (cfhammar) diff --git a/community/gsoc/project_ideas/vm_tuning.mdwn b/community/gsoc/project_ideas/vm_tuning.mdwn index 9e802188..ecc5f9f4 100644 --- a/community/gsoc/project_ideas/vm_tuning.mdwn +++ b/community/gsoc/project_ideas/vm_tuning.mdwn @@ -14,7 +14,7 @@ Hurd/[[microkernel/Mach]] presently make very bad use of the available physical system. Some of the problems are inherent to the system design (the kernel can't distinguish between important application data and discardable disk buffers for example), and can't be fixed without fundamental changes. Other -problems however are an ordinary lack of optimisation, like extremely crude +problems however are an ordinary lack of optimization, like extremely crude heuristics when to start paging. (See <http://lists.gnu.org/archive/html/bug-hurd/2007-08/msg00034.html> for example.) Many parameters are based on assumptions from a time when typical machines had like 16 MiB of RAM, or simply have been set to @@ -23,7 +23,7 @@ arbitrary values and never tuned for actual use. The goal of this project is to bring the virtual memory management in Hurd/Mach closer to that of modern mainstream kernels (Linux, FreeBSD), by comparing the implementation to other systems, implementing any worthwhile improvements, and -general optimisation/tuning. It requires very good understanding of the Mach +general optimization/tuning. It requires very good understanding of the Mach VM, and virtual memory in general. This project is related to [[!GNU_Savannah_task 5489]]. diff --git a/community/gsoc/project_ideas/xattr.mdwn b/community/gsoc/project_ideas/xattr.mdwn index 55961547..8ec4c42e 100644 --- a/community/gsoc/project_ideas/xattr.mdwn +++ b/community/gsoc/project_ideas/xattr.mdwn @@ -21,7 +21,7 @@ fields in the inode to store Hurd-specific metadata: most notable passive translator settings. As these fields are Hurd-specific, they can't be accessed by the standard methods from Linux for example, so it's not possible to fully work with a Hurd filesystem on GNU/Linux (copy, backup etc.); and also, even -when on Hurd, only tools that explicitely support the Hurd-specific information +when on Hurd, only tools that explicitly support the Hurd-specific information can handle them. Using extended attributes instead of custom fields for the Hurd-specific @@ -38,8 +38,8 @@ Completing this project will require digging into some parts of the Hurd, but it should be quite doable without previous Hurd experience. Some experience with xattrs might help a bit, but shouldn't be really necessary either. -Some previous work on xattr support is available in [[!GNU_Savannah_patch -5126]], and might serve as a starting point. +Some previous work on xattr support is [[available|open_issues/xattr]], and +might serve as a starting point. Possible mentors: Samuel Thibault (youpi) |