From d9d8ca2b16d67441d14182bb394dc15730425d99 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sun, 6 Mar 2011 00:00:07 +0100 Subject: community/gsoc/project_ideas/testsuites: Refer to gnulib testsuite, too. --- community/gsoc/project_ideas/testsuites.mdwn | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/community/gsoc/project_ideas/testsuites.mdwn b/community/gsoc/project_ideas/testsuites.mdwn index 7cb7d103..009aadf0 100644 --- a/community/gsoc/project_ideas/testsuites.mdwn +++ b/community/gsoc/project_ideas/testsuites.mdwn @@ -1,17 +1,17 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!meta title="Fix Compatibility Problems Exposed by Testsuites"]] A number of software packages come with extensive testsuites. -Some notable ones are libc, Perl, Python, GNU Coreutils, and glib. +Some notable ones are libc, gnulib, Perl, Python, GNU Coreutils, and glib. While these testsuites were written mostly to track regressions in the respective packages, some of the tests fail on the Hurd in general. -- cgit v1.2.3 From 4b77295183e89d5138673e6bcf832db1ff0369f3 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sun, 6 Mar 2011 00:05:55 +0100 Subject: community/gsoc/project_ideas/testsuites: Link to open_issues/unit_testing. --- community/gsoc/project_ideas/testsuites.mdwn | 3 +++ 1 file changed, 3 insertions(+) diff --git a/community/gsoc/project_ideas/testsuites.mdwn b/community/gsoc/project_ideas/testsuites.mdwn index 009aadf0..ee1dc10c 100644 --- a/community/gsoc/project_ideas/testsuites.mdwn +++ b/community/gsoc/project_ideas/testsuites.mdwn @@ -42,6 +42,9 @@ While tracking down the various issues, the student will be digging into the inner workings of the Hurd, and thus gradually gaining the knowledge required for Hurd development in general. +A complementary task is adding a proper [[open_issues/unit_testing]] framework +to the GNU Hurd's code base, and related packages. + Possible mentors: Samuel Thibault (youpi) Exercise: Take a stab at one of the testsuite failures, -- cgit v1.2.3 From b26e27692be487076bb547f7d6cb810bf7a3692c Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sun, 6 Mar 2011 00:16:30 +0100 Subject: community/gsoc/project_ideas/testsuites: Link to open_issues/open_posix_test_suite. --- community/gsoc/project_ideas/testsuites.mdwn | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/community/gsoc/project_ideas/testsuites.mdwn b/community/gsoc/project_ideas/testsuites.mdwn index ee1dc10c..8100bfb7 100644 --- a/community/gsoc/project_ideas/testsuites.mdwn +++ b/community/gsoc/project_ideas/testsuites.mdwn @@ -11,10 +11,14 @@ License|/fdl]]."]]"""]] [[!meta title="Fix Compatibility Problems Exposed by Testsuites"]] A number of software packages come with extensive testsuites. -Some notable ones are libc, gnulib, Perl, Python, GNU Coreutils, and glib. +Some notable ones are [[glibc|open_issues/glibc_testsuite]], gnulib, Perl, +Python, GNU Coreutils, and glib. While these testsuites were written mostly to track regressions in the respective packages, some of the tests fail on the Hurd in general. +There is also the [[Open POSIX Testsuite|open_issues/open_posix_test_suite]] +which is more of a whole system interface testing suite. + While in some cases these might point to wrong usage of system interfaces, most of the time such failures are actually caused by shortcomings in Hurd's implementation of these interfaces. These shortcomings are often not obvious in normal use, -- cgit v1.2.3 From ff8af7a8958e412a03445e71340aedab2dc5fb05 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sun, 6 Mar 2011 00:25:17 +0100 Subject: open_issues/unit_testing: Add 2011-03-05 IRC discussion. --- open_issues/unit_testing.mdwn | 163 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 163 insertions(+) diff --git a/open_issues/unit_testing.mdwn b/open_issues/unit_testing.mdwn index d7821fd3..8749e650 100644 --- a/open_issues/unit_testing.mdwn +++ b/open_issues/unit_testing.mdwn @@ -70,3 +70,166 @@ abandoned). * [LaBrea](https://github.com/dustin/labrea/wiki), or similar tools can be used for modelling certain aspects of system behavior (long response times, for example). + + +# Discussion + +freenode, #hurd channel, 2011-03-05: + + what about testing though? + like sort of "what's missing? lets write tests for it so that + when someone gets to implementing it, he knows what to do. Repeat" + project + you mean creating an automated testing framework? + this is actually a task I want to add for this year, if I get + around to it :-) + yeah I'd very much want to apply for that one + cuz I've never done Kernel hacking before, but I know that with + the right tasks like "test VM functionality", I would be able to write up + the automated tests and hopefully learn more about what breaks/makes the + kernel + (and it would make implementing the feature much less hand-wavy + and more correct) + antrik, I believe the framework(CUnit right?) exists, but we just + need to write the tests. + do you have prior experience implementing automated tests? + lots of tests! + yes, in Java mostly, but I've played around with CUnit + ah, great :-) + here's what I know from experience: 1) write basic tests. 2) + write ones that test multiple features 3) stress test [option 4) + benchmark and record to DB] + well, what we'd rather need is to fix the issues we already know + from the existing testsuites :) + +[[GSoC project propsal|community/gsoc/project_ideas/testsuites]]. + + youpi, true, and I think I should check what's available in way + of tests, but if the tests are "all or nothing" then it becomes really + hard to fix your mistakes + they're not all or nothing + youpi: the existing testsuites don't test specific system features + libc ones do + we could also check posixtestsuite which does too + +[[open_issues/open_posix_test_suite]]. + + AFAIK libc has very few failing tests + +[[open_issues/glibc_testsuite]]. + + err, like twenty? + € grep -v '^#' expected-results-i486-gnu-libc | wc -l + 67 + nope, even more + oh, sorry, I confused it with coreutils + plus the binutils ones, i guess + yes + +[[open_issues/binutils#weak]]. + + anyways, I don't think relying on external testsuites for + regression testing is a good plan + also, that doesn't cover unit testing at all + why ? + sure there can be unit testing at the translator etc. level + if we want to implement test-driven development, and/or do serious + refactoring without too much risk, we need a test harness where we can + add specific tests as needed + but more than often, the issues are at the libc / etc. level + because of a combination o fthings at the translator level, which unit + testing won't find out + * nixness yewzzz! + sure unit testing can find them out. if they're good "unit" tests + the problem is that you don't necessarily know what "good" means + e.g. for posix correctness + since it's not posix + but if they're composite clever tests, then you lose that + granularity + youpi, is that a blackbox test intended to be run at the very end + to check if you're posix compliant? + also, if we have our own test harness, we can run tests + automatically as part of the build process, which is a great plus IMHO + nixness: "that" = ? + oh nvm, I thought there was a test stuie called "posix + correctness" + there's the posixtestsuite yes + it's an external one however + antrik: being external doesn't mean we can't invoke it + automatically as part of the build process when it's available + youpi, but being internal means it can test the inner workings of + certain modules that you are unsure of, and not just the interface + sure, that's why I said it'd be useful too + but as I said too, most bugs I've seen were not easy to find out at + the unit level + but rather at the libc level + of course we can integrate external tests if they exist and are + suitable. but that that doesn't preclude adding our own ones too. in + either case, that integration work has to be done too + again, I've never said I was against internal testsuite + also, the major purpose of a test suite is checking known + behaviour. a low-level test won't directly point to a POSIX violation; + but it will catch any changes in behaviour that would lead to one + what I said is that it will be hard to write them tight enough to + find bugs + again, the problem is knowing what will lead to a POSIX violation + it's long work + while libc / posixtestsuite / etc. already do that + *any* unexpected change in behaviour is likely to cause bugs + somewher + but WHAT is "expected" ? + THAT is the issue + and libc/posixtessuite do know that + at the translator level we don't really + see the recent post about link() + +[link(dir,name) should fail with +EPERM](http://lists.gnu.org/archive/html/bug-hurd/2011-03/msg00007.html) + + in my memory jkoenig pointed it out for a lot of such calls + and that issue is clearly not known at the translator level + so you're saying that the tests have to be really really + low-level, and work our way up? + I'm saying that the translator level tests will be difficult to + write + why isn't it known at the translator level? if it's a translator + (not libc) bug, than obviously the translator must return something wrong + at some point, and that's something we can check + because you'll have to know all the details of the combinations + used in libc, to know whether they'll lead to posix issues + antrik: sure, but how did we detect that that was unexpected + behavior? + because of a glib test + at the translator level we didn't know it was an unexpected + behavior + gnulib actually + and if you had asked me, I wouldn't have known + again, we do *not* write a test suite to find existing bugs + right, took one for the other + doesn't really matter actually + antrik: ok, I don't care then + we write a test suite to prevent future bugs, or track status of + known bugs + (don't care *enough* for now, I mean) + hmm, so write something up antrik for GSoC :) and I'll be sure to + apply + now that we know some translators return a wrong error status in a + particular situation, we can write a test that checks exactly this error + status. that way we know when it is fixed, and also when it breaks again + nixness: great :-) + sweet. that kind of thing would also need a db backend + nixness: BTW, if you have a good idea, you can send an application + for it even if it's not listed among the proposed tasks... + so you don't strictly need a writeup from me to apply for this :-) + antrik, I'll keep that in mind, but I'll also be checking your + draft page + oh cool :) + (and it's a well known fact that those projects which students + proposed themselfs tend to be the most successful ones :-) ) + * nixness draft initiated + youpi: BTW, I'm surprised that you didn't mention libc testsuite + before... working up from there is probably a more effective plan than + starting with high-level test suites like Python etc... + wasn't it already in the gsoc proposal? + bummer + nope -- cgit v1.2.3 From afe6f5029552c8c6320e0074dd6b96f76db41a8d Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sun, 6 Mar 2011 00:29:22 +0100 Subject: open_issues/unit_testing: Might be a GSoC task. --- community/gsoc/project_ideas.mdwn | 3 ++- open_issues/unit_testing.mdwn | 4 ++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/community/gsoc/project_ideas.mdwn b/community/gsoc/project_ideas.mdwn index 649e05c1..e03dbede 100644 --- a/community/gsoc/project_ideas.mdwn +++ b/community/gsoc/project_ideas.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -104,6 +104,7 @@ See also the list of [Hurd-related X.org project ideas](http://wiki.x.org/wiki/H [[!inline pages="community/gsoc/project_ideas/cdparanoia" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/perl_python" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/testsuites" show=0 feeds=no actions=yes]] +[[!inline pages="open_issues/unit_testing" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/libcap" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/xattr" show=0 feeds=no actions=yes]] [[!inline pages="open_issues/valgrind" show=0 feeds=no actions=yes]] diff --git a/open_issues/unit_testing.mdwn b/open_issues/unit_testing.mdwn index 8749e650..95aa0b5f 100644 --- a/open_issues/unit_testing.mdwn +++ b/open_issues/unit_testing.mdwn @@ -8,6 +8,10 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] +This task may be suitable for [[community/GSoC]]. + +--- + A collection of thoughts with respect to unit testing. We definitely want to add unit test suites to our code base. -- cgit v1.2.3 From c318adc3b829acd0e1ce0fb6dd055a1c6d165e98 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sun, 6 Mar 2011 00:54:54 +0100 Subject: community/gsoc/project_ideas: Add a listing of the tasks. --- community/gsoc/project_ideas.mdwn | 7 +++++++ open_issues/locking.mdwn | 4 ++-- open_issues/performance/io_system.mdwn | 2 +- open_issues/unit_testing.mdwn | 2 +- open_issues/valgrind.mdwn | 5 ++++- 5 files changed, 15 insertions(+), 5 deletions(-) diff --git a/community/gsoc/project_ideas.mdwn b/community/gsoc/project_ideas.mdwn index e03dbede..4aa9b547 100644 --- a/community/gsoc/project_ideas.mdwn +++ b/community/gsoc/project_ideas.mdwn @@ -76,6 +76,13 @@ will assist you as well as we can. See also the list of [Hurd-related X.org project ideas](http://wiki.x.org/wiki/Hurd_Porting). +Here is a list of what we consider suitable tasks: + +[[!map show=titlepages="(community/gsoc/project_ideas/* and + !community/gsoc/project_ideas/*/*) or tagged(gsoc-task)"]] + +Here are these pages' content: + [[!inline pages="community/gsoc/project_ideas/language_bindings" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/virtualization" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/file_locking" show=0 feeds=no actions=yes]] diff --git a/open_issues/locking.mdwn b/open_issues/locking.mdwn index 11a10524..6e22f887 100644 --- a/open_issues/locking.mdwn +++ b/open_issues/locking.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -9,7 +9,7 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -[[!tag open_issue_hurd]] +[[!tag open_issue_hurd gsoc-task]] Every now and then, new locking issues are discovered in [[hurd/libdiskfs]] or [[hurd/translator/ext2fs]], for example. Nowadays diff --git a/open_issues/performance/io_system.mdwn b/open_issues/performance/io_system.mdwn index 4af093ba..8535eae3 100644 --- a/open_issues/performance/io_system.mdwn +++ b/open_issues/performance/io_system.mdwn @@ -11,7 +11,7 @@ License|/fdl]]."]]"""]] [[!meta title="I/O System"]] -[[!tag open_issue_hurd]] +[[!tag open_issue_hurd gsoc-task]] The most obvious reason for the Hurd feeling slow compared to mainstream systems like GNU/Linux, is a low I/O system performance, in particular very diff --git a/open_issues/unit_testing.mdwn b/open_issues/unit_testing.mdwn index 95aa0b5f..0ff7ecca 100644 --- a/open_issues/unit_testing.mdwn +++ b/open_issues/unit_testing.mdwn @@ -8,7 +8,7 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -This task may be suitable for [[community/GSoC]]. +This task may be suitable for [[community/GSoC]][[!tag gsoc-task]]. --- diff --git a/open_issues/valgrind.mdwn b/open_issues/valgrind.mdwn index 2b0624d7..bd45829c 100644 --- a/open_issues/valgrind.mdwn +++ b/open_issues/valgrind.mdwn @@ -1,4 +1,5 @@ -[[!meta copyright="Copyright © 2009, 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2009, 2010, 2011 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -10,6 +11,8 @@ License|/fdl]]."]]"""]] [[!meta title="Porting Valgrind to the Hurd"]] +[[!tag gsoc-task]] + [Valgrind](http://valgrind.org/) is an extremely useful debugging tool for memory errors. (And some other kinds of hard-to-find errors too.) Aside from being useful for program development in general, -- cgit v1.2.3 From b92e9ff476b7ab5bc177051fb951162b472bee75 Mon Sep 17 00:00:00 2001 From: GNU Hurd web pages engine Date: Sat, 5 Mar 2011 23:58:22 +0000 Subject: creating tag page tag/gsoc-task --- tag/gsoc-task.mdwn | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 tag/gsoc-task.mdwn diff --git a/tag/gsoc-task.mdwn b/tag/gsoc-task.mdwn new file mode 100644 index 00000000..7f0700ef --- /dev/null +++ b/tag/gsoc-task.mdwn @@ -0,0 +1,15 @@ +[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="gsoc-task"]] + +[[!map +pages="tagged(gsoc-task)" +show=title]] -- cgit v1.2.3 From 710268c7a65302f566841bd37966f85996d5a711 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sun, 6 Mar 2011 01:13:47 +0100 Subject: tag/gsoc-task: Actually simply redirect to the GSoC project ideas page. --- tag/gsoc-task.mdwn | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/tag/gsoc-task.mdwn b/tag/gsoc-task.mdwn index 7f0700ef..99758478 100644 --- a/tag/gsoc-task.mdwn +++ b/tag/gsoc-task.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -8,8 +8,4 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -[[!meta title="gsoc-task"]] - -[[!map -pages="tagged(gsoc-task)" -show=title]] +[[!meta redir=community/gsoc/project_ideas]] -- cgit v1.2.3 From a73a253a6ab7c2a6894f265c6b42f0b5bb17f1eb Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sun, 6 Mar 2011 01:14:20 +0100 Subject: community/gsoc/project_ideas: Fix fatal typo / missing space. --- community/gsoc/project_ideas.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/community/gsoc/project_ideas.mdwn b/community/gsoc/project_ideas.mdwn index 4aa9b547..da247847 100644 --- a/community/gsoc/project_ideas.mdwn +++ b/community/gsoc/project_ideas.mdwn @@ -78,7 +78,7 @@ See also the list of [Hurd-related X.org project ideas](http://wiki.x.org/wiki/H Here is a list of what we consider suitable tasks: -[[!map show=titlepages="(community/gsoc/project_ideas/* and +[[!map show=title pages="(community/gsoc/project_ideas/* and !community/gsoc/project_ideas/*/*) or tagged(gsoc-task)"]] Here are these pages' content: -- cgit v1.2.3 From 16572788d848f764e9e8f66ce199a99c6a9f4a1f Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sun, 6 Mar 2011 01:20:19 +0100 Subject: .templates/autotag.tmpl: This needs the New Year Procedure. --- .templates/autotag.tmpl | 2 +- contributing/web_pages.mdwn | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.templates/autotag.tmpl b/.templates/autotag.tmpl index 87b76eef..1a963267 100644 --- a/.templates/autotag.tmpl +++ b/.templates/autotag.tmpl @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this diff --git a/contributing/web_pages.mdwn b/contributing/web_pages.mdwn index 105d9dd5..351c82bd 100644 --- a/contributing/web_pages.mdwn +++ b/contributing/web_pages.mdwn @@ -234,6 +234,6 @@ download them from. Files to update: - * `/config_edittemplate/*.mdwn` + * `/config_edittemplate/*.mdwn`, `/.templates/autotag.tmpl` * `/contributing/web_pages/news/skeleton.mdwn` * `/copyright.mdwn` -- cgit v1.2.3 From 6ef71999bc49af76b201f36f11747148d32f5863 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sun, 6 Mar 2011 10:52:01 +0100 Subject: Some more IRC discussion. --- hurd/subhurd.mdwn | 48 ++++++++++++++++++++++++- open_issues/perlmagick.mdwn | 45 ++++++++++++++++++++++- open_issues/unit_testing.mdwn | 83 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 174 insertions(+), 2 deletions(-) diff --git a/hurd/subhurd.mdwn b/hurd/subhurd.mdwn index cb4a40a8..b8e595d3 100644 --- a/hurd/subhurd.mdwn +++ b/hurd/subhurd.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2007, 2008, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2007, 2008, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -21,6 +21,8 @@ possible to use [[debugging/gdb/noninvasive_debugging]], but this is less flexible.) Vice versa, it is also possible to use a subhurd to debug the *main* Hurd system, for example, the latter's root file system. +[[!toc levels=2]] + # Howto @@ -139,3 +141,47 @@ translator|translator/ext2fs]]'s [[PID]], but this is no problem, as currently [[PID]]s are visible across subhurd boundaries. (It is a [[!taglink open_issue_hurd]] whether this is the right thing to do in [[open_issues/virtualization]] contexts, but that's how it currently is.) + + + +## Unit Testing + +freenode, #hurd channel, 2011-03-06: + +From [[open_issues/unit_testing]]. + + it could be interesting to have scripts that automatically start a + sub-hurd to do the tests + though that'd catch sub-hurd issues :) + so a sub-hurd is a hurd that I can run on something that I know + works(like linux)? + Virtual machine I would think + and over a network connection it would submit results back to + the host :p + * foocraft brain damage + sub-hurd is a bit like chroot + except that it's more complete + oh okay + i.e. almost everything gets replaced with what you want, except the + micro-kernel + that way you can even test the exec server for instance, without + risks of damaging the host OS + and we know the micro-kernel works correctly, right youpi? + well, at least it's small enough that most bugs are not there + 1) all tests run in subhurd 2) output results in a place in the + subhurd 3) tester in the host checks the result and pretty-prints it 4) + rinse & repeat + the output can actually be redirected iirc + since you give the sub-hurd a "console" + youpi, yup yeah, so now it's more like chroot if that's the case + it really looks like chroot, yes + but again, there's this subset of tests that we need to have + that ensures that even the tester running on the subhurd is valid, and it + didn't break because of a bug in the subhurd + As long as you do in-system testing, you'll always (have to) + rely on some functionality provided by the host system. + the worst thing that could happen with unit testing is false + results that lead someone to try to fix something that isn't broken :p + Yes. + usually one tries to repeat the test by hand in a normal + environment diff --git a/open_issues/perlmagick.mdwn b/open_issues/perlmagick.mdwn index 1daac62b..8a57a8fd 100644 --- a/open_issues/perlmagick.mdwn +++ b/open_issues/perlmagick.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -57,6 +57,49 @@ Etc. +/usr/lib/gcc/i486-gnu/4.4.2/include/omp.h: +# State as of 2011-03-06 + +freenode, #hurd channel, 2011-03-06: + + tschwinge: (speaking on working perl, how did it en with that + "(glibc) double free" crash with perl?) + *end + I think I remember I suspected it's a libgomp (!) issue in the + end. I have not yet continued working on that. + libogmp? looks like you know more than me, then :) + tschwinge: oh, I'm interested + I know a bit about libgomp :) + I bisected this down to where Imagemagick added -fgomp (or + whatever it is). And then the perl library (Imagemagick.pm?) which loads + the imagemagick.so segfaulted. + ImageMagick did this change in the middle of a x.x.x.something + release.. + My next step would have been to test whether libgomp works at + all for us. + ./usr/sbin/debootstrap:DEBOOTSTRAP_CHECKSUM_FIELD="SHA$SHA_SIZE" + erf + so they switched to another checksum + but we don't have that one on all of our packages :) + tschwinge: + buildd@bach:~$ OMP_NUM_THREADS=2 ./test + I'm 0x1 + I'm 0x3 + libgomp works at least a bit + OK. + i guess we should hope the working bits don't stop at that point + ;) + If open_issues/perlmagick is to be believed a diff of 6.4.1-1 + and 6.4.1-2 should tell what exactly was changed. + Oh! + I even have it on the page already! ;-) + -fopenmp + I've tried the pragmas that imagemagick uses + they work + Might be the issue fixed itself? + I don't know, it's the latest libc here + (and latest hurd, to be uploaded) + + # Other [[!debbug 551017]] diff --git a/open_issues/unit_testing.mdwn b/open_issues/unit_testing.mdwn index 0ff7ecca..a5ffe19d 100644 --- a/open_issues/unit_testing.mdwn +++ b/open_issues/unit_testing.mdwn @@ -58,6 +58,8 @@ abandoned). Developers*](http://lwn.net/Articles/412302/) by Steven Rostedt, 2010-10-28. [v2](http://lwn.net/Articles/414064/), 2010-11-08. + * + # Related @@ -237,3 +239,84 @@ EPERM](http://lists.gnu.org/archive/html/bug-hurd/2011-03/msg00007.html) wasn't it already in the gsoc proposal? bummer nope + +freenode, #hurd channel, 2011-03-06: + + how's the hurd coding workflow, typically? + +*nixness* -> *foocraft*. + + we're discussing how TDD can work with Hurd (or general kernel + development) on #osdev + so what I wanted to know, since I haven't dealt much with GNU + Hurd, is how do you guys go about coding, in this case + Our current workflow scheme is... well... is... + Someone wants to work on something, or spots a bug, then works + on it, submits a patch, and 0 to 10 years later it is applied. + Roughly. + hmm so there's no indicator of whether things broke with that + patch + and how low do you think we can get with tests? A friend of mine + was telling me that with kernel dev, you really don't know whether, for + instance, the stack even exists, and a lot of things that I, as a + programmer, can assume while writing code break when it comes to writing + kernel code + Autotest seems promising + +See autotest link given above. + + but in any case, coming up with the testing framework that + doesn't break when the OS itself breaks is hard, it seems + not sure if autotest isolates the mistakes in the os from + finding their way in the validity of the tests themselves + it could be interesting to have scripts that automatically start a + sub-hurd to do the tests + +[[hurd/subhurd#unit_testing]]. + + foocraft: To answer one of your earlier questions: you can do + really low-level testing. Like testing Mach's message passing. A + million times. The questions is whether that makes sense. And / or if + it makes sense to do that as part of a unit testing framework. Or rather + do such things manually once you suspect an error somewhere. + The reason for the latter may be that Mach IPC is already + heavily tested during normal system operation. + And yet, there still may be (there are, surely) bugs. + But I guess that you have to stop at some (arbitrary?) level. + so we'd assume it works, and test from there + Otherwise you'd be implementing the exact counter-part of what + you're testing. + Which may be what you want, or may be not. Or it may just not + be feasible. + maybe the testing framework should have dependencies + which we can automate using make, and phony targets that run + tests + so everyone writes a test suite and says that it depends on A + and B working correctly + then it'd go try to run the tests for A etc. + Hmm, isn't that -- on a high level -- have you have by + different packages? For example, the perl testsuite depends (inherently) + on glibc working properly. A perl program's testsuite depends on perl + working properly. + yeah, but afaik, the ordering is done by hand + +freenode, #hurd channel, 2011-03-07: + + actually, I think for most tests it would be better not to use a + subhurd... that leads precisely to the problem that if something is + broken, you might have a hard time running the tests at all :-) + foocraft: most of the Hurd code isn't really low-level. you can + use normal debugging and testing methods + gnumach of course *does* have some low-level stuff, so if you add + unit tests to gnumach too, you might run into issues :-) + tschwinge: I think testing IPC is a good thing. as I already said, + automated testing is *not* to discover existing but unknown bugs, but to + prevent new ones creeping in, and tracking progress on known bugs + tschwinge: I think you are confusing unit testing and regression + testing. http://www.bddebian.com/~hurd-web/open_issues/unit_testing/ + talks about unit testing, but a lot (most?) of it is actually about + regression tests... + antrik: That may certainly be -- I'm not at all an expert in + this, and just generally though that some sort of automated testing is + needed, and thus started collecting ideas. + antrik: You're of course invited to fix that. -- cgit v1.2.3 From 2ee8dc077fc9cbdcc997a2771139578dce140644 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sun, 6 Mar 2011 11:28:14 +0100 Subject: GSoC 2010 is over, and 2011 has not yet started. --- community/gsoc.mdwn | 48 +++++++++++++++++------------------------------- community/gsoc/2010.mdwn | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+), 31 deletions(-) create mode 100644 community/gsoc/2010.mdwn diff --git a/community/gsoc.mdwn b/community/gsoc.mdwn index 2ce92d19..5bd28738 100644 --- a/community/gsoc.mdwn +++ b/community/gsoc.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -6,49 +6,35 @@ id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!meta title="Google Summer of Code"]] -# 2010 +# 2011 -This year we are again participating in [Google Summer of Code](http://socghop.appspot.com) -under the [GNU umbrella](http://socghop.appspot.com/gsoc/org/show/google/gsoc2010/gnuproject). +This year we're again planning to participate in [Google Summer of +Code](http://socghop.appspot.com) under the GNU umbrella. - * *[[Jeremie Koenig|jkoenig]]*, mentored by *[[Samuel - Thibault|samuelthibault]]*, is working on adapting the Debian Installer to - [produce working Debian GNU/Hurd installation - images](http://socghop.appspot.com/gsoc/student_project/show/google/gsoc2010/debian/t127230758239) - so we can easily offer up to date disc-sets. - ([Details](http://wiki.debian.org/SummerOfCode2010/HurdDebianInstaller/JeremieKoenig).) - * *[[Emilio Pozuelo Monfort|pochu]]*, mentored by *Carl Fredrik Hammar* (who - was a GSoC student in 2007), is working on a task that may be perceived as - less exciting from the outside, but yet is extremely valuable: [fixing - compatibility problems exposed by projects' - testsuites](http://socghop.appspot.com/gsoc/student_project/show/google/gsoc2010/gnuproject/t127230759396). - ([[Details|community/gsoc/project_ideas/testsuites]].) - - * *[[Karim Allah Ahmed|kam]]*, mentored by *Sergio López*, is working on - [tuning the VM Subsystem in - GNU/Hurd](http://socghop.appspot.com/gsoc/student_project/show/google/gsoc2010/gnuproject/t127230759587) - to bring the virtual memory management in Hurd/Mach up to date. - ([[Details|community/gsoc/project_ideas/vm_tuning]].) +## Applying for a Task We always ask students that want to apply for a task (in the course of the Google Summer of Code) to mind our distinct [[student_application_form]]. -Apart from our own [[project_ideas]], there are also two [Hurd-related ideas -for X.Org](http://wiki.x.org/wiki/Hurd_Porting). +We have a list of [[project_ideas]], and students are likewise encouraged to +submit their own project proposals. + +The Google Summer of Code 2011 has not yet started; we will later announce the +beginning of the students' application phase. However, working on one of these +projects is still a good opportunity to get started with Hurd development, even +outside of the GSoC context. -As the Google Summer of Code 2010 has already started, you can no longer -participate as a GSoC student. However, -working on one of these projects is still a good opportunity to get started with Hurd development. Please read up about [[contributing]] in general; and feel free to ask any questions you might have at one of our [[regular_IRC_meetings|IRC#regular_meetings]]. Generally it's a good idea to [[contact_us|communication]] when starting to work on some project. + # History In 2006 and [[2007]], we participated in GSoC under the umbrella of the GNU @@ -59,5 +45,5 @@ In the following year, we successfully participated on our own, not only as a suborganization of the GNU project. Read about our five students' success on the [[2008]] page. -The next year, we participated under the GNU umbrella with one slot again. -Read about it on the [[2009]] page. +The next two year, we participated under the GNU umbrella with one slot in +[[2009]], and three in [[2010]]. diff --git a/community/gsoc/2010.mdwn b/community/gsoc/2010.mdwn new file mode 100644 index 00000000..4388636b --- /dev/null +++ b/community/gsoc/2010.mdwn @@ -0,0 +1,35 @@ +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Google Summer of Code 2010"]] + +In 2010 we have again been participating in the Google Summer of Code +under the GNU umbrella, with three slots: + + * *[[Jeremie Koenig|jkoenig]]*, mentored by *[[Samuel + Thibault|samuelthibault]]*, has been working on adapting the Debian Installer to + [produce working Debian GNU/Hurd installation + images](http://socghop.appspot.com/gsoc/student_project/show/google/gsoc2010/debian/t127230758239) + so we can easily offer up to date disc-sets. + ([Details](http://wiki.debian.org/SummerOfCode2010/HurdDebianInstaller/JeremieKoenig).) + + * *[[Emilio Pozuelo Monfort|pochu]]*, mentored by *Carl Fredrik Hammar* (who + was a GSoC student in [[2007]]), has been working on a task that may be perceived as + less exciting from the outside, but yet is extremely valuable: [fixing + compatibility problems exposed by projects' + testsuites](http://socghop.appspot.com/gsoc/student_project/show/google/gsoc2010/gnuproject/t127230759396). + ([[Details|community/gsoc/project_ideas/testsuites]].) + + * *[[Karim Allah Ahmed|kam]]*, mentored by *Sergio López*, has been working on + [tuning the VM Subsystem in + GNU/Hurd](http://socghop.appspot.com/gsoc/student_project/show/google/gsoc2010/gnuproject/t127230759587) + to bring the virtual memory management in Hurd/Mach up to date. + ([[Details|community/gsoc/project_ideas/vm_tuning]].) -- cgit v1.2.3 From d463634b179b77b2802bf78110c3ce0a0d6ed64c Mon Sep 17 00:00:00 2001 From: antrik Date: Tue, 15 Mar 2011 17:17:11 +0100 Subject: FAQ/which_microkernel: rewrite Try to explain better what happened, and what it means for Hurd development. Also commented out the IRC discussion: I don't think it's appropriate here. Perhaps it could be included on a different page, or summarized here or elsewhere... --- hurd/faq/which_microkernel.mdwn | 41 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 37 insertions(+), 4 deletions(-) diff --git a/hurd/faq/which_microkernel.mdwn b/hurd/faq/which_microkernel.mdwn index c5026afa..f6225188 100644 --- a/hurd/faq/which_microkernel.mdwn +++ b/hurd/faq/which_microkernel.mdwn @@ -10,13 +10,44 @@ License|/fdl]]."]]"""]] [[!meta title="What happened to the L4 / Coyotos / Viengoos microkernels?"]] -L4 was promising but happened to not be suitable for implementing a general-purpose operating system on top of it. See [[history/port_to_l4]] for the historical details. +Encountering a number of fundamental design issues with the Mach microkernel +(mostly with resource management), +some of the Hurd developers began experimenting with using other microkernels for the Hurd. -Coyotos is abandoned upstream +The first attempt was reimplementing the Hurd on the L4 (Pistachio) microkernel. +This got going around 2003/2004, +and got pretty far (running some simple POSIX programs, such as "banner"); +however over time some lingering design issues turned out to be fundamental problems: +the original L4 is not suitable for building object capability systems like the Hurd. +Thus development was aborted in 2005. -Neal Walfield started working on a newly designed kernel called [[viengoos|microkernel/viengoos]]. Unfortunately, he currently lacks time and the projects it paused. +Following that, an attempt was started to use the kernel of the Coyotos system. +As Coyotos is an object-capability system througout, +the microkernel would obviously be more suitable for this purpose; +and it looked pretty promising in the beginning. +However, further investigations found +that there are some very fundamental philosophical differences +between the Coyotos and Hurd designs; +and thus this this attempt was also abandonned, around 2006/2007. +(This time before producing any actual code.) -In the meanwhile, people are thus continuing with [[microkernel/mach]]. +By now there were some new L4 variants available, +which added protected IPC paths and other features necessary for object capability systems; +so it might be possible to implement the Hurd on top of these. +However, by that time the developers concluded that microkernel design and system design +are interconnected in very intricate ways, +and thus trying to use a third-party microkernel will always result in trouble. +So Neal Walfield created the experimental [[Viengoos|microkernel/viengoos]] kernel instead -- +based on the experience from the previous experiments with L4 and Coyotos -- +for his research on resource management. +Currently he works in another research area though, and thus Viengoos is on hold. + +Note that while none of the microkernel work is active now, +the previous experiments already yielded a lot of experience, +which will be very useful in the further development/improvement +of the mainline (Mach-based) Hurd implementation. + + -- cgit v1.2.3 From bd54087dd78ef166297769fe2e8be34e1cc89c48 Mon Sep 17 00:00:00 2001 From: "http://arnebab.livejournal.com/" Date: Wed, 16 Mar 2011 09:53:27 +0000 Subject: Added link to Arch Hurd LiveCD --- hurd/running/live_cd.mdwn | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/hurd/running/live_cd.mdwn b/hurd/running/live_cd.mdwn index 42d85c5b..c9360594 100644 --- a/hurd/running/live_cd.mdwn +++ b/hurd/running/live_cd.mdwn @@ -1,4 +1,6 @@ -You can download a gzipped iso of a Hurd Live CD at +[[Arch Hurd|hurd/running/arch_hurd/]] offers Hurd LiveCDs at . + +Also you can download a gzipped iso of a Hurd Live CD at . The Superunpriveleged crew also offers a tiny Hurd Live CD that is under 10 -- cgit v1.2.3 From d6807eea355a50cedddb778d2a732cba8316128d Mon Sep 17 00:00:00 2001 From: "http://arnebab.livejournal.com/" Date: Wed, 16 Mar 2011 10:10:52 +0000 Subject: create qemu: Link to the d-i images --- hurd/running/qemu.mdwn | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/hurd/running/qemu.mdwn b/hurd/running/qemu.mdwn index c5b2cf1c..141ab2b1 100644 --- a/hurd/running/qemu.mdwn +++ b/hurd/running/qemu.mdwn @@ -27,6 +27,12 @@ Also you can use qemu to easily try one of our [[Hurd_LiveCDs|hurd/running/live_ # What is Needed to create a QEMU image +## Debian Installer + +Instructions for creating a qemu image from the install CDs from debian installer can be found in the README alongside the d-i Hurd images: + +## Old method + 1. First thing is to install [[/QEMU]]. 2. A [[grub]] boot disk for the floppy disk image needed for booting. The [0\.97 version](ftp://alpha.gnu.org/gnu/grub/grub-0.97-i386-pc.ext2fs) works fine. I downloaded it and renamed to `floppy.img`. Alternatively, the Debian grub-disk package (up till version 0.97-28) is fine as well. 3. You will need a [Debian/Hurd installation CD](http://www.debian.org/ports/hurd/hurd-cd). K16 works fine. -- cgit v1.2.3 From 01c30100822bdf5cf3312733b75b487212af6592 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide Date: Thu, 24 Mar 2011 20:07:16 +0100 Subject: tar cf → tar cvf MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn b/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn index d55527a7..006966e9 100644 --- a/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn +++ b/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn @@ -87,7 +87,7 @@ For that I want to use: cd ../.. mkdir test settrans -c test2 /usr/local/bin/nsmux test - tar -cf test/intro.tar repos/hurd_intro + tar -cvf test/intro.tar repos/hurd_intro ls test2/intro.tar,,tarfs ### clisp -- cgit v1.2.3 From d6f55e4eddd3ea30de9e4685059264cc4b3f21d8 Mon Sep 17 00:00:00 2001 From: Arne Babenhauserheide Date: Thu, 24 Mar 2011 20:40:34 +0100 Subject: working nsmux test with the hello translator --- community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn b/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn index 006966e9..00d09094 100644 --- a/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn +++ b/community/weblogs/ArneBab/Hurd-showcase-qemu-image.mdwn @@ -56,7 +56,9 @@ For that I want to use: ### Tarfs apt-get install zip libz-dev libbz2-dev - hg clone http://bitbucket.org/ArneBab/hurd-tarfs tarfs + git clone git://git.sv.gnu.org/hurd/incubator.git tarfs + cd tarfs/ + git checkout tarfs/master cd tarfs make make install @@ -86,9 +88,12 @@ For that I want to use: cd ../.. mkdir test - settrans -c test2 /usr/local/bin/nsmux test - tar -cvf test/intro.tar repos/hurd_intro - ls test2/intro.tar,,tarfs + touch test/hello + settrans -ca test2 /usr/local/bin/nsmux test + # tar -cvf test/intro.tar repos/hurd_intro + cat test2/hello + cat test2/hello,,hello + # Hello, World! ### clisp -- cgit v1.2.3 From 2462ca1a8be02689b5e0c5a13038472ad29a7888 Mon Sep 17 00:00:00 2001 From: antrik Date: Wed, 16 Mar 2011 03:00:08 +0100 Subject: gsoc/project_ideas: Restore all project ideas here While there is certainly some overlap with other areas, it is *not* acceptable to drop mentors and exercises from GSoC tasks, nor to add random crap, nor do any other changes that make them less useful as GSoC tasks -- and this is *not* obvious if they do not live in the project_ideas namespace. It's also confusing in general. I tried to preserve all valid changes to the task descriptions themself -- though I might have messed up some things. I did leave the now redundant entries in open_tasks in place. Not sure how to deal with them. As the content is virtually identical anyways, they probably should be just turned into stubs pointing here. Or don't list them explicitely at all -- we point out in other places that GSoC ideas are useful in other contexts too... For the future, please refrain from reorganising things here without prior discussion. --- community/gsoc/project_ideas.mdwn | 6 +- .../gsoc/project_ideas/disk_io_performance.mdwn | 48 +++++++++++++ community/gsoc/project_ideas/dtrace.mdwn | 49 +++++++++++++ .../gsoc/project_ideas/libdiskfs_locking.mdwn | 47 +++++++++++++ community/gsoc/project_ideas/procfs.mdwn | 45 ++++++++++++ community/gsoc/project_ideas/valgrind.mdwn | 81 ++++++++++++++++++++++ 6 files changed, 273 insertions(+), 3 deletions(-) create mode 100644 community/gsoc/project_ideas/disk_io_performance.mdwn create mode 100644 community/gsoc/project_ideas/dtrace.mdwn create mode 100644 community/gsoc/project_ideas/libdiskfs_locking.mdwn create mode 100644 community/gsoc/project_ideas/procfs.mdwn create mode 100644 community/gsoc/project_ideas/valgrind.mdwn diff --git a/community/gsoc/project_ideas.mdwn b/community/gsoc/project_ideas.mdwn index da247847..030c18f3 100644 --- a/community/gsoc/project_ideas.mdwn +++ b/community/gsoc/project_ideas.mdwn @@ -89,10 +89,10 @@ Here are these pages' content: [[!inline pages="community/gsoc/project_ideas/server_overriding" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/tcp_ip_stack" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/nfs" show=0 feeds=no actions=yes]] -[[!inline pages="open_issues/locking" show=0 feeds=no actions=yes]] +[[!inline pages="community/gsoc/project_ideas/libdiskfs_locking" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/pthreads" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/sound" show=0 feeds=no actions=yes]] -[[!inline pages="open_issues/performance/io_system" show=0 feeds=no actions=yes]] +[[!inline pages="community/gsoc/project_ideas/disk_io_performance" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/vm_tuning" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/mtab" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/gnumach_cleanup" show=0 feeds=no actions=yes]] @@ -114,4 +114,4 @@ Here are these pages' content: [[!inline pages="open_issues/unit_testing" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/libcap" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/xattr" show=0 feeds=no actions=yes]] -[[!inline pages="open_issues/valgrind" show=0 feeds=no actions=yes]] +[[!inline pages="community/gsoc/project_ideas/valgrind" show=0 feeds=no actions=yes]] diff --git a/community/gsoc/project_ideas/disk_io_performance.mdwn b/community/gsoc/project_ideas/disk_io_performance.mdwn new file mode 100644 index 00000000..b6f223c8 --- /dev/null +++ b/community/gsoc/project_ideas/disk_io_performance.mdwn @@ -0,0 +1,48 @@ +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Disk I/O Performance Tuning"]] + +The most obvious reason for the Hurd feeling slow compared to mainstream +systems like GNU/Linux, is a low I/O system performance, in particular very +slow hard disk access. + +The reason for this slowness is lack and/or bad implementation of common +optimization techniques, like scheduling reads and writes to minimize head +movement; effective block caching; effective reads/writes to partial blocks; +[[reading/writing multiple blocks at once|clustered_page_faults]]; and +[[read-ahead]]. The +[[ext2_filesystem_server|hurd/translator/ext2fs]] might also need some +optimizations at a higher logical level. + +The goal of this project is to analyze the current situation, and implement/fix +various optimizations, to achieve significantly better disk performance. It +requires understanding the data flow through the various layers involved in +disk access on the Hurd ([[filesystem|hurd/virtual_file_system]], +[[pager|hurd/libpager]], driver), and general experience with +optimizing complex systems. That said, the killing feature we are definitely +missing is the [[read-ahead]], and even a very simple implementation would bring +very big performance speedups. + +Here are some real testcases: + + * [[binutils_ld_64ksec]]; + + * running the Git testsuite which is mostly I/O bound; + + * use [[TopGit]] on a non-toy repository. + + +Possible mentors: Samuel Thibault (youpi) + +Exercise: Look through all the code involved in disk I/O, and try something +easy to improve. It's quite likely though that you will find nothing obvious -- +in this case, please contact us about a different exercise task. diff --git a/community/gsoc/project_ideas/dtrace.mdwn b/community/gsoc/project_ideas/dtrace.mdwn new file mode 100644 index 00000000..252c303a --- /dev/null +++ b/community/gsoc/project_ideas/dtrace.mdwn @@ -0,0 +1,49 @@ +[[!meta copyright="Copyright © 2008, 2009, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="dtrace Support"]] + +One of the main problems of the current Hurd implementation is very poor +[[performance]]. While we have a bunch of ideas what could cause the performance +problems, these are mostly just guesses. Better understanding what really +causes bad performance is necessary to improve the situation. + +For that, we need tools for performance measurements. While all kinds of more +or less specific [[profiling]] tools could be conceived, the most promising and +generic approach seems to be a framework for logging certain events in the +running system (both in the microkernel and in the Hurd servers). This would +allow checking how much time is spent in certain modules, how often certain +situations occur, how things interact, etc. It could also prove helpful in +debugging some issues that are otherwise hard to find because of complex +interactions. + +The most popular framework for that is Sun's dtrace; but there might be others. +The student has to evaluate the existing options, deciding which makes most +sense for the Hurd; and implement that one. (Apple's implementation of dtrace +in their Mach-based kernel might be helpful here...) + +This project requires ability to evaluate possible solutions, and experience +with integrating existing components as well as low-level programming. + +Possible mentors: Samuel Thibault (youpi) + +Related: [[profiling]], [[LTTng]], [[SystemTap]] + +Exercise: In lack of a good exercise directly related to this task, just pick +one of the kernel-related or generally low-level tasks from the bug/task +trackers on savannah, and make a go at it. You might not be able to finish the +task in a limited amount of time, but you should at least be able to make a +detailed analysis of the issue. + +*Status*: Andei Barbu was working on +[SystemTap](http://csclub.uwaterloo.ca/~abarbu/hurd/) for GSoC 2008, but it +turned out too Linux-specific. He implemented kernel probes, but there is no +nice frontend yet. diff --git a/community/gsoc/project_ideas/libdiskfs_locking.mdwn b/community/gsoc/project_ideas/libdiskfs_locking.mdwn new file mode 100644 index 00000000..0e38b6a0 --- /dev/null +++ b/community/gsoc/project_ideas/libdiskfs_locking.mdwn @@ -0,0 +1,47 @@ +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Fix libdiskfs Locking Issues"]] + +Every now and then, new locking issues are discovered in +[[hurd/libdiskfs]] or [[hurd/translator/ext2fs]], for example. Nowadays +these in fact seem to be the most often encountered cause of Hurd crashes +/ lockups. + +One of these could be traced +recently, and turned out to be a lock inside [[hurd/libdiskfs]] that was taken +and not released in some cases. There is reason to believe that there are more +faulty paths causing these lockups. + +The task is systematically checking the [[hurd/libdiskfs]] code for this kind of locking +issues. To achieve this, some kind of test harness has to be implemented: For +example instrumenting the code to check locking correctness constantly at +runtime. Or implementing a [[unit testing]] framework that explicitly checks +locking in various code paths. (The latter could serve as a template for +implementing unit tests in other parts of the Hurd codebase...) + +(A [[systematic code review|security]] would probably suffice to find the +existing locking +issues; but it wouldn't document the work in terms of actual code produced, and +thus it's not suitable for a GSoC project...) + +This task requires experience with debugging locking issues in +[[multithreaded|multithreading]] applications. + +Tools have been written for automated [[code analysis]]; these can help to +locate and fix such errors. + +Possible mentors: Samuel Thibault (youpi) + +Exercise: If you could actually track down and fix one of the existing locking +errors before the end of the application process, that would be excellent. This +might be rather tough though, so probably you need to talk to us about an +alternative exercise task... diff --git a/community/gsoc/project_ideas/procfs.mdwn b/community/gsoc/project_ideas/procfs.mdwn new file mode 100644 index 00000000..d4760aae --- /dev/null +++ b/community/gsoc/project_ideas/procfs.mdwn @@ -0,0 +1,45 @@ +[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled +[[GNU Free Documentation License|/fdl]]."]]"""]] + +[[!meta title="procfs"]] + +Although there is no standard (POSIX or other) for the layout of the `/proc` +pseudo-filesystem, it turned out a very useful facility in GNU/Linux and other +systems, and many tools concerned with process management use it. (`ps`, `top`, +`htop`, `gtop`, `killall`, `pkill`, ...) + +Instead of porting all these tools to use [[hurd/libps]] (Hurd's official method for +accessing process information), they could be made to run out of the box, by +implementing a Linux-compatible `/proc` filesystem for the Hurd. + +The goal is to implement all `/proc` functionality needed for the various process +management tools to work. (On Linux, the `/proc` filesystem is used also for +debugging purposes; but this is highly system-specific anyways, so there is +probably no point in trying to duplicate this functionality as well...) + +The [[existing_partially_working_procfs_implementation|hurd/translator/procfs]] +can serve as a starting point, but needs to be largely rewritten. (It should +use [[hurd/libnetfs]] rather than [[hurd/libtrivfs]]; the data format needs to +change to be more Linux-compatible; and it needs adaptation to newer system +interfaces.) + +This project requires learning [[hurd/translator]] programming, and +understanding some of the internals of process management in the Hurd. It +should not be too hard coding-wise; and the task is very nicely defined by the +existing Linux `/proc` interface -- no design considerations necessary. + +**Note**: We already have several applications for this task. + +Possible mentors: Olaf Buddenhagen (antrik) + +Exercise: Add or fix one piece in the existing procfs translator. + +*Status*: Madhusudan.C.S has implemented a new, fully functional [[procfs|madhusudancs]] for +GSoC 2008. He is still working on some outstanding issues. diff --git a/community/gsoc/project_ideas/valgrind.mdwn b/community/gsoc/project_ideas/valgrind.mdwn new file mode 100644 index 00000000..79c8bd1d --- /dev/null +++ b/community/gsoc/project_ideas/valgrind.mdwn @@ -0,0 +1,81 @@ +[[!meta copyright="Copyright © 2009, 2010, 2011 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Porting Valgrind to the Hurd"]] + +[Valgrind](http://valgrind.org/) is an extremely useful debugging tool for memory errors. +(And some other kinds of hard-to-find errors too.) +Aside from being useful for program development in general, +a Hurd port will help finding out why certain programs segfault on the Hurd, +although they work on Linux. +Even more importantly, it will help finding bugs in the Hurd servers themselfs. + +To keep track of memory use, +Valgrind however needs to know how each [[system call]] affects the validity of memory regions. +This knowledge is highly kernel-specific, +and thus Valgrind needs to be explicitely ported for every system. + +Such a port involves two major steps: +making Valgrind understand how kernel traps work in general on the system in question; +and how all the individual kernel calls affect memory. +The latter step is where most of the work is, +as the behaviour of each single [[system call]] needs to be described. + +Compared to Linux, +[[microkernel/Mach]] (the microkernel used by the Hurd) has very few kernel traps. +Almost all [[system call]]s are implemented as [[RPC]]s instead -- +either handled by Mach itself, or by the various [[Hurd servers|hurd/translator]]. +All RPCs use a pair of `mach_msg` invocations: +one to send a request message, and one to receive a reply. +However, while all RPCs use the same `mach_msg` trap, +the actual effect of the call varies greatly depending on which RPC is invoked -- +similar to the `ioctl` call on Linux. +Each request thus must be handled individually. + +Unlike `ioctl`, +the RPC invocations have explicit type information for the parameters though, +which can be retrieved from the message header. +By analyzing the parameters of the RPC reply message, +Valgrind can know exactly which memory regions are affected by that call, +even without specific knowledge of the RPC in question. +Thus implementing a general parser for the reply messages +will already give Valgrind a fairly good approximation of memory validity -- +without having to specify the exact semantic of each RPC by hand. + +While this should make Valgrind quite usable on the Hurd already, it's not perfect: +some RPCs might return a buffer that is only partially filled with valid data; +or some reply parameters might be optional, +and only contain valid data under certain conditions. +Such specific semantics can't be deduced from the message headers alone. +Thus for a complete port, +it will still be necessary to go through the list of all known RPCs, +and implement special handling in Valgrind for those RPCs that need it. + +The goal of this task is at minimum to make Valgrind grok Mach traps, +and to implement the generic RPC handler. +Ideally, specific handling for RPCs needing it should also be implemented. + +Completing this project will require digging into Valgrind's handling of [[system call]]s, +and into Hurd RPCs. +It is not an easy task, but a fairly predictable one -- +there shouldn't be any unexpected difficulties, +and no major design work is necessary. +It doesn't require any specific previous knowledge: +only good programming skills in general. +On the other hand, +the student will obtain a good understanding of Hurd RPCs while working on this task, +and thus perfect qualifications for Hurd development in general :-) + +Possible mentors: Samuel Thibault (youpi) + +Exercise: As a starter, +students can try to teach valgrind a couple of Linux ioctls, +as this will make them learn how to use the read/write primitives of valgrind. -- cgit v1.2.3 From 70d256d2bd3fea95898383e7bcf6e90442cb6069 Mon Sep 17 00:00:00 2001 From: antrik Date: Wed, 16 Mar 2011 03:14:36 +0100 Subject: gsoc/ideas/diskfs_locking: Revert wording change The new wording actually changes the meaning, to one which is wrong. The most frequent cause of Hurd hangs are probably ext2fs hangs; but they are *not* necessarily all caused by locking problems. (Actually most of them used to be caused by exec issues at some point... Though I'm not sure about the current situation.) Making it less diskfs-specific is probably useful though. This needs some further consideration. --- community/gsoc/project_ideas/libdiskfs_locking.mdwn | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/community/gsoc/project_ideas/libdiskfs_locking.mdwn b/community/gsoc/project_ideas/libdiskfs_locking.mdwn index 0e38b6a0..74937389 100644 --- a/community/gsoc/project_ideas/libdiskfs_locking.mdwn +++ b/community/gsoc/project_ideas/libdiskfs_locking.mdwn @@ -11,12 +11,8 @@ License|/fdl]]."]]"""]] [[!meta title="Fix libdiskfs Locking Issues"]] -Every now and then, new locking issues are discovered in -[[hurd/libdiskfs]] or [[hurd/translator/ext2fs]], for example. Nowadays -these in fact seem to be the most often encountered cause of Hurd crashes -/ lockups. - -One of these could be traced +Nowadays the most often encountered cause of Hurd crashes seems to be lockups +in the [[hurd/translator/ext2fs]] server. One of these could be traced recently, and turned out to be a lock inside [[hurd/libdiskfs]] that was taken and not released in some cases. There is reason to believe that there are more faulty paths causing these lockups. -- cgit v1.2.3 From 99b52de704d58fd8bb6c5c65d500fb9812c8f873 Mon Sep 17 00:00:00 2001 From: antrik Date: Wed, 16 Mar 2011 03:29:06 +0100 Subject: gsoc/ideas: Add back () to function names IMHO it's clearer that way -- especially in browsers that don't display all text attributes, i.e. all text mode browsers. --- community/gsoc/project_ideas/secure_chroot.mdwn | 2 +- community/gsoc/project_ideas/valgrind.mdwn | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/community/gsoc/project_ideas/secure_chroot.mdwn b/community/gsoc/project_ideas/secure_chroot.mdwn index 57739861..bfaf330b 100644 --- a/community/gsoc/project_ideas/secure_chroot.mdwn +++ b/community/gsoc/project_ideas/secure_chroot.mdwn @@ -12,7 +12,7 @@ License|/fdl]]."]]"""]] [[!meta title="Secure chroot Implementation"]] As the Hurd attempts to be (almost) fully [[UNIX]]-compatible, it also implements a -`chroot` [[system call]]. However, the current implementation is not really +`chroot()` [[system call]]. However, the current implementation is not really good, as it allows easily escaping the `chroot`, for example by use of [[passive_translators|hurd/translator]]. diff --git a/community/gsoc/project_ideas/valgrind.mdwn b/community/gsoc/project_ideas/valgrind.mdwn index 79c8bd1d..5fa748ff 100644 --- a/community/gsoc/project_ideas/valgrind.mdwn +++ b/community/gsoc/project_ideas/valgrind.mdwn @@ -33,14 +33,14 @@ Compared to Linux, [[microkernel/Mach]] (the microkernel used by the Hurd) has very few kernel traps. Almost all [[system call]]s are implemented as [[RPC]]s instead -- either handled by Mach itself, or by the various [[Hurd servers|hurd/translator]]. -All RPCs use a pair of `mach_msg` invocations: +All RPCs use a pair of `mach_msg()` invocations: one to send a request message, and one to receive a reply. -However, while all RPCs use the same `mach_msg` trap, +However, while all RPCs use the same `mach_msg()` trap, the actual effect of the call varies greatly depending on which RPC is invoked -- -similar to the `ioctl` call on Linux. +similar to the `ioctl()` call on Linux. Each request thus must be handled individually. -Unlike `ioctl`, +Unlike `ioctl()`, the RPC invocations have explicit type information for the parameters though, which can be retrieved from the message header. By analyzing the parameters of the RPC reply message, -- cgit v1.2.3 From 39692274c2a2a4fdab1ad49e37ec8088002b0caf Mon Sep 17 00:00:00 2001 From: antrik Date: Wed, 16 Mar 2011 03:40:12 +0100 Subject: gsoc/ideas: Drop autogenerated link list I don't see the point of having this redundant list. What's more, the autogenerated list is problematic for a number of reasons -- which is why I had switched to manually listing each task in the first place... --- community/gsoc/project_ideas.mdwn | 7 ------- 1 file changed, 7 deletions(-) diff --git a/community/gsoc/project_ideas.mdwn b/community/gsoc/project_ideas.mdwn index 030c18f3..a574192a 100644 --- a/community/gsoc/project_ideas.mdwn +++ b/community/gsoc/project_ideas.mdwn @@ -76,13 +76,6 @@ will assist you as well as we can. See also the list of [Hurd-related X.org project ideas](http://wiki.x.org/wiki/Hurd_Porting). -Here is a list of what we consider suitable tasks: - -[[!map show=title pages="(community/gsoc/project_ideas/* and - !community/gsoc/project_ideas/*/*) or tagged(gsoc-task)"]] - -Here are these pages' content: - [[!inline pages="community/gsoc/project_ideas/language_bindings" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/virtualization" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/file_locking" show=0 feeds=no actions=yes]] -- cgit v1.2.3 From e2c45c18055f6bd2613dc2bd694e70e806e15698 Mon Sep 17 00:00:00 2001 From: antrik Date: Wed, 16 Mar 2011 03:50:17 +0100 Subject: gsoc/ideas: Drop locking validity task for now It's probably too vague. Not sure it can be made any better -- perhaps just mention this aspect in the testing framework task instead?... --- community/gsoc/project_ideas.mdwn | 1 - 1 file changed, 1 deletion(-) diff --git a/community/gsoc/project_ideas.mdwn b/community/gsoc/project_ideas.mdwn index a574192a..12074168 100644 --- a/community/gsoc/project_ideas.mdwn +++ b/community/gsoc/project_ideas.mdwn @@ -82,7 +82,6 @@ See also the list of [Hurd-related X.org project ideas](http://wiki.x.org/wiki/H [[!inline pages="community/gsoc/project_ideas/server_overriding" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/tcp_ip_stack" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/nfs" show=0 feeds=no actions=yes]] -[[!inline pages="community/gsoc/project_ideas/libdiskfs_locking" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/pthreads" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/sound" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/disk_io_performance" show=0 feeds=no actions=yes]] -- cgit v1.2.3 From 0729272db4fe4563fee46488236d75e9c4700eee Mon Sep 17 00:00:00 2001 From: antrik Date: Thu, 24 Mar 2011 22:55:40 +0100 Subject: gsoc/ideas: Proper task description for "automated testing" --- community/gsoc/project_ideas.mdwn | 2 +- .../gsoc/project_ideas/testing_framework.mdwn | 55 ++++++++++++++++++++++ 2 files changed, 56 insertions(+), 1 deletion(-) create mode 100644 community/gsoc/project_ideas/testing_framework.mdwn diff --git a/community/gsoc/project_ideas.mdwn b/community/gsoc/project_ideas.mdwn index 12074168..8d31ed7f 100644 --- a/community/gsoc/project_ideas.mdwn +++ b/community/gsoc/project_ideas.mdwn @@ -103,7 +103,7 @@ See also the list of [Hurd-related X.org project ideas](http://wiki.x.org/wiki/H [[!inline pages="community/gsoc/project_ideas/cdparanoia" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/perl_python" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/testsuites" show=0 feeds=no actions=yes]] -[[!inline pages="open_issues/unit_testing" show=0 feeds=no actions=yes]] +[[!inline pages="community/gsoc/project_ideas/testing_framework" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/libcap" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/xattr" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/valgrind" show=0 feeds=no actions=yes]] diff --git a/community/gsoc/project_ideas/testing_framework.mdwn b/community/gsoc/project_ideas/testing_framework.mdwn new file mode 100644 index 00000000..0448bc6b --- /dev/null +++ b/community/gsoc/project_ideas/testing_framework.mdwn @@ -0,0 +1,55 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled +[[GNU Free Documentation License|/fdl]]."]]"""]] + +[[!meta title="Automated Testing Framework"]] + +Hurd development would benefit greatly from automated tests. +Unit tests should be added for all the major components +(Mach; Hurd servers and libraries). +Also, functional tests can be performed on various levels: +Mach; individual servers; and interaction of several servers. + +(The highest level would actually be testing libc functionality, +which in turn uses the various Hurd mechanisms. +glibc already comes with a test suite -- +though it might be desirabe to add some extra tests +for exercising specific aspects of the Hurd...) + +Our page on [[automated testing|open_issues/unit_testing]] collects some relevant material. + +The Goal of this task is to provide testing frameworks +that allow automatically running tests +as part of the Hurd and Mach build processes. +The student will have to create the necessary infrastrucure, +and a couple of sample tests employing it. +Ideally, all the aspects mentioned above should be covered. +At least some have to be ready for use and upstream merging +before the end of the summer. + +(As a bonus, +in addition to these explicit tests, +it would be helpful to integrate some methods +for testing [[locking validity|open_issues/locking]], +performing static code analysis etc.) + +This task probably requires some previous experience +with unit testing of C programs, +as well as dealing with complex build systems. +No in-depth knowledge about any specific parts of the Hurd should be necessary, +but some general understanding of the Hurd architecture +will have to be aquired to complete this project. +This makes it a very good project +to get started on Hurd development :-) + +Possible mentors: ? + +Exercise: Create a program performing some simple test(s) on the Hurd or Mach code. +It doesn't need to be integrated in the build process yet -- +a standalone progrem with it's own Makefile is fine for now. -- cgit v1.2.3 From 8ee6975b7c1c0557d21ca966c24ea09806104498 Mon Sep 17 00:00:00 2001 From: antrik Date: Fri, 25 Mar 2011 00:07:28 +0100 Subject: gsoc/ideas/driver_glue_code: Update and add back As zhengda seems MIA, this task is up for takers again. Update the text to reflect the fact that zhengda already did a lot of the DDE work. (Actually, the text is almost entirely rewritten...) Probably this should be split into several distinct tasks for the various aspects that need work. --- community/gsoc/project_ideas.mdwn | 1 + community/gsoc/project_ideas/driver_glue_code.mdwn | 84 ++++++++++++++-------- 2 files changed, 54 insertions(+), 31 deletions(-) diff --git a/community/gsoc/project_ideas.mdwn b/community/gsoc/project_ideas.mdwn index 8d31ed7f..b4b60dd6 100644 --- a/community/gsoc/project_ideas.mdwn +++ b/community/gsoc/project_ideas.mdwn @@ -107,3 +107,4 @@ See also the list of [Hurd-related X.org project ideas](http://wiki.x.org/wiki/H [[!inline pages="community/gsoc/project_ideas/libcap" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/xattr" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/valgrind" show=0 feeds=no actions=yes]] +[[!inline pages="community/gsoc/project_ideas/driver_glue_code" show=0 feeds=no actions=yes]] diff --git a/community/gsoc/project_ideas/driver_glue_code.mdwn b/community/gsoc/project_ideas/driver_glue_code.mdwn index 9c063e9f..65ea4f0f 100644 --- a/community/gsoc/project_ideas/driver_glue_code.mdwn +++ b/community/gsoc/project_ideas/driver_glue_code.mdwn @@ -1,5 +1,4 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -9,41 +8,64 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -[[!meta title="New Driver Glue Code"]] +[[!meta title="New Driver Framework"]] -Although a driver framework in user space would be desirable, presently the Hurd -uses kernel drivers in the microkernel, -[[GNU_Mach|microkernel/mach/gnumach]]. (And changing this would be far beyond a -GSoC project...) - -The problem is that the drivers in GNU Mach are presently old Linux drivers -(mostly from 2.0.x) accessed through a glue code layer. This is not an ideal -solution, but works quite OK, except that the drivers are very old. The goal of -this project is to redo the glue code, so we can use drivers from current Linux -versions, or from one of the free BSD variants. - -While it would be certainly possible to create custom glue code again, a more -sustainable and probably also easier approach is to use -[[DDE]] instead -- it already -does the hard work of providing an environment where the foreign drivers can -run, and has the additional advantage of being externally maintained. - -This is a doable, but pretty involved project. Previous experience with driver -programming probably is a must. (No Hurd-specific knowledge is required, -though.) +The Hurd presently uses hardware drivers +implemented in the microkernel, [[GNU_Mach|microkernel/mach/gnumach]]. +These drivers are old Linux drivers (mostly from 2.0.x) +accessed through a glue code layer. +This is not an ideal solution, but works quite OK, +except that the drivers are extremely old by now. +Thus we need a new framework, +so we can use drivers from current Linux versions instead, +or perhaps from one of the free BSD variants. This is [[!GNU_Savannah_task 5488]]. [[open issues/user-space device drivers]]. [[open issues/device drivers and io systems]]. +The most promising approach for getting newer drivers seems to be [[DDE]]: +it already does the hard work of providing an environment +where the foreign drivers can run, +and offers the additional benefit of being externally maintained. +DDE also offers the necessary facilities +for running all drivers in separate userspace processes, +which is more desirable than drivers running in the microkernel. -Possible mentors: Samuel Thibault (youpi) +[[Zheng Da|zhengda]] has already done considerable work on this. +The basic framework for using DDE in the Hurd is present, +and network card drivers are already working very well. +However, this work isn't fully integrated in the Hurd yet. +The additional kernel interfaces that were created for this +are still prototypes, and will need to be reworked. +Also, there is no build system for automatically compiling +all Linux network card drivers in one go. + +Other types of drivers are missing so far. +Support for IDE drivers has been partially implemented, +but isn't fully working yet. +To fully replace the old in-kernel drivers, +further infrastructure will be necessary +to make userspace disk drivers usable for the root filesystem. -Exercise: Take a driver for some newer piece of hardware (e.g. Intel e1000 -ethernet) from a recent system, and try to port it to run in the existing -driver framework in GNU Mach. Completing the port might be too involved for the -exercise; but it's pretty likely that you will find something else to improve -in the glue code while working on this... +Some other subsystems are missing or incomplete in DDE itself, +and will require additional work that is not specific to the Hurd implementation. + +The goal of this task is to fix at least one of the mentioned major shortcomings: +rework the kernel interfaces; +provide a streamlined build system for the drivers; +finish IDE support; +or implement support for some other subsystem. + + +This is a doable, but pretty involved project. +Previous experience with driver programming probably is a must. +To be able to work on the framework, +the student will also have to get a good understanding of certain aspects of Hurd, +such as memory management for example. + +Possible mentors: Samuel Thibault (youpi) -*Status*: [[Zheng Da|zhengda]] is working on DDE, and has mostly completed the -initial port. +Exercise: Get one of the not yet integrated Linux network card drivers to work. +(Note: This should be straightforward, +once you have the framework properly built and set up...) -- cgit v1.2.3 From b2958678e344de739216ec3769e770fb3b9a524f Mon Sep 17 00:00:00 2001 From: antrik Date: Fri, 25 Mar 2011 10:15:46 +0100 Subject: gsoc/ideas/dtrace: fix broken links --- community/gsoc/project_ideas/dtrace.mdwn | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/community/gsoc/project_ideas/dtrace.mdwn b/community/gsoc/project_ideas/dtrace.mdwn index 252c303a..ebb63d48 100644 --- a/community/gsoc/project_ideas/dtrace.mdwn +++ b/community/gsoc/project_ideas/dtrace.mdwn @@ -12,12 +12,12 @@ License|/fdl]]."]]"""]] [[!meta title="dtrace Support"]] One of the main problems of the current Hurd implementation is very poor -[[performance]]. While we have a bunch of ideas what could cause the performance +[[open_issues/performance]]. While we have a bunch of ideas what could cause the performance problems, these are mostly just guesses. Better understanding what really causes bad performance is necessary to improve the situation. For that, we need tools for performance measurements. While all kinds of more -or less specific [[profiling]] tools could be conceived, the most promising and +or less specific [[open_issues/profiling]] tools could be conceived, the most promising and generic approach seems to be a framework for logging certain events in the running system (both in the microkernel and in the Hurd servers). This would allow checking how much time is spent in certain modules, how often certain -- cgit v1.2.3 From ee947ab4eff2bda9bc6409ed69f6f68b5164602e Mon Sep 17 00:00:00 2001 From: antrik Date: Fri, 25 Mar 2011 10:20:54 +0100 Subject: gsoc/ideas/dtrace: Update and add back Doesn't look like Andrei will ever continue work on this, so it's up for takers again. Mention previous work, and update task accordingly. --- community/gsoc/project_ideas.mdwn | 1 + community/gsoc/project_ideas/dtrace.mdwn | 51 +++++++++++++++++++------------- 2 files changed, 32 insertions(+), 20 deletions(-) diff --git a/community/gsoc/project_ideas.mdwn b/community/gsoc/project_ideas.mdwn index b4b60dd6..0c249b62 100644 --- a/community/gsoc/project_ideas.mdwn +++ b/community/gsoc/project_ideas.mdwn @@ -108,3 +108,4 @@ See also the list of [Hurd-related X.org project ideas](http://wiki.x.org/wiki/H [[!inline pages="community/gsoc/project_ideas/xattr" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/valgrind" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/driver_glue_code" show=0 feeds=no actions=yes]] +[[!inline pages="community/gsoc/project_ideas/dtrace" show=0 feeds=no actions=yes]] diff --git a/community/gsoc/project_ideas/dtrace.mdwn b/community/gsoc/project_ideas/dtrace.mdwn index ebb63d48..f70598ca 100644 --- a/community/gsoc/project_ideas/dtrace.mdwn +++ b/community/gsoc/project_ideas/dtrace.mdwn @@ -9,7 +9,7 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -[[!meta title="dtrace Support"]] +[[!meta title="Kernel Instrumentation"]] One of the main problems of the current Hurd implementation is very poor [[open_issues/performance]]. While we have a bunch of ideas what could cause the performance @@ -25,25 +25,36 @@ situations occur, how things interact, etc. It could also prove helpful in debugging some issues that are otherwise hard to find because of complex interactions. -The most popular framework for that is Sun's dtrace; but there might be others. -The student has to evaluate the existing options, deciding which makes most -sense for the Hurd; and implement that one. (Apple's implementation of dtrace -in their Mach-based kernel might be helpful here...) - -This project requires ability to evaluate possible solutions, and experience -with integrating existing components as well as low-level programming. +The most popular kernel instrumentation framework is Sun's dtrace, +originally written for Solaris, +but also adopted by some other systems. +However, the GPL-incompatible license means it can't be used in Linux, +and thus Linux developers created their own frameworks instead: +first [[SystemTap]], and now [[LTTng]]. + +In 2008, Andrei Barbu did initial work on kernel probes for the Hurd. +However, not all of his patches got merged, +because some turned out not to be fully functional. +Also, he didn't get around to work on userspace probes, +nor on a nice frontend for writing test scripts employing the probes. + +The goal of this project is to make the instrumentation framework +more usable and complete, +and to better integrate it in the Hurd. +For that, the student will have to work +on some real profiling and/or debugging tasks, +and fix any shortcomings he encounters in the framework. + +This is a pretty involved task. +Previous experience with low-level programming is a must; +and it also requires a good grasp on interactions in complex systems. + +To work on this project, +the student will have to get familiar with GNU Mach. +(The microkernel employed by the Hurd.) +Some understanding of other aspects of the Hurd will also be required, +depending on the exact nature of the profiling/debugging performed. Possible mentors: Samuel Thibault (youpi) -Related: [[profiling]], [[LTTng]], [[SystemTap]] - -Exercise: In lack of a good exercise directly related to this task, just pick -one of the kernel-related or generally low-level tasks from the bug/task -trackers on savannah, and make a go at it. You might not be able to finish the -task in a limited amount of time, but you should at least be able to make a -detailed analysis of the issue. - -*Status*: Andei Barbu was working on -[SystemTap](http://csclub.uwaterloo.ca/~abarbu/hurd/) for GSoC 2008, but it -turned out too Linux-specific. He implemented kernel probes, but there is no -nice frontend yet. +Exercise: Use the existing probes to perform some simple measurement. -- cgit v1.2.3 From 10288350709d006710bcdfb747ba9d1a1208d69b Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 00:27:11 +0100 Subject: history/port_to_another_microkernel: New. Based on other pages. --- faq/which_microkernel.mdwn | 15 ++ history.mdwn | 10 +- history/port_to_another_microkernel.mdwn | 171 +++++++++++++++++++++ .../port_to_another_microkernel/discussion.mdwn | 69 +++++++++ history/port_to_l4.mdwn | 105 +------------ hurd-l4.mdwn | 8 +- hurd.mdwn | 8 +- hurd/faq/which_microkernel.mdwn | 114 -------------- hurd/ng.mdwn | 2 +- microkernel/coyotos.mdwn | 7 +- microkernel/l4.mdwn | 5 +- 11 files changed, 282 insertions(+), 232 deletions(-) create mode 100644 faq/which_microkernel.mdwn create mode 100644 history/port_to_another_microkernel.mdwn create mode 100644 history/port_to_another_microkernel/discussion.mdwn delete mode 100644 hurd/faq/which_microkernel.mdwn diff --git a/faq/which_microkernel.mdwn b/faq/which_microkernel.mdwn new file mode 100644 index 00000000..608e6b3f --- /dev/null +++ b/faq/which_microkernel.mdwn @@ -0,0 +1,15 @@ +[[!meta copyright="Copyright © 2009, 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="What happened with the ports to the L4 / Coyotos / Viengoos +microkernels?"]] + +This story is told on the page about the +[[history/port_to_another_microkernel]]. diff --git a/history.mdwn b/history.mdwn index 8f155b54..0abcbd52 100644 --- a/history.mdwn +++ b/history.mdwn @@ -1,13 +1,13 @@ -[[!meta copyright="Copyright © 1998, 1999, 2001, 2002, 2007, 2008, 2009 Free -Software Foundation, Inc."]] +[[!meta copyright="Copyright © 1998, 1999, 2001, 2002, 2007, 2008, 2009, 2011 +Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!tag stable_URL]] @@ -91,4 +91,4 @@ mailing lists. --- - * [[Port_to_L4]] + * [[Port_to_another_microkernel]] diff --git a/history/port_to_another_microkernel.mdwn b/history/port_to_another_microkernel.mdwn new file mode 100644 index 00000000..b347cf38 --- /dev/null +++ b/history/port_to_another_microkernel.mdwn @@ -0,0 +1,171 @@ +[[!meta copyright="Copyright © 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, +2009, 2010, 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Porting the Hurd to another microkernel"]] + +At first, there was an effort to port the Hurd from the [[Mach +microkernel|microkernel/mach]] to the [[L4_microkernel_family|microkernel/l4]]. +Then the story continued... + +[[!toc levels=2]] + + +# L4 + +## Initial Idea + +Encountering a number of fundamental design issues with the [[Mach +microkernel|microkernel/mach]] (mostly regarding [[resource +management|open_issues/resource_management_problems]]), some of the Hurd +developers began experimenting with using other microkernels for the Hurd +around the turn of the millenium. + +The idea of using L4 as a [[microkernel]] for a Hurd system was initially +voiced in the [[community]] by Okuji Yoshinori, who, for discussing this +purpose, created the [[mailing_lists/l4-hurd]] mailing list in November 2000. + +Over the years, a lot of discussion have been held on this mailing list, which +today is still the right place for [[next-generation Hurd|hurd/ng]] +discussions. + + +## Why? + +Even though that said resource management issues constitute a broad research +topic, there was no hope that the original Mach project would work on these: +[[microkernel/Mach]] wasn't maintained by its original authors anymore. Mach +had served its purpose as a research vehicle, and has been retired by its +stakeholders. + +Thus, switching to a well-maintained current [[microkernel]] was expected to +yield a more solid foundation for a Hurd system than the [[decaying +Mach|microkernel/mach/history]] design and implementation was able to. + +At that time, the [[L4 microkernel family|microkernel/L4]] was one obvious +choice. Being a second-generation microkernel, it was deemed to provide for a +faster system kernel implementation, especially in the time-critical [[IPC]] +paths. Also, as L4 was already implemented for a bunch of different +architectures (x86, Alpha, MIPS; also including SMP support), and the Hurd +itself being rather archtecture-agnostic, it was expected to be able to easily +support more platforms than with the existing system. + + +## Steps and Goals + +At the same time, the idea was -- while mucking with the system's core anyway +-- to improve on some fundamental design issues, too -- like the resource +management problems, for example. + +One goal of porting the Hurd to L4 was to make the Hurd independent of +[[microkernel/Mach]] interfaces, to make it somewhat microkernel-agnostic. + +One idea was to first introduce a Mach-on-L4 emulation layer, to easily get a +usable (though slow) Hurd-using-Mach-interfaces-on-L4 system, and then +gradually move the Hurd servers to use L4 intefaces rather than Mach ones. + +A design upon the lean L4 kernel would finally have made it feasible to move +devices drivers out of the kernel's [[TCB]]. + + +# Implementation + +The project itself then was mostly lead by Marcus Brinkmann and Neal Walfield. +Neal started the original Hurd/L4 port while visiting Karlsruhe university in +2002. He explains: + +> My intention was to adapt the Hurd to exploit L4's concepts and intended +> [[design_pattern]]s; it was not to simply provide a Mach +> [[compatibility_layer]] on top of L4. When I left Karlsruhe, I no longer had +> access to [[microkernel/l4/Pistachio]] as I was unwilling to sign an NDA. +> Although the specification was available, the Karlsruhe group only [released +> their code in May +> 2003](https://lists.ira.uni-karlsruhe.de/pipermail/l4ka/2003-May/000345.html). +> Around this time, Marcus began hacking on Pistachio. He created a relatively +> complete run-time. I didn't really become involved again until the second +> half of 2004, after I complete by Bachelors degree. + +Development of Hurd/L4 was done in the [CVS module +`hurd-l4`](http://savannah.gnu.org/cgi-bin/viewcvs/hurd/hurd-l4/). The `doc` +directory contains a design document that is worth reading for anyone who +wishes to learn more about Hurd/L4. + +Even though there was progress -- see, for example, the [[QEMU image for +L4|hurd/running/qemu/image_for_l4]] -- this port never reached a releasable +state. Simple POSIX programs, such as `banner` could run, but for more complex +system interfaces, a lot more work was needed. + +Eventually, a straight-forward port of the original Hurd's design wasn't deemed +feasible anymore by the developers, partly due to them not cosidering L4 +suitable for implementing a general-purpose operating system on top of it, and +because of deficiencies in the original Hurd's design, which they discovered +along their way. Neal goes on: + +> Before Marcus and I considered [[microkernel/Coyotos]], we had already +> rejected some parts of the Hurd's design. The +> [[open_issues/resource_management_problems]] were +> what prompted me to look at L4. Also, some of the problems with +> [[hurd/translator]]s were already well-known to us. (For a more detailed +> description of the problems we have identified, see our [[hurd/critique]] in the +> 2007 July's SIGOPS OSR. We have also written a forward-looking +> [[hurd/ng/position_paper]].) + +> We visited Jonathan Shapiro at Hopkins in January 2006. This resulted in a +> number of discussions, some quite influential, and not always in a way which +> aligned our position with that of Jonathan's. This was particularly true of +> a number of security issues. + +A lange number of discussion threads can be found in the archives of the +[[mailing_lists/l4-hurd]] mailing list. + +> Hurd-NG, as we originally called it, was an attempt to articulate the system +> that we had come to envision in terms of interfaces and description of the +> system's structure. The new name was selected, if I recall correctly, as it +> clearly wasn't the Hurd nor the Hurd based on L4. + + +## Termination + +As of 2005, development of Hurd/L4 has stopped. + + +# Coyotos + +Following that, an attempt was started to use the kernel of the +[[microkernel/Coyotos]] system. As Coyotos is an object capability system +througout, the microkernel would obviously be more suitable for this purpose; +and it looked pretty promising in the beginning. However, further +investigations found that there are some very fundamental philosophical +differences between the Coyotos and Hurd designs; and thus this this attempt +was also abandonned, around 2006 / 2007. (This time before producing any +actual code.) + + +# Viengoos + +By now (that is, after 2006), there were some new [[microkernel/L4]] variants +available, which added protected [[IPC]] paths and other features necessary for +object capability systems; so it might be possible to implement the Hurd on top +of these. However, by that time the developers concluded that microkernel +design and system design are interconnected in very intricate ways, and thus +trying to use a third-party microkernel will always result in trouble. So Neal +Walfield created the experimental [[microkernel/Viengoos]] kernel instead -- +based on the experience from the previous experiments with L4 and Coyotos -- +for his [[research on resource +management|open_issues/resource_management_problems]]. Currently he works in +another research area though, and thus Viengoos is on hold. + + +# Intermediate Results + +Note that while none of the microkernel work is active now, the previous +experiments already yielded a lot of experience, which will be very useful in +the further development / improvement of the mainline (Mach-based) Hurd +implementation. diff --git a/history/port_to_another_microkernel/discussion.mdwn b/history/port_to_another_microkernel/discussion.mdwn new file mode 100644 index 00000000..f2161195 --- /dev/null +++ b/history/port_to_another_microkernel/discussion.mdwn @@ -0,0 +1,69 @@ +[[!meta copyright="Copyright © 2009, 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +IRC, #hurd, 2011-01-12. + +[[!taglink open_issue_documentation]] + + Hello i am just curious of the development of Hurd - what's the + current mission on the microkernel i see projects like l4 and viengoos, + will one of these projects replace Mach? or will you stick with Mach + as i understand is that Mach is a first generation microkernel + that's very old in design and causes alot of issues + that's where l4 and viengoos comes in - they are trying to be the + next generation Mach - am i correct? + l4 is not a drop in replacement for Mach + it doesn't actually do much resource management + for instance, you still have to implement a memory manager + this is where several issues are with Mach + l4 doesn't address those issues; it punts to the operating system + and what about viengoos? + it's unfinished + and it implemented some untested ideas + i.e., parts of viengoos were research + there has not been a sufficient evaluation of those ideas to + determine whether they are a good approach + meaning that viengoos is a research kernel that could aid Mach? + I'm not sure I understand your question + Well is viengoos trying to be a replacement for Mach, or will + viengoos be an experiment of new ideas that could be implemented in Mach? + i am sorry for my limited english + viengoos was designed with a Hurd-like user-land in mind + in that sense it was a Mach replacement + (unlike L4) + viengoos consisted of a few experiments + one could implement them in mach + but it would require exposing new interfaces + in which case, I'm not sure you could call the result Mach + Well as i understand you develop two microkernels side by side, + wouldnt it be more effective to investigate viengoos more and maybe move + the focus to viengoos? + no + having something working all the time is crucial + it's very hard to motivate people to work on a project that might + be useful, in a couple of years, perhaps... + Well Mach is meant to be replaced one day - i see no reason to + keep on developing it just because it works at this moment + *if Mach is meant to be replaced + it's not at all clear that it will be replaced by something + completely different. I for my part believe that modifying the existing + Mach is a more promising approach + as i understand man power is something you need - and by spreading + out the developers just makes the progress more slow + but even if it *were* to be replaced one day, it doesn't change + the fact that we need it *now* + all software will be obsolete one day. doesn't mean it's not worth + working on + the vast majority of work is not on the microkernel anyways, but + on the system running on top of it + ahh i see + manpower is not something that comes from nowhere. again, having + something working is crucial in a volunteer project like this + there are no fixed plans diff --git a/history/port_to_l4.mdwn b/history/port_to_l4.mdwn index b58c0d91..3f951a64 100644 --- a/history/port_to_l4.mdwn +++ b/history/port_to_l4.mdwn @@ -1,108 +1,13 @@ -[[!meta copyright="Copyright © 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, -2009, 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] -[[!meta title="Porting the Hurd to L4: Hurd/L4"]] +[[!tag stable_URL]] -There was an effort to port the Hurd from [[microkernel/Mach]] to the -[[L4_microkernel_family|microkernel/L4]]. - -The idea of using L4 as a [[microkernel]] for a [[Hurd_system|hurd]] was -initially voiced in the [[Hurd_community|community]] by Okuji Yoshinori, who, -for discussing this purpose, created the [[mailing lists/l4-hurd]] mailing list -in November 2000. - -The project itself then was mostly lead by Marcus Brinkmann and Neal Walfield. -Even though there was progress -- see, for example, the -[[QEMU image for L4|hurd/running/qemu/image for l4]] -- this port never reached a -releasable state. Eventually, a straight-forward port of the original Hurd's -design wasn't deemed feasible anymore by the developers, partly due to them not -cosidering L4 suitable for implementing a general-purpose operating system on -top of it, and because of deficiencies in the original Hurd's design, which -they discovered along their way. Read the [[hurd/critique]] and a -[[hurd/ng/position paper]]. - -By now, the development of Hurd/L4 has stopped. However, Neal Walfield moved -on to working on a newly designed kernel called [[microkernel/viengoos]]. - -Over the years, a lot of discussion have been held on the -[[mailing lists/l4-hurd]] mailing list, which today is still the right place -for [[next-generation Hurd|hurd/ng]] discussions. - -Development of Hurd/L4 was done in the `hurd-l4` module of the Hurd CVS -repository. The `doc` directory contains a design document that is worth -reading for anyone who wishes to learn more about Hurd/L4. - - -One goal of porting the Hurd to L4 was to make the Hurd independend of Mach -interfaces, to make it somewhat microkernel-agnostic. - -Mach wasn't maintained by its original authors anymore, so switching to a -well-maintained current [[microkernel]] was expected to yield a more solid -foundation for a Hurd system than the decaying Mach design and implementation -was able to. - -L4 being a second-generation [[microkernel]] was deemed to provide for a faster -system kernel implementation, especially in the time-critical [[IPC]] paths. -Also, as L4 was already implemented for a bunch of different architectures -(IA32, Alpha, MIPS; SMP), and the Hurd itself being rather archtecture-unaware, -it was expected to be able to easily support more platforms than with the -existing system. - -A design upon the lean L4 kernel would finally have moved devices drivers out -of the kernel's [[TCB]]. - - -One idea was to first introduce a Mach-on-L4 emulation layer, to easily get a -usable (though slow) Hurd-using-Mach-interfaces-on-L4 system, and then -gradually move the Hurd servers to use L4 intefaces rather than Mach ones. - - -Neal Walfield started the original Hurd/L4 port while at Karlsruhe in 2002. He -explains: - -> My intention was to adapt the Hurd to exploit L4's concepts and intended -> [[design_pattern]]s; it was not to simply provide a Mach -> [[compatibility_layer]] on top of L4. When I left Karlsruhe, I no longer had -> access to [[microkernel/l4/Pistachio]] as I was unwilling to sign an NDA. -> Although the specification was available, the Karlsruhe group only [released -> their code in May -> 2003](https://lists.ira.uni-karlsruhe.de/pipermail/l4ka/2003-May/000345.html). -> Around this time, Marcus began hacking on Pistachio. He created a relatively -> complete run-time. I didn't really become involved again until the second -> half of 2004, after I complete by Bachelors degree. - -> Before Marcus and I considered [[microkernel/Coyotos]], we had already -> rejected some parts of the Hurd's design. The -> [[open issues/resource management problems]] were -> what prompted me to look at L4. Also, some of the problems with -> [[hurd/translator]]s were already well-known to us. (For a more detailed -> description of the problems we have identified, see our [[hurd/critique]] in the -> 2007 July's SIGOPS OSR. We have also written a forward-looking -> [[hurd/ng/position paper]].) - -> We visited Jonathan Shapiro at Hopkins in January 2006. This resulted in a -> number of discussions, some quite influential, and not always in a way which -> aligned our position with that of Jonathan's. This was particularly true of -> a number of security issues. - -A lange number of discussion threads can be found in the archives of the -[[mailing lists/l4-hurd]] mailing list. - -> Hurd-NG, as we originally called it, was an attempt to articulate the system -> that we had come to envision in terms of interfaces and description of the -> system's structure. The new name was selected, if I recall correctly, as it -> clearly wasn't the Hurd nor the Hurd based on L4. - - -The source code is still available in [CVS module -`hurd-l4`](http://savannah.gnu.org/cgi-bin/viewcvs/hurd/hurd-l4/) (note that -this repository has in the beginning also been used for Neal's -[[microkernel/Viengoos]]). +[[!meta redir=port_to_another_microkernel]] diff --git a/hurd-l4.mdwn b/hurd-l4.mdwn index 579c1190..afc8f8f3 100644 --- a/hurd-l4.mdwn +++ b/hurd-l4.mdwn @@ -1,13 +1,13 @@ -[[!meta copyright="Copyright © 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2009, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!tag stable_URL]] -[[!meta redir=history/port_to_l4]] +[[!meta redir=history/port_to_another_microkernel]] diff --git a/hurd.mdwn b/hurd.mdwn index 010e9100..b0db8a64 100644 --- a/hurd.mdwn +++ b/hurd.mdwn @@ -1,13 +1,13 @@ [[!meta copyright="Copyright © 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, -2009, 2010 Free Software Foundation, Inc."]] +2009, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] The GNU Hurd is under active development. Because of that, there is no *stable* version. We distribute the Hurd sources only through [[Git|source_repositories]] at present. @@ -31,7 +31,7 @@ in the *unstable* branch of the Debian archive. * [[What_Is_the_GNU_Hurd]] - A Brief Description * [[Advantages]]. And [[challenges]]. * [[History]] - * [[history/Port_to_L4]] + * [[history/Port_to_another_microkernel]] * [[Logo]] * [[Status]] * [[KnownHurdLimits]] diff --git a/hurd/faq/which_microkernel.mdwn b/hurd/faq/which_microkernel.mdwn deleted file mode 100644 index f6225188..00000000 --- a/hurd/faq/which_microkernel.mdwn +++ /dev/null @@ -1,114 +0,0 @@ -[[!meta copyright="Copyright © 2009, 2011 Free Software Foundation, Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -[[!meta title="What happened to the L4 / Coyotos / Viengoos microkernels?"]] - -Encountering a number of fundamental design issues with the Mach microkernel -(mostly with resource management), -some of the Hurd developers began experimenting with using other microkernels for the Hurd. - -The first attempt was reimplementing the Hurd on the L4 (Pistachio) microkernel. -This got going around 2003/2004, -and got pretty far (running some simple POSIX programs, such as "banner"); -however over time some lingering design issues turned out to be fundamental problems: -the original L4 is not suitable for building object capability systems like the Hurd. -Thus development was aborted in 2005. - -Following that, an attempt was started to use the kernel of the Coyotos system. -As Coyotos is an object-capability system througout, -the microkernel would obviously be more suitable for this purpose; -and it looked pretty promising in the beginning. -However, further investigations found -that there are some very fundamental philosophical differences -between the Coyotos and Hurd designs; -and thus this this attempt was also abandonned, around 2006/2007. -(This time before producing any actual code.) - -By now there were some new L4 variants available, -which added protected IPC paths and other features necessary for object capability systems; -so it might be possible to implement the Hurd on top of these. -However, by that time the developers concluded that microkernel design and system design -are interconnected in very intricate ways, -and thus trying to use a third-party microkernel will always result in trouble. -So Neal Walfield created the experimental [[Viengoos|microkernel/viengoos]] kernel instead -- -based on the experience from the previous experiments with L4 and Coyotos -- -for his research on resource management. -Currently he works in another research area though, and thus Viengoos is on hold. - -Note that while none of the microkernel work is active now, -the previous experiments already yielded a lot of experience, -which will be very useful in the further development/improvement -of the mainline (Mach-based) Hurd implementation. - - diff --git a/hurd/ng.mdwn b/hurd/ng.mdwn index de33949d..481386a4 100644 --- a/hurd/ng.mdwn +++ b/hurd/ng.mdwn @@ -13,7 +13,7 @@ This section explains the motivations behind the new design: * [[Issues_with_L4_Pistachio]] * [[Limitations_of_the_original_Hurd_design]] - * History of the [[history/port_to_L4]] + * History of the [[history/port_to_another_microkernel]] # Work already done diff --git a/microkernel/coyotos.mdwn b/microkernel/coyotos.mdwn index 5ecea688..fec023ba 100644 --- a/microkernel/coyotos.mdwn +++ b/microkernel/coyotos.mdwn @@ -1,5 +1,5 @@ -[[!meta copyright="Copyright © 2006, 2007, 2008, 2010 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2006, 2007, 2008, 2010, 2011 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -28,3 +28,6 @@ design. The coyotos microkernel specification can be found [here](http://www.coyotos.org/docs/ukernel/spec.html). + +There once was the idea of a GNU/Hurd [[port using the Coyotos +microkernel|history/port_to_another_microkernel]], but this didn't come live. diff --git a/microkernel/l4.mdwn b/microkernel/l4.mdwn index 45929842..7af5e6fc 100644 --- a/microkernel/l4.mdwn +++ b/microkernel/l4.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2004, 2006, 2007, 2008, 2010 Free Software +[[!meta copyright="Copyright © 2004, 2006, 2007, 2008, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -20,7 +20,8 @@ on formally verifying an L4 microkernel. * {{$sel4}} -There was a GNU/Hurd [[history/port_to_L4]], which is now stalled. +There was a GNU/Hurd [[port to L4|history/port_to_another_microkernel]], which +is now stalled. [[!ymlfront data=""" -- cgit v1.2.3 From 817df620bedae9c1daa0497f64a901d51e5bd2dd Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 00:52:08 +0100 Subject: Some more IRC discussions. --- open_issues/anatomy_of_a_hurd_system.mdwn | 73 +++++++++ open_issues/ext2fs_page_cache_swapping_leak.mdwn | 23 +++ open_issues/pfinet_vs_system_time_changes.mdwn | 42 ++++++ ...dez-vous_leading_to_duplicate_port_destroy.mdwn | 163 +++++++++++++++++++++ open_issues/sudo_date_crash.mdwn | 16 -- open_issues/unit_testing.mdwn | 20 +++ 6 files changed, 321 insertions(+), 16 deletions(-) create mode 100644 open_issues/anatomy_of_a_hurd_system.mdwn create mode 100644 open_issues/ext2fs_page_cache_swapping_leak.mdwn create mode 100644 open_issues/pfinet_vs_system_time_changes.mdwn create mode 100644 open_issues/rpc_to_self_with_rendez-vous_leading_to_duplicate_port_destroy.mdwn delete mode 100644 open_issues/sudo_date_crash.mdwn diff --git a/open_issues/anatomy_of_a_hurd_system.mdwn b/open_issues/anatomy_of_a_hurd_system.mdwn new file mode 100644 index 00000000..e1d5c9d8 --- /dev/null +++ b/open_issues/anatomy_of_a_hurd_system.mdwn @@ -0,0 +1,73 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!taglink open_issue_documentation]] + +A bunch of this should also be covered in other (introductionary) material, +like Bushnell's Hurd paper. All this should be unfied and streamlined. + +IRC, freenode, #hurd, 2011-03-08 + + I've a question on what are the "units" in the hurd project, if + you were to divide them into units if they aren't, and what are the + dependency relations between those units(roughly, nothing too pedantic + for now) + there is GNU Mach (the microkernel); there are the server + libraries in the Hurd package; there are the actual servers in the same; + and there is the POSIX implementation layer in glibc + relations are a bit tricky + Mach is the base layer which implements IPC and memory management + hmm I'll probably allocate time for dependency graph generation, + in the worst case + on top of this, the Hurd servers, using the server libraries, + implement various aspects of the system functionality + client programs use libc calls to use the servers + (servers also use libc to communicate with other servers and/or + Mach though) + so every server depends solely on mach, and no other server? + s/mach/mach and/or libc/ + I think these things should be pretty clear one you are somewhat + familiar with the Hurd architecture... nothing really tricky there + no + servers often depend on other servers for certain functionality + +--- + +IRC, freenode, #hurd, 2011-03-12 + + when mach first starts up, does it have some basic i/o or fs + functionality built into it to start up the initial hurd translators? + I/O is presently completely in Mach + filesystems are in userspace + the root filesystem and exec server are loaded by grub + o I see + so in order to start hurd, you would have to start mach and + simultaneously start the root filesystem and exec server? + not exactly + GRUB loads all three, and then starts Mach. Mach in turn starts + the servers according to the multiboot information passed from GRUB + ok, so does GRUB load them into ram? + I'm trying to figure out in my mind how hurd is initially started + up from a low-level pov + yes, as I said, GRUB loads them + ok, thanks antrik...I'm new to the idea of microkernels, but a + veteran of monolithic kernels + although I just learned that windows nt is a hybrid kernel which I + never knew! + note there's a /hurd/ext2fs.static + I belive that's what is used initially... right? + yes + loading the shared libraries in addition to the actual server + would be unweildy + so the root FS server is linked statically instead + what does the root FS server do? + well, it serves the root FS ;-) + it also does some bootstrapping work during startup, to bring the + rest of the system up diff --git a/open_issues/ext2fs_page_cache_swapping_leak.mdwn b/open_issues/ext2fs_page_cache_swapping_leak.mdwn new file mode 100644 index 00000000..0ace5cd3 --- /dev/null +++ b/open_issues/ext2fs_page_cache_swapping_leak.mdwn @@ -0,0 +1,23 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +IRC, OFTC, #debian-hurd, 2011-03-24 + + I still believe we have an ext2fs page cache swapping leak, however + as the 1.8GiB swap was full, yet the ld process was only 1.5GiB big + a leak at swapping time, you mean? + I mean the ext2fs page cache being swapped out instead of simply + dropped + ah + so the swap tends to accumulate unuseful stuff, i see + yes + the disk content, basicallyt :) diff --git a/open_issues/pfinet_vs_system_time_changes.mdwn b/open_issues/pfinet_vs_system_time_changes.mdwn new file mode 100644 index 00000000..a9e1e242 --- /dev/null +++ b/open_issues/pfinet_vs_system_time_changes.mdwn @@ -0,0 +1,42 @@ +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +IRC, unknown channel, unknown date. + + I did a sudo date... + and the machine hangs + +This was very likely as misdiagnosis: + +IRC, freenode, #hurd, 2011-03-25 + + antrik: I suspect it'S some timing stuff in pfinet that perhaps + uses absolute time, and somehow wildely gets confused? + tschwinge: BTW, pfinet doesn't actually die I think -- it just + drops open connections... + perhaps it thinks they timed out + antrik: Isn't the translator restarted instead? + don't think so + when pfinet actually dies, I also loose the NFS mounts, which + doesn't happen in this case + hehe "... and the machine hangs" + he didn't bother to check that the machine is perfectly fine, only + the SSH connection got dropped + Ah, I see. So it'S perhaps indeed simply closes TCP + connections that have been without data for ``too long''? + yeah, that's my guess + my clock is speeding, so ntpdate sets it in the past + perhaps there is some math that concludes the connection have been + inactive for -200 seconds, which (unsigned) is more than any timeout :-) + (The other way round, you might likely get some integer + wrap-around, and thus the same result.) + Yes. diff --git a/open_issues/rpc_to_self_with_rendez-vous_leading_to_duplicate_port_destroy.mdwn b/open_issues/rpc_to_self_with_rendez-vous_leading_to_duplicate_port_destroy.mdwn new file mode 100644 index 00000000..9db92250 --- /dev/null +++ b/open_issues/rpc_to_self_with_rendez-vous_leading_to_duplicate_port_destroy.mdwn @@ -0,0 +1,163 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +[RPC to self with rendez-vous leading to duplicate port +destroy](http://lists.gnu.org/archive/html/bug-hurd/2011-03/msg00045.html) + +IRC, freenode, #hurd, 2011-03-14 + + youpi: I wonder, why does the root FS call diskfs_S_dir_lookup() + at all?... + errr, because a client asked for it? + (problem with RPCs is you can't easily know where they come from :) + ) + (especially when it's the root fs...) + ah, it's about a client request... didn't see that + well, I just said "is called", yes + I do not really understand though why it tries to reauthenticate + against itself... + I fear my memory of the lookup mechanism grew a bit dim + see the source + it's about a translated entry + (and I never fully understood some aspects anyways...) + it needs to start the translated entry as another user, possibly + yes, but a translated entry normally would be served by *another* + process?... + sure, but ext2fs has to prepare it + thus reauthenticate to prepare the correct set of rights + prepare what? + rights + so the process is not root, doesn't have / opened as root, etc. + rights for what? + err, about everything + IIRC the reauthentication is done by the parent FS on the port to + the *translated* node + and the translated node should be a different process?... + that's not what I read in the source + fshelp_fetch_root + ports[INIT_PORT_CRDIR] = reauth (getcrdir ()); + here, getcrdir() returns ext2fs itself + well, perhaps the issue is that I have no idea what + fshelp_fetch_root() does, nor why it is called here... + it notably starts the translator that dir_lookup is looking at, if + needed + possibly as a different user, thus reauthentication of CRDIR + so this is about a port that is passed to the translator being + started? + no + well, depends on what you mean by "port" + it's about reauthenticating a port to be passed to the translator + being started + and for that a rendez-vous port is needed for the reauthentication + and that's the one at stake + yeah, I meant the port that is reauthenticated + what is CRDIR? + current root dir ... + so the parent translator passes it's own root dir to the child + translator; and the issue is that for the root FS the root dir points to + the root FS itself... + yes + OK, that makes sense + (but that's only one example, rgrep mach_port_destroy hurd/ show + other potential issues) + well, that's actually what I wanted to mention next... why is the + rendez-vous port destroyed, instead of just deallocating the port right + and letting reference counting to it's thing?... + do its thing + "just to make sure" I guess + it's pretty obvious that this will cause trouble for any RPC + referencing itself... + well, follow-up with that on the list + with roland/tb in CC + only they would know any real reason for destroy + btw, if you knew how we could make _hurd_select()'s raw __mach_msg + call be interruptible by signals, that'll permit to fix sudo + (damn, I need sleep, my tenses are all wrong) + BTW, does this cause any actual trouble?... + I don't know much about interruption... cfhammer might have a + better idea, he look into that stuff quite a bit AIUI + looked + (hehe, it's not only your tenses... guess there's something in the + ether ;-) ) + it makes sudo, mailq, etc. fail sometimes + I mean the rendez-vous thing + that's it, yes + sudo etc. fail at least due to this + so these are two different problems that both affect sudo? + (rendez-vous and interruption I mean) + yes + with my patch the buildds have much fewer issues, but still some + (my interrupt-related patch) + I'm installing a s/destroy/deallocate/ version of ext2fs on the + buildds, we'll see how it behaves + (it fixes my testcase at least) + interrupt-related patch? + only thing interrupt-related I remember was the reauthentication + race... + that's what I mean + well, cfhammer investigated this is quite some depth, explaining + quite well why the race is only mitigated but still exists... problem is + that we didn't know how to fix it properly + because nobody seems to understand the cancellation code, except + perhaps for Roland and Thomas + (and I'm not even entirely sure about them :-) ) + I think his findings and our conclusions are documented on the + ML... + by "much fewer issues", I mean that some of the symptoms have + disappeared, others haven't + BTW, couldn't the rendez-vous thing be worked around by simply + ignoring the errors from the failing deallocate?... + no, failing deallocate are actually dangerous + why? + since the name might have been reused for something else in the + meanwhile + that's the whole point of the warning I had added in the kernel + itself + I see + such things really deserve tracking, since they can have any kind + of consequence + does Mach try to reuse names quickly, rather than only after + wrapping around?... + it seems to + OK, then this is a serious problem indeed + (note: I rarely divine issues when there aren't actual frequent + symptoms :) ) + well, the problem with the warning is that it only shows in the + cases that do *not* cause a problem... so it's hard to associate them + with any specific issues + well, most of the time the port is not reused quickly enough + so in most case it shows up more often than causing problem + +IRC, freenode, #hurd, 2011-03-14 + + ok, mach_port_deallocate actually can't be used + since mach_reply_port() returns a receive right, not a send right + * youpi guesses he will really have to manage to understand all that port + stuff completely + oh, right + youpi: hm... now I'm confused though. if one client holds a + receive right, the other client (or in this case the same process) should + have a send or send-once right -- these should *not* share the same name + in my understanding + destroying the receive right should turn the send right into a + dead name + so unless I'm missing something, the destroy shouldn't be a + problem, and there must be something else going wrong + hm... actually I'm probably wrong + yeah, definitely wrong. receive rights and "ordinary" send rights + share the name. only send-once rights are special + I wonder whether the problem could be worked around by using a + send-once right... + mach_port_mod_refs(mach_task_self(), name, + MACH_PORT_RIGHT_RECEIVE, -1) can be used to deallocate only the receive + right + oh, you already figured that out :-) diff --git a/open_issues/sudo_date_crash.mdwn b/open_issues/sudo_date_crash.mdwn deleted file mode 100644 index 53303abc..00000000 --- a/open_issues/sudo_date_crash.mdwn +++ /dev/null @@ -1,16 +0,0 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -[[!tag open_issue_gnumach]] - -IRC, unknown channel, unknown date. - - I did a sudo date... - and the machine hangs diff --git a/open_issues/unit_testing.mdwn b/open_issues/unit_testing.mdwn index a5ffe19d..feda3be4 100644 --- a/open_issues/unit_testing.mdwn +++ b/open_issues/unit_testing.mdwn @@ -320,3 +320,23 @@ freenode, #hurd channel, 2011-03-07: this, and just generally though that some sort of automated testing is needed, and thus started collecting ideas. antrik: You're of course invited to fix that. + +IRC, freenode, #hurd, 2011-03-08 + +(After discussing the [[anatomy_of_a_hurd_system]].) + + so that's what your question is actually about? + so what I would imagine is a set of only-this-server tests for + each server, and then we can have fun adding composite tests + thus making debugging the composite scenarios a bit less tricky + indeed + and if you were trying to pass a composite test, it would also + help knowing that you still didn't break the server-only test + there are so many different things that can be tested... the + summer will only suffice to dip into this really :-) + yeah, I'm designing my proposal to focus on 1) make/use a + testing framework that fits the Hurd case very well 2) write some tests + and docs on how to write good tests + well, doesn't have to be *one* framework... unit testing and + regression testing are quite different things, which can be covered by + different frameworks -- cgit v1.2.3 From c82754dbb4bf6efd9ba50cbf5e5d0586ffaf816c Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 00:52:52 +0100 Subject: open_issues/gdb_noninvasive_mode_new_threads: New. --- open_issues/gdb_noninvasive_mode_new_threads.mdwn | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 open_issues/gdb_noninvasive_mode_new_threads.mdwn diff --git a/open_issues/gdb_noninvasive_mode_new_threads.mdwn b/open_issues/gdb_noninvasive_mode_new_threads.mdwn new file mode 100644 index 00000000..9b3992f4 --- /dev/null +++ b/open_issues/gdb_noninvasive_mode_new_threads.mdwn @@ -0,0 +1,15 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_gdb]] + +Debugging a translator. `gdb binary`. `set noninvasive on`. `attach [PID]`. +Translator does some work. GDB doesn't notice new threads. `detach`. `attach +[PID]` -- now new threads are visible. -- cgit v1.2.3 From 64f60ab301c296aa6abe6a2dbc4ad27df3763034 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 00:53:40 +0100 Subject: open_issues/gdb_thread_ids: Another issue. --- open_issues/gdb_thread_ids.mdwn | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/open_issues/gdb_thread_ids.mdwn b/open_issues/gdb_thread_ids.mdwn index c31a9967..c04a10ee 100644 --- a/open_issues/gdb_thread_ids.mdwn +++ b/open_issues/gdb_thread_ids.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -6,8 +6,8 @@ id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!meta title="GDB: thread ids"]] @@ -23,3 +23,9 @@ GNU GDB's Pedro Alves: Also see [[thread numbering of ps and GDB]]. + +--- + +`attach` to a multi-threaded process. See threads 1 to 5. `detach`. `attach` +again -- thread numbers continue where they stopped last time: now they're +threads 6 to 10. -- cgit v1.2.3 From 2ad64644b9290ed1b63108b01ef63939adba5a93 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 01:29:05 +0100 Subject: Re-integrate GSoC pages with the non-GSoC world. Remove duplicates, apart from procfs, which should rather be removed from the GSoC items. --- .../gsoc/project_ideas/disk_io_performance.mdwn | 10 +-- community/gsoc/project_ideas/dtrace.mdwn | 8 ++- .../gsoc/project_ideas/libdiskfs_locking.mdwn | 10 +-- community/gsoc/project_ideas/procfs.mdwn | 8 ++- community/gsoc/project_ideas/valgrind.mdwn | 2 + lttng.mdwn | 2 +- open_issues/code_analysis.mdwn | 4 +- open_issues/debugging.mdwn | 4 +- open_issues/dtrace.mdwn | 47 ------------ open_issues/locking.mdwn | 40 ----------- open_issues/performance.mdwn | 4 +- open_issues/performance/io_system.mdwn | 50 ------------- .../performance/io_system/binutils_ld_64ksec.mdwn | 5 +- .../io_system/clustered_page_faults.mdwn | 2 + open_issues/performance/io_system/read-ahead.mdwn | 2 + open_issues/profiling.mdwn | 2 +- open_issues/valgrind.mdwn | 83 ---------------------- systemtap.mdwn | 2 +- topgit.mdwn | 5 +- 19 files changed, 45 insertions(+), 245 deletions(-) delete mode 100644 open_issues/dtrace.mdwn delete mode 100644 open_issues/locking.mdwn delete mode 100644 open_issues/performance/io_system.mdwn delete mode 100644 open_issues/valgrind.mdwn diff --git a/community/gsoc/project_ideas/disk_io_performance.mdwn b/community/gsoc/project_ideas/disk_io_performance.mdwn index b6f223c8..ae634709 100644 --- a/community/gsoc/project_ideas/disk_io_performance.mdwn +++ b/community/gsoc/project_ideas/disk_io_performance.mdwn @@ -11,6 +11,8 @@ License|/fdl]]."]]"""]] [[!meta title="Disk I/O Performance Tuning"]] +[[!tag open_issue_hurd]] + The most obvious reason for the Hurd feeling slow compared to mainstream systems like GNU/Linux, is a low I/O system performance, in particular very slow hard disk access. @@ -18,8 +20,8 @@ slow hard disk access. The reason for this slowness is lack and/or bad implementation of common optimization techniques, like scheduling reads and writes to minimize head movement; effective block caching; effective reads/writes to partial blocks; -[[reading/writing multiple blocks at once|clustered_page_faults]]; and -[[read-ahead]]. The +[[reading/writing multiple blocks at once|open_issues/performance/io_system/clustered_page_faults]]; and +[[open_issues/performance/io_system/read-ahead]]. The [[ext2_filesystem_server|hurd/translator/ext2fs]] might also need some optimizations at a higher logical level. @@ -29,12 +31,12 @@ requires understanding the data flow through the various layers involved in disk access on the Hurd ([[filesystem|hurd/virtual_file_system]], [[pager|hurd/libpager]], driver), and general experience with optimizing complex systems. That said, the killing feature we are definitely -missing is the [[read-ahead]], and even a very simple implementation would bring +missing is the [[open_issues/performance/io_system/read-ahead]], and even a very simple implementation would bring very big performance speedups. Here are some real testcases: - * [[binutils_ld_64ksec]]; + * [[open_issues/performance/io_system/binutils_ld_64ksec]]; * running the Git testsuite which is mostly I/O bound; diff --git a/community/gsoc/project_ideas/dtrace.mdwn b/community/gsoc/project_ideas/dtrace.mdwn index 252c303a..8fd3231f 100644 --- a/community/gsoc/project_ideas/dtrace.mdwn +++ b/community/gsoc/project_ideas/dtrace.mdwn @@ -11,13 +11,15 @@ License|/fdl]]."]]"""]] [[!meta title="dtrace Support"]] +[[!tag open_issue_gnumach]] + One of the main problems of the current Hurd implementation is very poor -[[performance]]. While we have a bunch of ideas what could cause the performance +[[open_issues/performance]]. While we have a bunch of ideas what could cause the performance problems, these are mostly just guesses. Better understanding what really causes bad performance is necessary to improve the situation. For that, we need tools for performance measurements. While all kinds of more -or less specific [[profiling]] tools could be conceived, the most promising and +or less specific [[open_issues/profiling]] tools could be conceived, the most promising and generic approach seems to be a framework for logging certain events in the running system (both in the microkernel and in the Hurd servers). This would allow checking how much time is spent in certain modules, how often certain @@ -35,7 +37,7 @@ with integrating existing components as well as low-level programming. Possible mentors: Samuel Thibault (youpi) -Related: [[profiling]], [[LTTng]], [[SystemTap]] +Related: [[open_issues/profiling]], [[LTTng]], [[SystemTap]] Exercise: In lack of a good exercise directly related to this task, just pick one of the kernel-related or generally low-level tasks from the bug/task diff --git a/community/gsoc/project_ideas/libdiskfs_locking.mdwn b/community/gsoc/project_ideas/libdiskfs_locking.mdwn index 0e38b6a0..f26efb7f 100644 --- a/community/gsoc/project_ideas/libdiskfs_locking.mdwn +++ b/community/gsoc/project_ideas/libdiskfs_locking.mdwn @@ -11,6 +11,8 @@ License|/fdl]]."]]"""]] [[!meta title="Fix libdiskfs Locking Issues"]] +[[!tag open_issue_hurd]] + Every now and then, new locking issues are discovered in [[hurd/libdiskfs]] or [[hurd/translator/ext2fs]], for example. Nowadays these in fact seem to be the most often encountered cause of Hurd crashes @@ -24,19 +26,19 @@ faulty paths causing these lockups. The task is systematically checking the [[hurd/libdiskfs]] code for this kind of locking issues. To achieve this, some kind of test harness has to be implemented: For example instrumenting the code to check locking correctness constantly at -runtime. Or implementing a [[unit testing]] framework that explicitly checks +runtime. Or implementing a [[open_issues/unit_testing]] framework that explicitly checks locking in various code paths. (The latter could serve as a template for implementing unit tests in other parts of the Hurd codebase...) -(A [[systematic code review|security]] would probably suffice to find the +(A [[systematic code review|open_issues/security]] would probably suffice to find the existing locking issues; but it wouldn't document the work in terms of actual code produced, and thus it's not suitable for a GSoC project...) This task requires experience with debugging locking issues in -[[multithreaded|multithreading]] applications. +[[multithreaded|open_issues/multithreading]] applications. -Tools have been written for automated [[code analysis]]; these can help to +Tools have been written for automated [[open_issues/code_analysis]]; these can help to locate and fix such errors. Possible mentors: Samuel Thibault (youpi) diff --git a/community/gsoc/project_ideas/procfs.mdwn b/community/gsoc/project_ideas/procfs.mdwn index d4760aae..b4d1dc5f 100644 --- a/community/gsoc/project_ideas/procfs.mdwn +++ b/community/gsoc/project_ideas/procfs.mdwn @@ -1,4 +1,5 @@ -[[!meta copyright="Copyright © 2008, 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2008, 2009, 2011 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -43,3 +44,8 @@ Exercise: Add or fix one piece in the existing procfs translator. *Status*: Madhusudan.C.S has implemented a new, fully functional [[procfs|madhusudancs]] for GSoC 2008. He is still working on some outstanding issues. + +--- + +Note that [[jkoenig's `procfs` re-write|hurd/translator/procfs/jkoenig]] should +address all these issues already. diff --git a/community/gsoc/project_ideas/valgrind.mdwn b/community/gsoc/project_ideas/valgrind.mdwn index 79c8bd1d..16cd1373 100644 --- a/community/gsoc/project_ideas/valgrind.mdwn +++ b/community/gsoc/project_ideas/valgrind.mdwn @@ -11,6 +11,8 @@ License|/fdl]]."]]"""]] [[!meta title="Porting Valgrind to the Hurd"]] +[[!tag open_issue_gnumach open_issues_hurd]] + [Valgrind](http://valgrind.org/) is an extremely useful debugging tool for memory errors. (And some other kinds of hard-to-find errors too.) Aside from being useful for program development in general, diff --git a/lttng.mdwn b/lttng.mdwn index 5776baa8..840312c4 100644 --- a/lttng.mdwn +++ b/lttng.mdwn @@ -19,7 +19,7 @@ License|/fdl]]."]]"""]] # Related - * [[open_issues/dtrace]] + * [[community/gsoc/project_ideas/dtrace]] * [[SystemTap]] diff --git a/open_issues/code_analysis.mdwn b/open_issues/code_analysis.mdwn index ad59f962..21e09089 100644 --- a/open_issues/code_analysis.mdwn +++ b/open_issues/code_analysis.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -40,7 +40,7 @@ analysis|performance]], [[formal_verification]], as well as general * - * [[Valgrind]] + * [[community/gsoc/project_ideas/Valgrind]] * [Smatch](http://smatch.sourceforge.net/) diff --git a/open_issues/debugging.mdwn b/open_issues/debugging.mdwn index e66a086f..e087f484 100644 --- a/open_issues/debugging.mdwn +++ b/open_issues/debugging.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -42,7 +42,7 @@ We have debugging infrastructure. For example: Continues: , which introduces . - * [[locking]] + * [[community/gsoc/project_ideas/libdiskfs_locking]] * , or -- just two examples; there's a lot of such stuff for Linux. diff --git a/open_issues/dtrace.mdwn b/open_issues/dtrace.mdwn deleted file mode 100644 index cbac28fb..00000000 --- a/open_issues/dtrace.mdwn +++ /dev/null @@ -1,47 +0,0 @@ -[[!meta copyright="Copyright © 2008, 2009, 2011 Free Software Foundation, -Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -One of the main problems of the current Hurd implementation is very poor -[[performance]]. While we have a bunch of ideas what could cause the performance -problems, these are mostly just guesses. Better understanding what really -causes bad performance is necessary to improve the situation. - -For that, we need tools for performance measurements. While all kinds of more -or less specific [[profiling]] tools could be conceived, the most promising and -generic approach seems to be a framework for logging certain events in the -running system (both in the microkernel and in the Hurd servers). This would -allow checking how much time is spent in certain modules, how often certain -situations occur, how things interact, etc. It could also prove helpful in -debugging some issues that are otherwise hard to find because of complex -interactions. - -The most popular framework for that is Sun's dtrace; but there might be others. -The student has to evaluate the existing options, deciding which makes most -sense for the Hurd; and implement that one. (Apple's implementation of dtrace -in their Mach-based kernel might be helpful here...) - -This project requires ability to evaluate possible solutions, and experience -with integrating existing components as well as low-level programming. - -Possible mentors: Samuel Thibault (youpi) - -Related: [[profiling]], [[LTTng]], [[SystemTap]] - -Exercise: In lack of a good exercise directly related to this task, just pick -one of the kernel-related or generally low-level tasks from the bug/task -trackers on savannah, and make a go at it. You might not be able to finish the -task in a limited amount of time, but you should at least be able to make a -detailed analysis of the issue. - -*Status*: Andei Barbu was working on -[SystemTap](http://csclub.uwaterloo.ca/~abarbu/hurd/) for GSoC 2008, but it -turned out too Linux-specific. He implemented kernel probes, but there is no -nice frontend yet. diff --git a/open_issues/locking.mdwn b/open_issues/locking.mdwn deleted file mode 100644 index 6e22f887..00000000 --- a/open_issues/locking.mdwn +++ /dev/null @@ -1,40 +0,0 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, -Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -[[!tag open_issue_hurd gsoc-task]] - -Every now and then, new locking issues are discovered in -[[hurd/libdiskfs]] or [[hurd/translator/ext2fs]], for example. Nowadays -these in fact seem to be the most often encountered cause of Hurd crashes -/ lockups. - -One of these could be traced -recently, and turned out to be a lock inside [[hurd/libdiskfs]] that was taken -and not released in some cases. There is reason to believe that there are more -faulty paths causing these lockups. - -The task is systematically checking the [[hurd/libdiskfs]] code for this kind of locking -issues. To achieve this, some kind of test harness has to be implemented: For -example instrumenting the code to check locking correctness constantly at -runtime. Or implementing a [[unit testing]] framework that explicitly checks -locking in various code paths. (The latter could serve as a template for -implementing unit tests in other parts of the Hurd codebase...) - -(A [[systematic code review|security]] would probably suffice to find the -existing locking -issues; but it wouldn't document the work in terms of actual code produced, and -thus it's not suitable for a GSoC project...) - -This task requires experience with debugging locking issues in -[[multithreaded|multithreading]] applications. - -Tools have been written for automated [[code analysis]]; these can help to -locate and fix such errors. diff --git a/open_issues/performance.mdwn b/open_issues/performance.mdwn index 9b3701b3..eb9f3f8a 100644 --- a/open_issues/performance.mdwn +++ b/open_issues/performance.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -18,7 +18,7 @@ In [[microkernel]]-based systems, there is generally a considerable [[RPC]] overhead. In a multi-server system, it is non-trivial to implement a high-performance -[[I/O System|io_system]]. +[[I/O System|community/gsoc/project_ideas/disk_io_performance]]. When providing [[faq/POSIX_compatibility]] (and similar interfaces) in an environemnt that doesn't natively implement these interfaces, there may be a diff --git a/open_issues/performance/io_system.mdwn b/open_issues/performance/io_system.mdwn deleted file mode 100644 index 8535eae3..00000000 --- a/open_issues/performance/io_system.mdwn +++ /dev/null @@ -1,50 +0,0 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, -Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -[[!meta title="I/O System"]] - -[[!tag open_issue_hurd gsoc-task]] - -The most obvious reason for the Hurd feeling slow compared to mainstream -systems like GNU/Linux, is a low I/O system performance, in particular very -slow hard disk access. - -The reason for this slowness is lack and/or bad implementation of common -optimization techniques, like scheduling reads and writes to minimize head -movement; effective block caching; effective reads/writes to partial blocks; -[[reading/writing multiple blocks at once|clustered_page_faults]]; and -[[read-ahead]]. The -[[ext2_filesystem_server|hurd/translator/ext2fs]] might also need some -optimizations at a higher logical level. - -The goal of this project is to analyze the current situation, and implement/fix -various optimizations, to achieve significantly better disk performance. It -requires understanding the data flow through the various layers involved in -disk access on the Hurd ([[filesystem|hurd/virtual_file_system]], -[[pager|hurd/libpager]], driver), and general experience with -optimizing complex systems. That said, the killing feature we are definitely -missing is the [[read-ahead]], and even a very simple implementation would bring -very big performance speedups. - -Here are some real testcases: - - * [[binutils_ld_64ksec]]; - - * running the Git testsuite which is mostly I/O bound; - - * use [[TopGit]] on a non-toy repository. - - -Possible mentors: Samuel Thibault (youpi) - -Exercise: Look through all the code involved in disk I/O, and try something -easy to improve. It's quite likely though that you will find nothing obvious -- -in this case, please contact us about a different exercise task. diff --git a/open_issues/performance/io_system/binutils_ld_64ksec.mdwn b/open_issues/performance/io_system/binutils_ld_64ksec.mdwn index b59a87a7..79c2300f 100644 --- a/open_issues/performance/io_system/binutils_ld_64ksec.mdwn +++ b/open_issues/performance/io_system/binutils_ld_64ksec.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -10,7 +10,8 @@ License|/fdl]]."]]"""]] [[!tag open_issue_hurd]] -This one may be considered as a testcase for I/O system optimization. +This one may be considered as a testcase for [[I/O system +optimization|community/gsoc/project_ideas/disk_io_performance]]. It is taken from the [[binutils testsuite|binutils]], `ld/ld-elf/sec64k.exp`, where this diff --git a/open_issues/performance/io_system/clustered_page_faults.mdwn b/open_issues/performance/io_system/clustered_page_faults.mdwn index 3a187523..37433e06 100644 --- a/open_issues/performance/io_system/clustered_page_faults.mdwn +++ b/open_issues/performance/io_system/clustered_page_faults.mdwn @@ -10,6 +10,8 @@ License|/fdl]]."]]"""]] [[!tag open_issue_gnumach open_issue_hurd]] +[[community/gsoc/project_ideas/disk_io_performance]]. + IRC, freenode, #hurd, 2011-02-16 exceptfor the kernel, everything in an address space is diff --git a/open_issues/performance/io_system/read-ahead.mdwn b/open_issues/performance/io_system/read-ahead.mdwn index 3ee30b5d..b6851edd 100644 --- a/open_issues/performance/io_system/read-ahead.mdwn +++ b/open_issues/performance/io_system/read-ahead.mdwn @@ -10,6 +10,8 @@ License|/fdl]]."]]"""]] [[!tag open_issue_gnumach open_issue_hurd]] +[[community/gsoc/project_ideas/disk_io_performance]] + IRC, #hurd, freenode, 2011-02-13: youpi: Would libdiskfs/diskfs.h be in the right place to make diff --git a/open_issues/profiling.mdwn b/open_issues/profiling.mdwn index e04fb08a..7e3c7350 100644 --- a/open_issues/profiling.mdwn +++ b/open_issues/profiling.mdwn @@ -17,7 +17,7 @@ done for [[performance analysis|performance]] reasons. Should be working, but some issues have been reported, regarding GCC spec files. Should be possible to fix (if not yet done) easily. - * [[dtrace]] + * [[community/gsoc/project_ideas/dtrace]] Have a look at this, integrate it into the main trees. diff --git a/open_issues/valgrind.mdwn b/open_issues/valgrind.mdwn deleted file mode 100644 index bd45829c..00000000 --- a/open_issues/valgrind.mdwn +++ /dev/null @@ -1,83 +0,0 @@ -[[!meta copyright="Copyright © 2009, 2010, 2011 Free Software Foundation, -Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -[[!meta title="Porting Valgrind to the Hurd"]] - -[[!tag gsoc-task]] - -[Valgrind](http://valgrind.org/) is an extremely useful debugging tool for memory errors. -(And some other kinds of hard-to-find errors too.) -Aside from being useful for program development in general, -a Hurd port will help finding out why certain programs segfault on the Hurd, -although they work on Linux. -Even more importantly, it will help finding bugs in the Hurd servers themselfs. - -To keep track of memory use, -Valgrind however needs to know how each [[system call]] affects the validity of memory regions. -This knowledge is highly kernel-specific, -and thus Valgrind needs to be explicitely ported for every system. - -Such a port involves two major steps: -making Valgrind understand how kernel traps work in general on the system in question; -and how all the individual kernel calls affect memory. -The latter step is where most of the work is, -as the behaviour of each single [[system call]] needs to be described. - -Compared to Linux, -[[microkernel/Mach]] (the microkernel used by the Hurd) has very few kernel traps. -Almost all [[system call]]s are implemented as [[RPC]]s instead -- -either handled by Mach itself, or by the various [[Hurd servers|hurd/translator]]. -All RPCs use a pair of `mach_msg` invocations: -one to send a request message, and one to receive a reply. -However, while all RPCs use the same `mach_msg` trap, -the actual effect of the call varies greatly depending on which RPC is invoked -- -similar to the `ioctl` call on Linux. -Each request thus must be handled individually. - -Unlike `ioctl`, -the RPC invocations have explicit type information for the parameters though, -which can be retrieved from the message header. -By analyzing the parameters of the RPC reply message, -Valgrind can know exactly which memory regions are affected by that call, -even without specific knowledge of the RPC in question. -Thus implementing a general parser for the reply messages -will already give Valgrind a fairly good approximation of memory validity -- -without having to specify the exact semantic of each RPC by hand. - -While this should make Valgrind quite usable on the Hurd already, it's not perfect: -some RPCs might return a buffer that is only partially filled with valid data; -or some reply parameters might be optional, -and only contain valid data under certain conditions. -Such specific semantics can't be deduced from the message headers alone. -Thus for a complete port, -it will still be necessary to go through the list of all known RPCs, -and implement special handling in Valgrind for those RPCs that need it. - -The goal of this task is at minimum to make Valgrind grok Mach traps, -and to implement the generic RPC handler. -Ideally, specific handling for RPCs needing it should also be implemented. - -Completing this project will require digging into Valgrind's handling of [[system call]]s, -and into Hurd RPCs. -It is not an easy task, but a fairly predictable one -- -there shouldn't be any unexpected difficulties, -and no major design work is necessary. -It doesn't require any specific previous knowledge: -only good programming skills in general. -On the other hand, -the student will obtain a good understanding of Hurd RPCs while working on this task, -and thus perfect qualifications for Hurd development in general :-) - -Possible mentors: Samuel Thibault (youpi) - -Exercise: As a starter, -students can try to teach valgrind a couple of Linux ioctls, -as this will make them learn how to use the read/write primitives of valgrind. diff --git a/systemtap.mdwn b/systemtap.mdwn index abd1961e..ba64c0d4 100644 --- a/systemtap.mdwn +++ b/systemtap.mdwn @@ -19,7 +19,7 @@ License|/fdl]]."]]"""]] # Related - * [[open_issues/dtrace]] + * [[community/gsoc/project_ideas/dtrace]] * [[LTTng]] diff --git a/topgit.mdwn b/topgit.mdwn index b71038ec..fb374337 100644 --- a/topgit.mdwn +++ b/topgit.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -30,7 +30,8 @@ development branches, for example [[source_repositories/binutils]] or # Running it on GNU/Hurd Nothing special to that, technically, *only* that our [[I/O system's (non-) -performance|open_issues/performance/io_system]] will render this unbearably +performance|community/gsoc/project_ideas/disk_io_performance]] will render this +unbearably slow for anything but simple test cases. So don't try to run it on the [[GCC]] or [[glibc]] repositories. Talk to [[tschwinge]] about how he's using it on a GNU/Linux machine and push the resulting trees to GNU/Hurd systems. -- cgit v1.2.3 From a2a4176bcd74dc6c607d48131f26cb5fa4affb3d Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 01:42:25 +0100 Subject: open_issues/unit_testing: Move discussion to community/gsoc/project_ideas/testing_framework/discussion. --- .../gsoc/project_ideas/testing_framework.mdwn | 2 +- .../testing_framework/discussion.mdwn | 270 +++++++++++++++++++++ open_issues/unit_testing.mdwn | 266 +------------------- 3 files changed, 276 insertions(+), 262 deletions(-) create mode 100644 community/gsoc/project_ideas/testing_framework/discussion.mdwn diff --git a/community/gsoc/project_ideas/testing_framework.mdwn b/community/gsoc/project_ideas/testing_framework.mdwn index 0448bc6b..ff9899d9 100644 --- a/community/gsoc/project_ideas/testing_framework.mdwn +++ b/community/gsoc/project_ideas/testing_framework.mdwn @@ -36,7 +36,7 @@ before the end of the summer. (As a bonus, in addition to these explicit tests, it would be helpful to integrate some methods -for testing [[locking validity|open_issues/locking]], +for testing [[locking validity|libdiskfs_locking]], performing static code analysis etc.) This task probably requires some previous experience diff --git a/community/gsoc/project_ideas/testing_framework/discussion.mdwn b/community/gsoc/project_ideas/testing_framework/discussion.mdwn new file mode 100644 index 00000000..872d0eb7 --- /dev/null +++ b/community/gsoc/project_ideas/testing_framework/discussion.mdwn @@ -0,0 +1,270 @@ +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +freenode, #hurd channel, 2011-03-05: + + what about testing though? + like sort of "what's missing? lets write tests for it so that + when someone gets to implementing it, he knows what to do. Repeat" + project + you mean creating an automated testing framework? + this is actually a task I want to add for this year, if I get + around to it :-) + yeah I'd very much want to apply for that one + cuz I've never done Kernel hacking before, but I know that with + the right tasks like "test VM functionality", I would be able to write up + the automated tests and hopefully learn more about what breaks/makes the + kernel + (and it would make implementing the feature much less hand-wavy + and more correct) + antrik, I believe the framework(CUnit right?) exists, but we just + need to write the tests. + do you have prior experience implementing automated tests? + lots of tests! + yes, in Java mostly, but I've played around with CUnit + ah, great :-) + here's what I know from experience: 1) write basic tests. 2) + write ones that test multiple features 3) stress test [option 4) + benchmark and record to DB] + well, what we'd rather need is to fix the issues we already know + from the existing testsuites :) + +[[GSoC project propsal|community/gsoc/project_ideas/testsuites]]. + + youpi, true, and I think I should check what's available in way + of tests, but if the tests are "all or nothing" then it becomes really + hard to fix your mistakes + they're not all or nothing + youpi: the existing testsuites don't test specific system features + libc ones do + we could also check posixtestsuite which does too + +[[open_issues/open_posix_test_suite]]. + + AFAIK libc has very few failing tests + +[[open_issues/glibc_testsuite]]. + + err, like twenty? + € grep -v '^#' expected-results-i486-gnu-libc | wc -l + 67 + nope, even more + oh, sorry, I confused it with coreutils + plus the binutils ones, i guess + yes + +[[open_issues/binutils#weak]]. + + anyways, I don't think relying on external testsuites for + regression testing is a good plan + also, that doesn't cover unit testing at all + why ? + sure there can be unit testing at the translator etc. level + if we want to implement test-driven development, and/or do serious + refactoring without too much risk, we need a test harness where we can + add specific tests as needed + but more than often, the issues are at the libc / etc. level + because of a combination o fthings at the translator level, which unit + testing won't find out + * nixness yewzzz! + sure unit testing can find them out. if they're good "unit" tests + the problem is that you don't necessarily know what "good" means + e.g. for posix correctness + since it's not posix + but if they're composite clever tests, then you lose that + granularity + youpi, is that a blackbox test intended to be run at the very end + to check if you're posix compliant? + also, if we have our own test harness, we can run tests + automatically as part of the build process, which is a great plus IMHO + nixness: "that" = ? + oh nvm, I thought there was a test stuie called "posix + correctness" + there's the posixtestsuite yes + it's an external one however + antrik: being external doesn't mean we can't invoke it + automatically as part of the build process when it's available + youpi, but being internal means it can test the inner workings of + certain modules that you are unsure of, and not just the interface + sure, that's why I said it'd be useful too + but as I said too, most bugs I've seen were not easy to find out at + the unit level + but rather at the libc level + of course we can integrate external tests if they exist and are + suitable. but that that doesn't preclude adding our own ones too. in + either case, that integration work has to be done too + again, I've never said I was against internal testsuite + also, the major purpose of a test suite is checking known + behaviour. a low-level test won't directly point to a POSIX violation; + but it will catch any changes in behaviour that would lead to one + what I said is that it will be hard to write them tight enough to + find bugs + again, the problem is knowing what will lead to a POSIX violation + it's long work + while libc / posixtestsuite / etc. already do that + *any* unexpected change in behaviour is likely to cause bugs + somewher + but WHAT is "expected" ? + THAT is the issue + and libc/posixtessuite do know that + at the translator level we don't really + see the recent post about link() + +[link(dir,name) should fail with +EPERM](http://lists.gnu.org/archive/html/bug-hurd/2011-03/msg00007.html) + + in my memory jkoenig pointed it out for a lot of such calls + and that issue is clearly not known at the translator level + so you're saying that the tests have to be really really + low-level, and work our way up? + I'm saying that the translator level tests will be difficult to + write + why isn't it known at the translator level? if it's a translator + (not libc) bug, than obviously the translator must return something wrong + at some point, and that's something we can check + because you'll have to know all the details of the combinations + used in libc, to know whether they'll lead to posix issues + antrik: sure, but how did we detect that that was unexpected + behavior? + because of a glib test + at the translator level we didn't know it was an unexpected + behavior + gnulib actually + and if you had asked me, I wouldn't have known + again, we do *not* write a test suite to find existing bugs + right, took one for the other + doesn't really matter actually + antrik: ok, I don't care then + we write a test suite to prevent future bugs, or track status of + known bugs + (don't care *enough* for now, I mean) + hmm, so write something up antrik for GSoC :) and I'll be sure to + apply + now that we know some translators return a wrong error status in a + particular situation, we can write a test that checks exactly this error + status. that way we know when it is fixed, and also when it breaks again + nixness: great :-) + sweet. that kind of thing would also need a db backend + nixness: BTW, if you have a good idea, you can send an application + for it even if it's not listed among the proposed tasks... + so you don't strictly need a writeup from me to apply for this :-) + antrik, I'll keep that in mind, but I'll also be checking your + draft page + oh cool :) + (and it's a well known fact that those projects which students + proposed themselfs tend to be the most successful ones :-) ) + * nixness draft initiated + youpi: BTW, I'm surprised that you didn't mention libc testsuite + before... working up from there is probably a more effective plan than + starting with high-level test suites like Python etc... + wasn't it already in the gsoc proposal? + bummer + nope + +freenode, #hurd channel, 2011-03-06: + + how's the hurd coding workflow, typically? + +*nixness* -> *foocraft*. + + we're discussing how TDD can work with Hurd (or general kernel + development) on #osdev + so what I wanted to know, since I haven't dealt much with GNU + Hurd, is how do you guys go about coding, in this case + Our current workflow scheme is... well... is... + Someone wants to work on something, or spots a bug, then works + on it, submits a patch, and 0 to 10 years later it is applied. + Roughly. + hmm so there's no indicator of whether things broke with that + patch + and how low do you think we can get with tests? A friend of mine + was telling me that with kernel dev, you really don't know whether, for + instance, the stack even exists, and a lot of things that I, as a + programmer, can assume while writing code break when it comes to writing + kernel code + Autotest seems promising + +See autotest link given above. + + but in any case, coming up with the testing framework that + doesn't break when the OS itself breaks is hard, it seems + not sure if autotest isolates the mistakes in the os from + finding their way in the validity of the tests themselves + it could be interesting to have scripts that automatically start a + sub-hurd to do the tests + +[[hurd/subhurd#unit_testing]]. + + foocraft: To answer one of your earlier questions: you can do + really low-level testing. Like testing Mach's message passing. A + million times. The questions is whether that makes sense. And / or if + it makes sense to do that as part of a unit testing framework. Or rather + do such things manually once you suspect an error somewhere. + The reason for the latter may be that Mach IPC is already + heavily tested during normal system operation. + And yet, there still may be (there are, surely) bugs. + But I guess that you have to stop at some (arbitrary?) level. + so we'd assume it works, and test from there + Otherwise you'd be implementing the exact counter-part of what + you're testing. + Which may be what you want, or may be not. Or it may just not + be feasible. + maybe the testing framework should have dependencies + which we can automate using make, and phony targets that run + tests + so everyone writes a test suite and says that it depends on A + and B working correctly + then it'd go try to run the tests for A etc. + Hmm, isn't that -- on a high level -- have you have by + different packages? For example, the perl testsuite depends (inherently) + on glibc working properly. A perl program's testsuite depends on perl + working properly. + yeah, but afaik, the ordering is done by hand + +freenode, #hurd channel, 2011-03-07: + + actually, I think for most tests it would be better not to use a + subhurd... that leads precisely to the problem that if something is + broken, you might have a hard time running the tests at all :-) + foocraft: most of the Hurd code isn't really low-level. you can + use normal debugging and testing methods + gnumach of course *does* have some low-level stuff, so if you add + unit tests to gnumach too, you might run into issues :-) + tschwinge: I think testing IPC is a good thing. as I already said, + automated testing is *not* to discover existing but unknown bugs, but to + prevent new ones creeping in, and tracking progress on known bugs + tschwinge: I think you are confusing unit testing and regression + testing. http://www.bddebian.com/~hurd-web/open_issues/unit_testing/ + talks about unit testing, but a lot (most?) of it is actually about + regression tests... + antrik: That may certainly be -- I'm not at all an expert in + this, and just generally though that some sort of automated testing is + needed, and thus started collecting ideas. + antrik: You're of course invited to fix that. + +IRC, freenode, #hurd, 2011-03-08 + +(After discussing the [[open_issues/anatomy_of_a_hurd_system]].) + + so that's what your question is actually about? + so what I would imagine is a set of only-this-server tests for + each server, and then we can have fun adding composite tests + thus making debugging the composite scenarios a bit less tricky + indeed + and if you were trying to pass a composite test, it would also + help knowing that you still didn't break the server-only test + there are so many different things that can be tested... the + summer will only suffice to dip into this really :-) + yeah, I'm designing my proposal to focus on 1) make/use a + testing framework that fits the Hurd case very well 2) write some tests + and docs on how to write good tests + well, doesn't have to be *one* framework... unit testing and + regression testing are quite different things, which can be covered by + different frameworks diff --git a/open_issues/unit_testing.mdwn b/open_issues/unit_testing.mdwn index feda3be4..1378be85 100644 --- a/open_issues/unit_testing.mdwn +++ b/open_issues/unit_testing.mdwn @@ -8,7 +8,8 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -This task may be suitable for [[community/GSoC]][[!tag gsoc-task]]. +This task may be suitable for [[community/GSoC]]: +[[community/gsoc/project_ideas/testing_framework]] --- @@ -80,263 +81,6 @@ abandoned). # Discussion -freenode, #hurd channel, 2011-03-05: - - what about testing though? - like sort of "what's missing? lets write tests for it so that - when someone gets to implementing it, he knows what to do. Repeat" - project - you mean creating an automated testing framework? - this is actually a task I want to add for this year, if I get - around to it :-) - yeah I'd very much want to apply for that one - cuz I've never done Kernel hacking before, but I know that with - the right tasks like "test VM functionality", I would be able to write up - the automated tests and hopefully learn more about what breaks/makes the - kernel - (and it would make implementing the feature much less hand-wavy - and more correct) - antrik, I believe the framework(CUnit right?) exists, but we just - need to write the tests. - do you have prior experience implementing automated tests? - lots of tests! - yes, in Java mostly, but I've played around with CUnit - ah, great :-) - here's what I know from experience: 1) write basic tests. 2) - write ones that test multiple features 3) stress test [option 4) - benchmark and record to DB] - well, what we'd rather need is to fix the issues we already know - from the existing testsuites :) - -[[GSoC project propsal|community/gsoc/project_ideas/testsuites]]. - - youpi, true, and I think I should check what's available in way - of tests, but if the tests are "all or nothing" then it becomes really - hard to fix your mistakes - they're not all or nothing - youpi: the existing testsuites don't test specific system features - libc ones do - we could also check posixtestsuite which does too - -[[open_issues/open_posix_test_suite]]. - - AFAIK libc has very few failing tests - -[[open_issues/glibc_testsuite]]. - - err, like twenty? - € grep -v '^#' expected-results-i486-gnu-libc | wc -l - 67 - nope, even more - oh, sorry, I confused it with coreutils - plus the binutils ones, i guess - yes - -[[open_issues/binutils#weak]]. - - anyways, I don't think relying on external testsuites for - regression testing is a good plan - also, that doesn't cover unit testing at all - why ? - sure there can be unit testing at the translator etc. level - if we want to implement test-driven development, and/or do serious - refactoring without too much risk, we need a test harness where we can - add specific tests as needed - but more than often, the issues are at the libc / etc. level - because of a combination o fthings at the translator level, which unit - testing won't find out - * nixness yewzzz! - sure unit testing can find them out. if they're good "unit" tests - the problem is that you don't necessarily know what "good" means - e.g. for posix correctness - since it's not posix - but if they're composite clever tests, then you lose that - granularity - youpi, is that a blackbox test intended to be run at the very end - to check if you're posix compliant? - also, if we have our own test harness, we can run tests - automatically as part of the build process, which is a great plus IMHO - nixness: "that" = ? - oh nvm, I thought there was a test stuie called "posix - correctness" - there's the posixtestsuite yes - it's an external one however - antrik: being external doesn't mean we can't invoke it - automatically as part of the build process when it's available - youpi, but being internal means it can test the inner workings of - certain modules that you are unsure of, and not just the interface - sure, that's why I said it'd be useful too - but as I said too, most bugs I've seen were not easy to find out at - the unit level - but rather at the libc level - of course we can integrate external tests if they exist and are - suitable. but that that doesn't preclude adding our own ones too. in - either case, that integration work has to be done too - again, I've never said I was against internal testsuite - also, the major purpose of a test suite is checking known - behaviour. a low-level test won't directly point to a POSIX violation; - but it will catch any changes in behaviour that would lead to one - what I said is that it will be hard to write them tight enough to - find bugs - again, the problem is knowing what will lead to a POSIX violation - it's long work - while libc / posixtestsuite / etc. already do that - *any* unexpected change in behaviour is likely to cause bugs - somewher - but WHAT is "expected" ? - THAT is the issue - and libc/posixtessuite do know that - at the translator level we don't really - see the recent post about link() - -[link(dir,name) should fail with -EPERM](http://lists.gnu.org/archive/html/bug-hurd/2011-03/msg00007.html) - - in my memory jkoenig pointed it out for a lot of such calls - and that issue is clearly not known at the translator level - so you're saying that the tests have to be really really - low-level, and work our way up? - I'm saying that the translator level tests will be difficult to - write - why isn't it known at the translator level? if it's a translator - (not libc) bug, than obviously the translator must return something wrong - at some point, and that's something we can check - because you'll have to know all the details of the combinations - used in libc, to know whether they'll lead to posix issues - antrik: sure, but how did we detect that that was unexpected - behavior? - because of a glib test - at the translator level we didn't know it was an unexpected - behavior - gnulib actually - and if you had asked me, I wouldn't have known - again, we do *not* write a test suite to find existing bugs - right, took one for the other - doesn't really matter actually - antrik: ok, I don't care then - we write a test suite to prevent future bugs, or track status of - known bugs - (don't care *enough* for now, I mean) - hmm, so write something up antrik for GSoC :) and I'll be sure to - apply - now that we know some translators return a wrong error status in a - particular situation, we can write a test that checks exactly this error - status. that way we know when it is fixed, and also when it breaks again - nixness: great :-) - sweet. that kind of thing would also need a db backend - nixness: BTW, if you have a good idea, you can send an application - for it even if it's not listed among the proposed tasks... - so you don't strictly need a writeup from me to apply for this :-) - antrik, I'll keep that in mind, but I'll also be checking your - draft page - oh cool :) - (and it's a well known fact that those projects which students - proposed themselfs tend to be the most successful ones :-) ) - * nixness draft initiated - youpi: BTW, I'm surprised that you didn't mention libc testsuite - before... working up from there is probably a more effective plan than - starting with high-level test suites like Python etc... - wasn't it already in the gsoc proposal? - bummer - nope - -freenode, #hurd channel, 2011-03-06: - - how's the hurd coding workflow, typically? - -*nixness* -> *foocraft*. - - we're discussing how TDD can work with Hurd (or general kernel - development) on #osdev - so what I wanted to know, since I haven't dealt much with GNU - Hurd, is how do you guys go about coding, in this case - Our current workflow scheme is... well... is... - Someone wants to work on something, or spots a bug, then works - on it, submits a patch, and 0 to 10 years later it is applied. - Roughly. - hmm so there's no indicator of whether things broke with that - patch - and how low do you think we can get with tests? A friend of mine - was telling me that with kernel dev, you really don't know whether, for - instance, the stack even exists, and a lot of things that I, as a - programmer, can assume while writing code break when it comes to writing - kernel code - Autotest seems promising - -See autotest link given above. - - but in any case, coming up with the testing framework that - doesn't break when the OS itself breaks is hard, it seems - not sure if autotest isolates the mistakes in the os from - finding their way in the validity of the tests themselves - it could be interesting to have scripts that automatically start a - sub-hurd to do the tests - -[[hurd/subhurd#unit_testing]]. - - foocraft: To answer one of your earlier questions: you can do - really low-level testing. Like testing Mach's message passing. A - million times. The questions is whether that makes sense. And / or if - it makes sense to do that as part of a unit testing framework. Or rather - do such things manually once you suspect an error somewhere. - The reason for the latter may be that Mach IPC is already - heavily tested during normal system operation. - And yet, there still may be (there are, surely) bugs. - But I guess that you have to stop at some (arbitrary?) level. - so we'd assume it works, and test from there - Otherwise you'd be implementing the exact counter-part of what - you're testing. - Which may be what you want, or may be not. Or it may just not - be feasible. - maybe the testing framework should have dependencies - which we can automate using make, and phony targets that run - tests - so everyone writes a test suite and says that it depends on A - and B working correctly - then it'd go try to run the tests for A etc. - Hmm, isn't that -- on a high level -- have you have by - different packages? For example, the perl testsuite depends (inherently) - on glibc working properly. A perl program's testsuite depends on perl - working properly. - yeah, but afaik, the ordering is done by hand - -freenode, #hurd channel, 2011-03-07: - - actually, I think for most tests it would be better not to use a - subhurd... that leads precisely to the problem that if something is - broken, you might have a hard time running the tests at all :-) - foocraft: most of the Hurd code isn't really low-level. you can - use normal debugging and testing methods - gnumach of course *does* have some low-level stuff, so if you add - unit tests to gnumach too, you might run into issues :-) - tschwinge: I think testing IPC is a good thing. as I already said, - automated testing is *not* to discover existing but unknown bugs, but to - prevent new ones creeping in, and tracking progress on known bugs - tschwinge: I think you are confusing unit testing and regression - testing. http://www.bddebian.com/~hurd-web/open_issues/unit_testing/ - talks about unit testing, but a lot (most?) of it is actually about - regression tests... - antrik: That may certainly be -- I'm not at all an expert in - this, and just generally though that some sort of automated testing is - needed, and thus started collecting ideas. - antrik: You're of course invited to fix that. - -IRC, freenode, #hurd, 2011-03-08 - -(After discussing the [[anatomy_of_a_hurd_system]].) - - so that's what your question is actually about? - so what I would imagine is a set of only-this-server tests for - each server, and then we can have fun adding composite tests - thus making debugging the composite scenarios a bit less tricky - indeed - and if you were trying to pass a composite test, it would also - help knowing that you still didn't break the server-only test - there are so many different things that can be tested... the - summer will only suffice to dip into this really :-) - yeah, I'm designing my proposal to focus on 1) make/use a - testing framework that fits the Hurd case very well 2) write some tests - and docs on how to write good tests - well, doesn't have to be *one* framework... unit testing and - regression testing are quite different things, which can be covered by - different frameworks +See the [[GSoC project idea|community/gsoc/project_ideas/testing_framework]]'s +[[discussion +subpage|community/gsoc/project_ideas/testing_framework/discussion]]. -- cgit v1.2.3 From 546c76ea22295c1b889dc63c1b4332fe24340323 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 01:43:55 +0100 Subject: tag/gsoc-task: Remove. --- tag/gsoc-task.mdwn | 11 ----------- 1 file changed, 11 deletions(-) delete mode 100644 tag/gsoc-task.mdwn diff --git a/tag/gsoc-task.mdwn b/tag/gsoc-task.mdwn deleted file mode 100644 index 99758478..00000000 --- a/tag/gsoc-task.mdwn +++ /dev/null @@ -1,11 +0,0 @@ -[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -[[!meta redir=community/gsoc/project_ideas]] -- cgit v1.2.3 From 511bcd2ac7194ca47c33e45131479b8bfdef33a9 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 01:55:07 +0100 Subject: Link to google-coredumper library. --- open_issues/crash_server.mdwn | 8 +++++++- open_issues/debugging.mdwn | 3 +++ open_issues/gdb_gcore.mdwn | 5 ++++- 3 files changed, 14 insertions(+), 2 deletions(-) diff --git a/open_issues/crash_server.mdwn b/open_issues/crash_server.mdwn index d97f5458..7ed4afbf 100644 --- a/open_issues/crash_server.mdwn +++ b/open_issues/crash_server.mdwn @@ -1,4 +1,5 @@ -[[!meta copyright="Copyright © 2009, 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2009, 2010, 2011 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -187,3 +188,8 @@ one... /home/tschwinge/tmp/gnumach/gnumach-1-branch-Xen-branch.build/../gnumach-1-branch-Xen-branch/kern/ipc_kobject.c:76 mach_msg_trap /home/tschwinge/tmp/gnumach/gnumach-1-branch-Xen-branch.build/../gnumach-1-branch-Xen-branch/ipc/mach_msg.c:1367 + +--- + +If someone is working in this area, they may want to have a look at +[[GDB_gcore]], and port , too. diff --git a/open_issues/debugging.mdwn b/open_issues/debugging.mdwn index e087f484..e5fbf7a0 100644 --- a/open_issues/debugging.mdwn +++ b/open_issues/debugging.mdwn @@ -42,6 +42,9 @@ We have debugging infrastructure. For example: Continues: , which introduces . + * [[crash_server}}, [[GDB_gcore]], + + * [[community/gsoc/project_ideas/libdiskfs_locking]] * , or -- diff --git a/open_issues/gdb_gcore.mdwn b/open_issues/gdb_gcore.mdwn index 7d4980f1..69211ac0 100644 --- a/open_issues/gdb_gcore.mdwn +++ b/open_issues/gdb_gcore.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2009, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -21,3 +21,6 @@ GDB's `gcore` command doesn't work / needs to be implemented / ported in GDB: /media/data/home/tschwinge/core.cA0ICY:2: Error in sourced command file: Undefined command: "gcore". Try "help". gcore: failed to create core.8371 + +If someone is working in this area, they may want to port +, too. -- cgit v1.2.3 From 8a6c384f88eb1312aa70c75cf0791c3fee50e3ed Mon Sep 17 00:00:00 2001 From: GNU Hurd web pages engine Date: Sat, 26 Mar 2011 10:43:51 +0000 Subject: creating tag page tag/open_issues_hurd --- tag/open_issues_hurd.mdwn | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 tag/open_issues_hurd.mdwn diff --git a/tag/open_issues_hurd.mdwn b/tag/open_issues_hurd.mdwn new file mode 100644 index 00000000..0ab74c08 --- /dev/null +++ b/tag/open_issues_hurd.mdwn @@ -0,0 +1,15 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="open_issues_hurd"]] + +[[!map +pages="tagged(open_issues_hurd)" +show=title]] -- cgit v1.2.3 From a2409d8ae8be4cf14bfea4531162f0ea3924466e Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 11:57:10 +0100 Subject: Typo fix. --- community/gsoc/project_ideas/valgrind.mdwn | 2 +- tag/open_issues_hurd.mdwn | 15 --------------- 2 files changed, 1 insertion(+), 16 deletions(-) delete mode 100644 tag/open_issues_hurd.mdwn diff --git a/community/gsoc/project_ideas/valgrind.mdwn b/community/gsoc/project_ideas/valgrind.mdwn index 17385f75..e9e94857 100644 --- a/community/gsoc/project_ideas/valgrind.mdwn +++ b/community/gsoc/project_ideas/valgrind.mdwn @@ -11,7 +11,7 @@ License|/fdl]]."]]"""]] [[!meta title="Porting Valgrind to the Hurd"]] -[[!tag open_issue_gnumach open_issues_hurd]] +[[!tag open_issue_gnumach open_issue_hurd]] [Valgrind](http://valgrind.org/) is an extremely useful debugging tool for memory errors. (And some other kinds of hard-to-find errors too.) diff --git a/tag/open_issues_hurd.mdwn b/tag/open_issues_hurd.mdwn deleted file mode 100644 index 0ab74c08..00000000 --- a/tag/open_issues_hurd.mdwn +++ /dev/null @@ -1,15 +0,0 @@ -[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -[[!meta title="open_issues_hurd"]] - -[[!map -pages="tagged(open_issues_hurd)" -show=title]] -- cgit v1.2.3 From 6d15d94576f499facb9eb562baf992f10cc7c264 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 12:20:23 +0100 Subject: GSoC 2011. --- community/gsoc.mdwn | 18 +++++++++++------- index.mdwn | 6 ++++++ 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/community/gsoc.mdwn b/community/gsoc.mdwn index 5bd28738..45d3a31d 100644 --- a/community/gsoc.mdwn +++ b/community/gsoc.mdwn @@ -13,8 +13,14 @@ License|/fdl]]."]]"""]] # 2011 -This year we're again planning to participate in [Google Summer of -Code](http://socghop.appspot.com) under the GNU umbrella. +We're in! The GNU Hurd project will again participate in the [Google Summer of +Code](http://socghop.appspot.com) under the [GNU +umbrella](http://www.gnu.org/software/soc-projects/). + +On Monday, 2011-03-28 the *student application period* will +[begin](http://socghop.appspot.com/document/show/gsoc_program/google/gsoc2011/timeline), +the deadline is on Friday, 2011-04-08. That is plenty of time for discussing +your applications -- but please don't wait to the last minute! ## Applying for a Task @@ -25,15 +31,13 @@ Google Summer of Code) to mind our distinct [[student_application_form]]. We have a list of [[project_ideas]], and students are likewise encouraged to submit their own project proposals. -The Google Summer of Code 2011 has not yet started; we will later announce the -beginning of the students' application phase. However, working on one of these -projects is still a good opportunity to get started with Hurd development, even -outside of the GSoC context. - Please read up about [[contributing]] in general; and feel free to ask any questions you might have at one of our [[regular_IRC_meetings|IRC#regular_meetings]]. Generally it's a good idea to [[contact_us|communication]] when starting to work on some project. +Working on one of these projects is generally a good opportunity to get started +with Hurd development, even outside of the GSoC context. + # History diff --git a/index.mdwn b/index.mdwn index 9feb44ec..d2a4a8be 100644 --- a/index.mdwn +++ b/index.mdwn @@ -31,6 +31,12 @@ computing environment as possible. --- +[[!template id=note text="""**Breaking News** + +The *Google Summer of Code 2011* is on! If you're a student, consider applying +for a GNU/Hurd project -- details on our [[community/GSoC]] page."""]] + + [[!toc levels=2]] -- cgit v1.2.3 From aca932f35cc4c03d7cf0e7e75646899a978c5d44 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 15:04:23 +0100 Subject: Make the sidebar fixed-width. (As per style.css.) Alternatively, we could use ``width: auto; max-width: ...;''. --- local.css | 1 - 1 file changed, 1 deletion(-) diff --git a/local.css b/local.css index 8669e5e3..297a1e78 100644 --- a/local.css +++ b/local.css @@ -161,7 +161,6 @@ a:hover /* Placement. */ .sidebar { - width: auto; margin-left: 20px; } -- cgit v1.2.3 From 0fcb0d0ea237512591c9e45c6b9a37f90092ea5b Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 15:05:22 +0100 Subject: templates/highlight: New, for [[!template id=highlight]]. As [[!template id=note]], but text is not in a box shifted to the right of the page, but instead in-place. --- templates/highlight.mdwn | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 templates/highlight.mdwn diff --git a/templates/highlight.mdwn b/templates/highlight.mdwn new file mode 100644 index 00000000..c8da1377 --- /dev/null +++ b/templates/highlight.mdwn @@ -0,0 +1,3 @@ +
+ +
-- cgit v1.2.3 From 669b3e37801efc7ce3a7d0072fc48b6bc92ff3d0 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 15:16:26 +0100 Subject: GSoC 2011 in the sidebar and as a news item. --- index.mdwn | 6 ------ news/2011-03-26.mdwn | 15 +++++++++++++++ sidebar.mdwn | 11 +++++++++++ 3 files changed, 26 insertions(+), 6 deletions(-) create mode 100644 news/2011-03-26.mdwn diff --git a/index.mdwn b/index.mdwn index d2a4a8be..9feb44ec 100644 --- a/index.mdwn +++ b/index.mdwn @@ -31,12 +31,6 @@ computing environment as possible. --- -[[!template id=note text="""**Breaking News** - -The *Google Summer of Code 2011* is on! If you're a student, consider applying -for a GNU/Hurd project -- details on our [[community/GSoC]] page."""]] - - [[!toc levels=2]] diff --git a/news/2011-03-26.mdwn b/news/2011-03-26.mdwn new file mode 100644 index 00000000..588f5fcf --- /dev/null +++ b/news/2011-03-26.mdwn @@ -0,0 +1,15 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta date="2011-03-26 14:20 UTC"]] + +The **Google Summer of Code 2011** is on! If you're a student, consider +applying for a GNU Hurd project -- details to be found on our +*[[community/GSoC]] page*. diff --git a/sidebar.mdwn b/sidebar.mdwn index 501431e1..6dd2a35d 100644 --- a/sidebar.mdwn +++ b/sidebar.mdwn @@ -11,6 +11,17 @@ is included in the section entitled Welcome to... [[!img /logo/boxes-redrawn.png link=/logo]] ... the GNU Hurd! +--- + +[[!template id=highlight text="""**Breaking News** + +--- + +The **Google Summer of Code 2011** is on! If you're a student, consider +applying for a GNU Hurd project -- details to be found on our +*[[community/GSoC]] page*."""]] + +--- * **[[Home|/index]]** * **[[Community]]** * **[[Documentation]]** -- cgit v1.2.3 From a3cbf7911c321951adfe60640a93d72999150273 Mon Sep 17 00:00:00 2001 From: antrik Date: Sat, 26 Mar 2011 19:07:03 +0100 Subject: history/port_to_another_microkernel (discussion): comment on the recent merge --- history/port_to_another_microkernel/discussion.mdwn | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/history/port_to_another_microkernel/discussion.mdwn b/history/port_to_another_microkernel/discussion.mdwn index f2161195..6af8ac53 100644 --- a/history/port_to_another_microkernel/discussion.mdwn +++ b/history/port_to_another_microkernel/discussion.mdwn @@ -8,6 +8,18 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] +I can't say that I like the merge of FAQ entry an Hurd/L4 page. The stuff taken +from the L4 page creates the impression that Mach is a relic that we urgently +need to get rid of -- an impression I carefully tried to avoid in my own +writeup. Also, it's way too long IMHO. (In fact my own writeup is already on +the fringe of being too wordy I fear.) And I don't think that bare links to +other pages make good FAQ answers in general. + +IMHO the original FAQ entry should be restored (with some details fixed); while +the Hurd/L4 page can be referenced for further details. + +-- antrik + IRC, #hurd, 2011-01-12. [[!taglink open_issue_documentation]] -- cgit v1.2.3 From 2eda0c930873e95212b70fe4ca0d6b14274506a6 Mon Sep 17 00:00:00 2001 From: antrik Date: Sat, 26 Mar 2011 19:10:49 +0100 Subject: open_issues/pfinet_vs_system_time_changes: typo --- open_issues/pfinet_vs_system_time_changes.mdwn | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/open_issues/pfinet_vs_system_time_changes.mdwn b/open_issues/pfinet_vs_system_time_changes.mdwn index a9e1e242..714c8784 100644 --- a/open_issues/pfinet_vs_system_time_changes.mdwn +++ b/open_issues/pfinet_vs_system_time_changes.mdwn @@ -15,7 +15,7 @@ IRC, unknown channel, unknown date. I did a sudo date... and the machine hangs -This was very likely as misdiagnosis: +This was very likely a misdiagnosis: IRC, freenode, #hurd, 2011-03-25 -- cgit v1.2.3 From 163cf6e7a91f05ea73443fd95fd8f8b1a703b4a3 Mon Sep 17 00:00:00 2001 From: antrik Date: Sat, 26 Mar 2011 19:16:26 +0100 Subject: gsoc/ideas: Drop obsolete D-I task for now There are still some issues open with D-I, so perhaps it could be resurrected after updating -- or some of the non-trivial issues can become distinct tasks. --- community/gsoc/project_ideas.mdwn | 1 - 1 file changed, 1 deletion(-) diff --git a/community/gsoc/project_ideas.mdwn b/community/gsoc/project_ideas.mdwn index 0c249b62..4a6c2675 100644 --- a/community/gsoc/project_ideas.mdwn +++ b/community/gsoc/project_ideas.mdwn @@ -94,7 +94,6 @@ See also the list of [Hurd-related X.org project ideas](http://wiki.x.org/wiki/H [[!inline pages="community/gsoc/project_ideas/lexical_dot-dot" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/secure_chroot" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/package_manager" show=0 feeds=no actions=yes]] -[[!inline pages="community/gsoc/project_ideas/debian_installer" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/download_backends" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/libgtop" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/maxpath" show=0 feeds=no actions=yes]] -- cgit v1.2.3 From 88f00eed14d898e233677b6cc73c0f24786a9fde Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 21:02:42 +0100 Subject: open_issues/file_system_exerciser: New. --- community/gsoc/project_ideas/testsuites.mdwn | 3 +++ open_issues/file_system_exerciser.mdwn | 15 +++++++++++++++ 2 files changed, 18 insertions(+) create mode 100644 open_issues/file_system_exerciser.mdwn diff --git a/community/gsoc/project_ideas/testsuites.mdwn b/community/gsoc/project_ideas/testsuites.mdwn index 8100bfb7..f5ee2084 100644 --- a/community/gsoc/project_ideas/testsuites.mdwn +++ b/community/gsoc/project_ideas/testsuites.mdwn @@ -19,6 +19,9 @@ some of the tests fail on the Hurd in general. There is also the [[Open POSIX Testsuite|open_issues/open_posix_test_suite]] which is more of a whole system interface testing suite. +Then, there is the [[open_issues/File_System_Exerciser]] which we can use to +test our file system servers for conformity. + While in some cases these might point to wrong usage of system interfaces, most of the time such failures are actually caused by shortcomings in Hurd's implementation of these interfaces. These shortcomings are often not obvious in normal use, diff --git a/open_issues/file_system_exerciser.mdwn b/open_issues/file_system_exerciser.mdwn new file mode 100644 index 00000000..4277e5e7 --- /dev/null +++ b/open_issues/file_system_exerciser.mdwn @@ -0,0 +1,15 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +Test our file system implementations with the File System Exerciser. + + * -- cgit v1.2.3 From 172b698e79c56234fca2c95ca1bdf6aabd18bdcd Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 21:03:30 +0100 Subject: open_issues/debian_cross_toolchain: New. --- open_issues/debian_cross_toolchain.mdwn | 15 +++++++++++++++ open_issues/nightly_builds.mdwn | 4 +++- open_issues/nightly_builds_deb_packages.mdwn | 6 +++++- 3 files changed, 23 insertions(+), 2 deletions(-) create mode 100644 open_issues/debian_cross_toolchain.mdwn diff --git a/open_issues/debian_cross_toolchain.mdwn b/open_issues/debian_cross_toolchain.mdwn new file mode 100644 index 00000000..e0665466 --- /dev/null +++ b/open_issues/debian_cross_toolchain.mdwn @@ -0,0 +1,15 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +Have a look at the Debian *Cross Toolchain* project, +, +. diff --git a/open_issues/nightly_builds.mdwn b/open_issues/nightly_builds.mdwn index 506697bb..5d1257fb 100644 --- a/open_issues/nightly_builds.mdwn +++ b/open_issues/nightly_builds.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -15,6 +15,8 @@ Resources: * [[toolchain/cross-gnu]] + * [[Debian_Cross_Toolchain]] + * As reported in the [[news/2010-05-31]] news, there's Hydra doing nightly builds / Nix packages. diff --git a/open_issues/nightly_builds_deb_packages.mdwn b/open_issues/nightly_builds_deb_packages.mdwn index 9f5e2373..11fc4c79 100644 --- a/open_issues/nightly_builds_deb_packages.mdwn +++ b/open_issues/nightly_builds_deb_packages.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -24,4 +24,8 @@ There is infrastructure available to test whole OS installations. --- +[[Debian_Cross_Toolchain]] for cross-building? + +--- + See also [[nightly_builds]]. -- cgit v1.2.3 From 4f90536eeffb05781f7f33c153f17041eeeeb482 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 21:04:06 +0100 Subject: open_issues/multithreading: Concurrency Kit. --- open_issues/multithreading.mdwn | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/open_issues/multithreading.mdwn b/open_issues/multithreading.mdwn index 39203333..addc29c3 100644 --- a/open_issues/multithreading.mdwn +++ b/open_issues/multithreading.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -10,7 +10,21 @@ License|/fdl]]."]]"""]] [[!tag open_issue_hurd]] -Hurd servers / VFS libraries are multithreaded, roughly using one thread per +Hurd servers / VFS libraries are multithreaded. + + +# Implementation + + * well-known threading libraries + + * [[hurd/libthreads]] + + * [[hurd/libpthread]] + + +# Design + +Roughly using one thread per incoming request. This is not the best approach: it doesn't really make sense to scale the number of worker threads with the number of incoming requests, but instead they should be scaled according to the backends' characteristics. @@ -18,7 +32,9 @@ instead they should be scaled according to the backends' characteristics. The [[hurd/Critique]] should have some more on this. -Alternative approaches: +# Alternative approaches: + + * * Continuation-passing style -- cgit v1.2.3 From 3268d34951db85a913d227b637759c57e6d13f37 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 21:14:54 +0100 Subject: open_issues/performance/fork: Suggest some reading about Windows / Cygwin issues. --- open_issues/performance/fork.mdwn | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/open_issues/performance/fork.mdwn b/open_issues/performance/fork.mdwn index 2748be53..5ceb6455 100644 --- a/open_issues/performance/fork.mdwn +++ b/open_issues/performance/fork.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -14,3 +14,24 @@ Our [[`fork` implementation|glibc/fork]] is nontrivial. To do: hard numbers. [[Microbenchmarks]]? + + +# Windows / Cygwin + + * + + * + + In particular, *5.6. Process Creation*. + + * + + * + + > Cygwin has recently adopted something called the "cygwin heap". This is + > an internal heap that is inherited by forked/execed children. It + > consists of process specific information that should be inherited. So + > things like the file descriptor table, the current working directory, and + > the chroot value live there. + + * -- cgit v1.2.3 From 7200ca98a219bba9241f84c494ebaf447002669c Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sat, 26 Mar 2011 21:20:18 +0100 Subject: news/2010: French article by Manuel Menal, Gnu : L'année 2010 du Hurd. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- news/2010.mdwn | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/news/2010.mdwn b/news/2010.mdwn index f6a86052..2ba85266 100644 --- a/news/2010.mdwn +++ b/news/2010.mdwn @@ -122,4 +122,9 @@ interested in contributing to Hurd development in general, please see talk to us at or the `#hurd` IRC channel on freenode. +--- + +French article by Manuel Menal, [*Gnu : L'année 2010 du +Hurd*](http://linuxfr.org/news/lann%C3%A9e-2010-du-hurd). + """]] -- cgit v1.2.3 From 29384d59a2503c5a3cc46ce12c95bedc8653986d Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sun, 27 Mar 2011 11:22:22 +0200 Subject: open_issues/sync_but_still_unclean_filesystem: Link to Savannah bug report. --- open_issues/sync_but_still_unclean_filesystem.mdwn | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/open_issues/sync_but_still_unclean_filesystem.mdwn b/open_issues/sync_but_still_unclean_filesystem.mdwn index f1fbb4e0..83c7951e 100644 --- a/open_issues/sync_but_still_unclean_filesystem.mdwn +++ b/open_issues/sync_but_still_unclean_filesystem.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -10,9 +10,19 @@ License|/fdl]]."]]"""]] [[!tag open_issue_gnumach open_issue_hurd]] +Also filed as [[!GNU_Savannah_bug 29292]]. + \#hurd, 2010, end of May / beginning of June [runnign sync, but sill unclean filesystem at next boot] - guillem: when libpager syncs an object, it sends an m_o_lock_request and waits (if the synchronous argument was specified) for a m_o_lock_completed. But m_o_lock_completed only means that dirty pages have been sent to the translator, and this one still needs to write them to the backing storage - guillem: there's no problem if sync() returns before actually writting the changes to disk, but this also happens when shutting down the translator - guillem: in theory, locking mechanisms in libpager should prevent this from happening by keeping track of write operations, but this seems to fail in some situations + guillem: when libpager syncs an object, it sends an m_o_lock_request + and waits (if the synchronous argument was specified) for a + m_o_lock_completed. But m_o_lock_completed only means that dirty pages + have been sent to the translator, and this one still needs to write them + to the backing storage + guillem: there's no problem if sync() returns before actually + writting the changes to disk, but this also happens when shutting down + the translator + guillem: in theory, locking mechanisms in libpager should prevent + this from happening by keeping track of write operations, but this seems + to fail in some situations -- cgit v1.2.3 From 77b67e621d1d3efc94c065f31a61fe1bdc7a3381 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sun, 27 Mar 2011 12:26:59 +0200 Subject: community/gsoc/project_ideas: Two X.org Hurd ideas pages? --- community/gsoc/project_ideas.mdwn | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/community/gsoc/project_ideas.mdwn b/community/gsoc/project_ideas.mdwn index 0c249b62..e71ac193 100644 --- a/community/gsoc/project_ideas.mdwn +++ b/community/gsoc/project_ideas.mdwn @@ -76,6 +76,11 @@ will assist you as well as we can. See also the list of [Hurd-related X.org project ideas](http://wiki.x.org/wiki/Hurd_Porting). + + [[!inline pages="community/gsoc/project_ideas/language_bindings" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/virtualization" show=0 feeds=no actions=yes]] [[!inline pages="community/gsoc/project_ideas/file_locking" show=0 feeds=no actions=yes]] -- cgit v1.2.3 From 5920d96d58d9df2e8322c2d85f0ec9c72b205ae9 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sun, 27 Mar 2011 12:43:59 +0200 Subject: history/port_to_another_microkernel/discussion: Ack. --- history/port_to_another_microkernel/discussion.mdwn | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/history/port_to_another_microkernel/discussion.mdwn b/history/port_to_another_microkernel/discussion.mdwn index 6af8ac53..1a06c8ba 100644 --- a/history/port_to_another_microkernel/discussion.mdwn +++ b/history/port_to_another_microkernel/discussion.mdwn @@ -20,6 +20,12 @@ the Hurd/L4 page can be referenced for further details. -- antrik +> OK, I understand (and accept) your rationale; [[!taglink +> open_issue_documentation]]. --[[tschwinge]] + + +--- + IRC, #hurd, 2011-01-12. [[!taglink open_issue_documentation]] -- cgit v1.2.3