Some more IRC discussions.

6 files changed, 321 insertions, 16 deletions

diff --git a/open_issues/anatomy_of_a_hurd_system.mdwn b/open_issues/anatomy_of_a_hurd_system.mdwn
new file mode 100644
index 00000000..e1d5c9d8
--- /dev/null
+++ b/open_issues/anatomy_of_a_hurd_system.mdwn
@@ -0,0 +1,73 @@
+[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
+
+[[!taglink open_issue_documentation]]
+
+A bunch of this should also be covered in other (introductionary) material,
+like Bushnell's Hurd paper.  All this should be unfied and streamlined.
+
+IRC, freenode, #hurd, 2011-03-08
+
+    <foocraft> I've a question on what are the "units" in the hurd project, if
+      you were to divide them into units if they aren't, and what are the
+      dependency relations between those units(roughly, nothing too pedantic
+      for now)
+    <antrik> there is GNU Mach (the microkernel); there are the server
+      libraries in the Hurd package; there are the actual servers in the same;
+      and there is the POSIX implementation layer in glibc
+    <antrik> relations are a bit tricky
+    <antrik> Mach is the base layer which implements IPC and memory management
+    <foocraft> hmm I'll probably allocate time for dependency graph generation,
+      in the worst case
+    <antrik> on top of this, the Hurd servers, using the server libraries,
+      implement various aspects of the system functionality
+    <antrik> client programs use libc calls to use the servers
+    <antrik> (servers also use libc to communicate with other servers and/or
+      Mach though)
+    <foocraft> so every server depends solely on mach, and no other server?
+    <foocraft> s/mach/mach and/or libc/
+    <antrik> I think these things should be pretty clear one you are somewhat
+      familiar with the Hurd architecture... nothing really tricky there
+    <antrik> no
+    <antrik> servers often depend on other servers for certain functionality
+
+---
+
+IRC, freenode, #hurd, 2011-03-12
+
+    <dEhiN> when mach first starts up, does it have some basic i/o or fs
+      functionality built into it to start up the initial hurd translators?
+    <antrik> I/O is presently completely in Mach
+    <antrik> filesystems are in userspace
+    <antrik> the root filesystem and exec server are loaded by grub
+    <dEhiN> o I see
+    <dEhiN> so in order to start hurd, you would have to start mach and
+      simultaneously start the root filesystem and exec server?
+    <antrik> not exactly
+    <antrik> GRUB loads all three, and then starts Mach. Mach in turn starts
+      the servers according to the multiboot information passed from GRUB
+    <dEhiN> ok, so does GRUB load them into ram?
+    <dEhiN> I'm trying to figure out in my mind how hurd is initially started
+      up from a low-level pov
+    <antrik> yes, as I said, GRUB loads them
+    <dEhiN> ok, thanks antrik...I'm new to the idea of microkernels, but a
+      veteran of monolithic kernels
+    <dEhiN> although I just learned that windows nt is a hybrid kernel which I
+      never knew!
+    <rm> note there's a /hurd/ext2fs.static
+    <rm> I belive that's what is used initially... right?
+    <antrik> yes
+    <antrik> loading the shared libraries in addition to the actual server
+      would be unweildy
+    <antrik> so the root FS server is linked statically instead
+    <dEhiN> what does the root FS server do?
+    <antrik> well, it serves the root FS ;-)
+    <antrik> it also does some bootstrapping work during startup, to bring the
+      rest of the system up
diff --git a/open_issues/ext2fs_page_cache_swapping_leak.mdwn b/open_issues/ext2fs_page_cache_swapping_leak.mdwn
new file mode 100644
index 00000000..0ace5cd3
--- /dev/null
+++ b/open_issues/ext2fs_page_cache_swapping_leak.mdwn
@@ -0,0 +1,23 @@
+[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
+
+[[!tag open_issue_hurd]]
+
+IRC, OFTC, #debian-hurd, 2011-03-24
+
+    <youpi> I still believe we have an ext2fs page cache swapping leak, however
+    <youpi> as the 1.8GiB swap was full, yet the ld process was only 1.5GiB big
+    <pinotree> a leak at swapping time, you mean?
+    <youpi> I mean the ext2fs page cache being swapped out instead of simply
+      dropped
+    <pinotree> ah
+    <pinotree> so the swap tends to accumulate unuseful stuff, i see
+    <youpi> yes
+    <youpi> the disk content, basicallyt :)
diff --git a/open_issues/pfinet_vs_system_time_changes.mdwn b/open_issues/pfinet_vs_system_time_changes.mdwn
new file mode 100644
index 00000000..a9e1e242
--- /dev/null
+++ b/open_issues/pfinet_vs_system_time_changes.mdwn
@@ -0,0 +1,42 @@
+[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
+
+[[!tag open_issue_hurd]]
+
+IRC, unknown channel, unknown date.
+
+    <grey_gandalf> I did a sudo date...
+    <grey_gandalf> and the machine hangs
+
+This was very likely as misdiagnosis:
+
+IRC, freenode, #hurd, 2011-03-25
+
+    <tschwinge> antrik: I suspect it'S some timing stuff in pfinet that perhaps
+      uses absolute time, and somehow wildely gets confused?
+    <antrik> tschwinge: BTW, pfinet doesn't actually die I think -- it just
+      drops open connections...
+    <antrik> perhaps it thinks they timed out
+    <tschwinge> antrik: Isn't the translator restarted instead?
+    <antrik> don't think so
+    <antrik> when pfinet actually dies, I also loose the NFS mounts, which
+      doesn't happen in this case
+    <antrik> hehe "... and the machine hangs"
+    <antrik> he didn't bother to check that the machine is perfectly fine, only
+      the SSH connection got dropped
+    <tschwinge> Ah, I see.  So it'S perhaps indeed simply closes TCP
+      connections that have been without data for ``too long''?
+    <antrik> yeah, that's my guess
+    <antrik> my clock is speeding, so ntpdate sets it in the past
+    <antrik> perhaps there is some math that concludes the connection have been
+      inactive for -200 seconds, which (unsigned) is more than any timeout :-)
+    <tschwinge> (The other way round, you might likely get some integer
+      wrap-around, and thus the same result.)
+    <tschwinge> Yes.
diff --git a/open_issues/rpc_to_self_with_rendez-vous_leading_to_duplicate_port_destroy.mdwn b/open_issues/rpc_to_self_with_rendez-vous_leading_to_duplicate_port_destroy.mdwn
new file mode 100644
index 00000000..9db92250
--- /dev/null
+++ b/open_issues/rpc_to_self_with_rendez-vous_leading_to_duplicate_port_destroy.mdwn
@@ -0,0 +1,163 @@
+[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
+
+[[!tag open_issue_hurd]]
+
+[RPC to self with rendez-vous leading to duplicate port
+destroy](http://lists.gnu.org/archive/html/bug-hurd/2011-03/msg00045.html)
+
+IRC, freenode, #hurd, 2011-03-14
+
+    <antrik> youpi: I wonder, why does the root FS call diskfs_S_dir_lookup()
+      at all?...
+    <youpi> errr, because a client asked for it?
+    <youpi> (problem with RPCs is you can't easily know where they come from :)
+      )
+    <youpi> (especially when it's the root fs...)
+    <antrik> ah, it's about a client request... didn't see that
+    <youpi> well, I just said "is called", yes
+    <antrik> I do not really understand though why it tries to reauthenticate
+      against itself...
+    <antrik> I fear my memory of the lookup mechanism grew a bit dim
+    <youpi> see the source
+    <youpi> it's about a translated entry
+    <antrik> (and I never fully understood some aspects anyways...)
+    <youpi> it needs to start the translated entry as another user, possibly
+    <antrik> yes, but a translated entry normally would be served by *another*
+      process?...
+    <youpi> sure, but ext2fs has to prepare it
+    <youpi> thus reauthenticate to prepare the correct set of rights
+    <antrik> prepare what?
+    <youpi> rights
+    <youpi> so the process is not root, doesn't have / opened as root, etc.
+    <antrik> rights for what?
+    <youpi> err, about everything
+    <antrik> IIRC the reauthentication is done by the parent FS on the port to
+      the *translated* node
+    <antrik> and the translated node should be a different process?...
+    <youpi> that's not what I read in the source
+    <youpi> fshelp_fetch_root
+    <youpi> ports[INIT_PORT_CRDIR] = reauth (getcrdir ());
+    <youpi> here, getcrdir() returns ext2fs itself
+    <antrik> well, perhaps the issue is that I have no idea what
+      fshelp_fetch_root() does, nor why it is called here...
+    <youpi> it notably starts the translator that dir_lookup is looking at, if
+      needed
+    <youpi> possibly as a different user, thus reauthentication of CRDIR
+    <antrik> so this is about a port that is passed to the translator being
+      started?
+    <youpi> no
+    <youpi> well, depends on what you mean by "port"
+    <youpi> it's about reauthenticating a port to be passed to the translator
+      being started
+    <youpi> and for that a rendez-vous port is needed for the reauthentication
+    <youpi> and that's the one at stake
+    <antrik> yeah, I meant the port that is reauthenticated
+    <antrik> what is CRDIR?
+    <youpi> current root dir ...
+    <antrik> so the parent translator passes it's own root dir to the child
+      translator; and the issue is that for the root FS the root dir points to
+      the root FS itself...
+    <youpi> yes
+    <antrik> OK, that makes sense
+    <youpi> (but that's only one example, rgrep mach_port_destroy hurd/ show
+      other potential issues)
+    <antrik> well, that's actually what I wanted to mention next... why is the
+      rendez-vous port destroyed, instead of just deallocating the port right
+      and letting reference counting to it's thing?...
+    <antrik> do its thing
+    <youpi> "just to make sure" I guess
+    <antrik> it's pretty obvious that this will cause trouble for any RPC
+      referencing itself...
+    <youpi> well, follow-up with that on the list
+    <youpi> with roland/tb in CC
+    <youpi> only they would know any real reason for destroy
+    <youpi> btw, if you knew how we could make _hurd_select()'s raw __mach_msg
+      call be interruptible by signals, that'll permit to fix sudo
+    <youpi> (damn, I need sleep, my tenses are all wrong)
+    <antrik> BTW, does this cause any actual trouble?...
+    <antrik> I don't know much about interruption... cfhammer might have a
+      better idea, he look into that stuff quite a bit AIUI
+    <antrik> looked
+    <antrik> (hehe, it's not only your tenses... guess there's something in the
+      ether ;-) )
+    <youpi> it makes sudo, mailq, etc. fail sometimes
+    <antrik> I mean the rendez-vous thing
+    <youpi> that's it, yes 
+    <youpi> sudo etc. fail at least due to this
+    <antrik> so these are two different problems that both affect sudo?
+    <antrik> (rendez-vous and interruption I mean)
+    <youpi> yes
+    <youpi> with my patch the buildds have much fewer issues, but still some
+    <youpi> (my interrupt-related patch)
+    <youpi> I'm installing a s/destroy/deallocate/ version of ext2fs on the
+      buildds, we'll see how it behaves
+    <youpi> (it fixes my testcase at least)
+    <antrik> interrupt-related patch?
+    <antrik> only thing interrupt-related I remember was the reauthentication
+      race...
+    <youpi> that's what I mean
+    <antrik> well, cfhammer investigated this is quite some depth, explaining
+      quite well why the race is only mitigated but still exists... problem is
+      that we didn't know how to fix it properly
+    <antrik> because nobody seems to understand the cancellation code, except
+      perhaps for Roland and Thomas
+    <antrik> (and I'm not even entirely sure about them :-) )
+    <antrik> I think his findings and our conclusions are documented on the
+      ML...
+    <youpi> by "much fewer issues", I mean that some of the symptoms have
+      disappeared, others haven't
+    <antrik> BTW, couldn't the rendez-vous thing be worked around by simply
+      ignoring the errors from the failing deallocate?...
+    <youpi> no, failing deallocate are actually dangerous
+    <antrik> why?
+    <youpi> since the name might have been reused for something else in the
+      meanwhile
+    <youpi> that's the whole point of the warning I had added in the kernel
+      itself
+    <antrik> I see
+    <youpi> such things really deserve tracking, since they can have any kind
+      of consequence
+    <antrik> does Mach try to reuse names quickly, rather than only after
+      wrapping around?...
+    <youpi> it seems to
+    <antrik> OK, then this is a serious problem indeed
+    <youpi> (note: I rarely divine issues when there aren't actual frequent
+      symptoms :) )
+    <antrik> well, the problem with the warning is that it only shows in the
+      cases that do *not* cause a problem... so it's hard to associate them
+      with any specific issues
+    <youpi> well, most of the time the port is not reused quickly enough
+    <youpi> so in most case it shows up more often than causing problem
+
+IRC, freenode, #hurd, 2011-03-14
+
+    <youpi> ok, mach_port_deallocate actually can't be used
+    <youpi> since mach_reply_port() returns a receive right, not a send right
+    * youpi guesses he will really have to manage to understand all that port
+        stuff completely
+    <antrik> oh, right
+    <antrik> youpi: hm... now I'm confused though. if one client holds a
+      receive right, the other client (or in this case the same process) should
+      have a send or send-once right -- these should *not* share the same name
+      in my understanding
+    <antrik> destroying the receive right should turn the send right into a
+      dead name
+    <antrik> so unless I'm missing something, the destroy shouldn't be a
+      problem, and there must be something else going wrong
+    <antrik> hm... actually I'm probably wrong
+    <antrik> yeah, definitely wrong. receive rights and "ordinary" send rights
+      share the name. only send-once rights are special
+    <antrik> I wonder whether the problem could be worked around by using a
+      send-once right...
+    <antrik> mach_port_mod_refs(mach_task_self(), name,
+      MACH_PORT_RIGHT_RECEIVE, -1) can be used to deallocate only the receive
+      right
+    <antrik> oh, you already figured that out :-)
diff --git a/open_issues/sudo_date_crash.mdwn b/open_issues/sudo_date_crash.mdwn
deleted file mode 100644
index 53303abc..00000000
--- a/open_issues/sudo_date_crash.mdwn
+++ /dev/null
@@ -1,16 +0,0 @@
-[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]]
-
-[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
-id="license" text="Permission is granted to copy, distribute and/or modify this
-document under the terms of the GNU Free Documentation License, Version 1.2 or
-any later version published by the Free Software Foundation; with no Invariant
-Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
-is included in the section entitled [[GNU Free Documentation
-License|/fdl]]."]]"""]]
-
-[[!tag open_issue_gnumach]]
-
-IRC, unknown channel, unknown date.
-
-    <grey_gandalf> I did a sudo date...
-    <grey_gandalf> and the machine hangs
diff --git a/open_issues/unit_testing.mdwn b/open_issues/unit_testing.mdwn
index a5ffe19d..feda3be4 100644
--- a/open_issues/unit_testing.mdwn
+++ b/open_issues/unit_testing.mdwn
@@ -320,3 +320,23 @@ freenode, #hurd channel, 2011-03-07:
       this, and just generally though that some sort of automated testing is
       needed, and thus started collecting ideas.
     <tschwinge> antrik: You're of course invited to fix that.
+
+IRC, freenode, #hurd, 2011-03-08
+
+(After discussing the [[anatomy_of_a_hurd_system]].)
+
+    <antrik> so that's what your question is actually about?
+    <foocraft> so what I would imagine is a set of only-this-server tests for
+      each server, and then we can have fun adding composite tests
+    <foocraft> thus making debugging the composite scenarios a bit less tricky
+    <antrik> indeed
+    <foocraft> and if you were trying to pass a composite test, it would also
+      help knowing that you still didn't break the server-only test
+    <antrik> there are so many different things that can be tested... the
+      summer will only suffice to dip into this really :-)
+    <foocraft> yeah, I'm designing my proposal to focus on 1) make/use a
+      testing framework that fits the Hurd case very well 2) write some tests
+      and docs on how to write good tests
+    <antrik> well, doesn't have to be *one* framework... unit testing and
+      regression testing are quite different things, which can be covered by
+      different frameworks