summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
-rw-r--r--hurd/faq/slash_usr_symlink/discussion.mdwn45
-rw-r--r--hurd/translator.mdwn5
-rw-r--r--hurd/translator/devfs.mdwn25
-rw-r--r--hurd/translator/tmpfs/discussion.mdwn93
-rw-r--r--microkernel/mach/memory_object/discussion.mdwn13
-rw-r--r--microkernel/mach/pmap.mdwn74
-rw-r--r--microkernel/viengoos/documentation.mdwn14
-rw-r--r--microkernel/viengoos/documentation/irc_2012-02-23.mdwn159
-rw-r--r--open_issues/boehm_gc.mdwn42
-rw-r--r--open_issues/bpf.mdwn122
-rw-r--r--open_issues/dde.mdwn350
-rw-r--r--open_issues/default_pager.mdwn8
-rw-r--r--open_issues/glibc_madvise_vs_static_linking.mdwn18
-rw-r--r--open_issues/gnumach_memory_management.mdwn10
-rw-r--r--open_issues/linux_as_the_kernel.mdwn42
-rw-r--r--open_issues/memory_object_model_vs_block-level_cache.mdwn273
-rw-r--r--open_issues/select.mdwn187
-rw-r--r--open_issues/trust_the_behavior_of_translators.mdwn181
-rw-r--r--public_hurd_boxen/xen_handling.mdwn9
19 files changed, 1643 insertions, 27 deletions
diff --git a/hurd/faq/slash_usr_symlink/discussion.mdwn b/hurd/faq/slash_usr_symlink/discussion.mdwn
new file mode 100644
index 00000000..219e14e4
--- /dev/null
+++ b/hurd/faq/slash_usr_symlink/discussion.mdwn
@@ -0,0 +1,45 @@
+[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
+
+[[!tag open_issue_documentation]]
+
+
+# IRC, freenode, #hurd, 2012-02-01
+
+ <marcusb> I remember the time when we had a /usr symlink. Now fedora 17
+ will move / to /usr and have /foo symlinks. :)
+ <marcusb> braunr:
+ http://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge
+ <marcusb> braunr: fedora and others are merging /bin, /sbin and some other
+ into /usr
+ <marcusb> braunr: back in 1998 we tried for two years or so to have /usr ->
+ .. in Debian GNU/Hurd, but eventually we gave up on it, because it broke
+ some stuff
+ <gnu_srs> marcusb: Hi, which one is better (in your opinion): / or /usr?
+ <marcusb> gnu_srs: fedora says that using /usr allows better separation of
+ distribution files and machine-local files
+ <braunr> marcusb: won't it break remote /usr ?
+ <marcusb> so you can atomically mount the OS files to /usr
+ <marcusb> gnu_srs: but in the end, it's a wash
+ <marcusb> personally, I think every package should get its own directory
+ <braunr> marcusb: what PATH then ?
+ <marcusb> braunr: well, I guess you'd want to assemble a union filesystem
+ for a POSIX shell
+ <braunr> marcusb: i don't see what you mean :/
+ <braunr> ah this comes from Lennart Poettering
+ <marcusb> braunr: check out for example how http://nixos.org/ does it
+ <manuel> braunr: something like, union /package1/bin /package2/bin
+ /package3/bin for /bin, /package1/lib /package2/lib /package3/lib for
+ /lib, etc. I guess
+ <braunr> manuel: would that scale well ?
+ <marcusb> the idea that there is only one correct binary for each program
+ with the name foo is noble, but a complete illusion that hides the
+ complexity of the actual configuration management task
+ <braunr> marcusb: right
diff --git a/hurd/translator.mdwn b/hurd/translator.mdwn
index 3527267f..619c0db5 100644
--- a/hurd/translator.mdwn
+++ b/hurd/translator.mdwn
@@ -1,4 +1,4 @@
-[[!meta copyright="Copyright © 2007, 2008, 2009, 2010, 2011 Free Software
+[[!meta copyright="Copyright © 2007, 2008, 2009, 2010, 2011, 2012 Free Software
Foundation, Inc."]]
[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
@@ -75,7 +75,8 @@ available.
Read about translator [[short-circuiting]].
The [[concept|concepts]] of translators creates its own problems, too:
-[[open_issues/translators_set_up_by_untrusted_users]].
+[[open_issues/translators_set_up_by_untrusted_users]], and
+[[trust_the_behavior_of_translators]].
# Existing Translators
diff --git a/hurd/translator/devfs.mdwn b/hurd/translator/devfs.mdwn
index 27df23aa..8784e998 100644
--- a/hurd/translator/devfs.mdwn
+++ b/hurd/translator/devfs.mdwn
@@ -1,12 +1,12 @@
-[[!meta copyright="Copyright © 2009 Free Software Foundation, Inc."]]
+[[!meta copyright="Copyright © 2009, 2012 Free Software Foundation, Inc."]]
[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
id="license" text="Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License, Version 1.2 or
any later version published by the Free Software Foundation; with no Invariant
Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license
-is included in the section entitled
-[[GNU Free Documentation License|/fdl]]."]]"""]]
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
`devfs` is a translator sitting on `/dev` and providing what is to be provided
in there in a dynamic fashion -- as compared to static passive translator
@@ -18,3 +18,22 @@ settings as they're used now.
If applicable, it has to be taken care that all code concerning the page-in
path is resident at all times.
+
+---
+
+# IRC, freenode, #hurd, 2012-01-29
+
+ <pinotree> what would be an hurdish way to achieve something like the
+ various system (udev, devfs, devd, etc) for populating devices files
+ automatically according to the found system devices?
+ <pinotree> (not that i plan anything about that, just curious)
+ <youpi> it's not really a stupid question at all :)
+ <youpi> I guess translators in /dev
+ <youpi> such as a blockfs on /dev/block
+ <antrik> pinotree: in an ideal world (userspace drivers and all), the
+ device nodes will be exported by the drivers themselfs; and the drivers
+ will be launched by the bus respective bus driver
+ <antrik> an interesting aspect is what to do if we want a traditional flat
+ /dev directory with unique device names... probably need some
+ unionfs-like translator that collects the individual driver nodes in an
+ intelligent manner
diff --git a/hurd/translator/tmpfs/discussion.mdwn b/hurd/translator/tmpfs/discussion.mdwn
index 0409f046..1d441c7d 100644
--- a/hurd/translator/tmpfs/discussion.mdwn
+++ b/hurd/translator/tmpfs/discussion.mdwn
@@ -283,3 +283,96 @@ License|/fdl]]."]]"""]]
<mcsim> what kind of log do you mean?
<antrik> vmstat 1 I mean
<mcsim> ah...
+
+
+## IRC, freenode, #hurd, 2012-02-01
+
+ <mcsim> I run fsx with this command: fsx -N3000 foo/bar -S4
+ -l$((1024*1024*8)). And after 70 commands it breaks.
+ <mcsim> The strangeness is at address 0xc000 there is text, which was
+ printed in fsx with vfprintf
+ <mcsim> I've lost log. Wait a bit, while I generate new
+ <jkoenig_> mcsim, what's fsx / where can I find it ?
+ <mcsim> fsx is filesystem exersiser
+ <mcsim> http://codemonkey.org.uk/projects/fsx/
+ <jkoenig_> ok thanks
+ <mcsim> i use it to test tmpfs
+ <mcsim> here is fsx that compiles on linux: http://paste.debian.net/154390/
+ and Makefile for it: http://paste.debian.net/154392/
+ <jkoenig_> mcsim, hmm, I get a failure with ext2fs too, is it expected?
+ <mcsim> yes
+ <mcsim> i'll show you logs with tmpfs. They slightly differ
+ <mcsim> here: http://paste.debian.net/154399/
+ <mcsim> pre last operation is truncate
+ <mcsim> and last is read
+ <mcsim> during pre-last (or last) starting from address 0xa000, every
+ 0x1000 bytes appears text
+ <mcsim> skipping zero size read
+ <mcsim> skipping zero size read
+ <mcsim> truncating to largest ever: 0x705f4b
+ <mcsim> signal 2
+ <mcsim> testcalls = 38
+ <mcsim> this text is printed by fsx, by function prt
+ <mcsim> I've mistaken: this text appears even from every beginning
+ <mcsim> I know that this text appears exactly at this moment, because I
+ added check of the whole file after every step. And this error appeared
+ only after last truncation.
+ <mcsim> I think that the problem is in defpager (I'm fixing it), but I
+ don't understand where defpager could get this text
+ <jkoenig_> wow I get java code and debconf templates
+ <mcsim> So, my question is: is it possible for defpager to get somehow this
+ text?
+ <jkoenig_> possibly recycled, non-zeroed pages?
+ <mcsim> hmmm... probably you're right
+ <jkoenig_> 0x1000 bytes is consistent with the page size
+ <mcsim> Should I clean these pages in tmpfs?
+ <mcsim> or in defpager?
+ <mcsim> What is proper way?
+ <jkoenig_> mcsim, I'd say defpager should do it, to avoid leaking
+ information, I'm not sure though.
+ <jkoenig_> maybe tmpfs should also not assume the pages have been blanked
+ out.
+ <mcsim> if i do it in both, it could have big influence on performance.
+ <mcsim> i'll do it only in defpager so far.
+ <mcsim> jkoenig_: Thank you a lot
+ <jkoenig_> mcsim, no problem.
+
+
+## IRC, freenode, #hurd, 2012-02-08
+
+ <tschwinge> mcsim: You pushed another branch with cleaned-up patches?
+ <mcsim> yes.
+ <tschwinge> mcsim: Anyway, any data from your report that we could be
+ interested in? (Though it's not in English.)
+ <mcsim> It's completely in ukrainian an and mostly describes some aspects
+ of hurd's work.
+ <tschwinge> mcsim: OK. So you ran out of time to do the benchmarking,
+ etc.?
+ <tschwinge> Comparing tmpfs to ext2fs with RAM backend, etc., I mean.
+ <mcsim> tschwinge: I made benchmarking and it turned out that tmpfs up to 6
+ times faster than ext2fs
+ <mcsim> tschwinge: is it possible to have a review of work, I've already
+ done, even if parallel writing doesn't work?
+ <tschwinge> mcsim: Do you need this for university or just a general review
+ for inclusion in the Git master branch?
+ <mcsim> general review
+ <tschwinge> Will need to find someone who feels competent to do that...
+ <mcsim> the branch that should be checked is tmpfs-final
+ <pinotree> cool, i guess you tested also special types of files like
+ sockets and pipes? (they are used in eg /run, /var/run or similar)
+ <mcsim> Oh. I accidentally created this branch. It is my private
+ branch. I'll delete it now and merge everything to mplaneta/tmpfs/master
+ <mcsim> pinotree: Completely forgot about them :( I'll do it by all means
+ <pinotree> mcsim: no worries :)
+ <mcsim> tschwinge: Ready. The right branch is mplaneta/tmpfs/master
+
+
+## IRC, freenode, #hurd, 2012-03-07
+
+ <pinotree> did you test it with sockets and pipes?
+ <mcsim> pinotree: pipes work and sockets seems to work too (I've created
+ new pfinet device for them and pinged it).
+ <pinotree> try with simple C apps
+ <mcsim> Anyway all these are just translators, so there shouldn't be any
+ problems.
+ <mcsim> pinotree: works
diff --git a/microkernel/mach/memory_object/discussion.mdwn b/microkernel/mach/memory_object/discussion.mdwn
index a2a1514b..907f859a 100644
--- a/microkernel/mach/memory_object/discussion.mdwn
+++ b/microkernel/mach/memory_object/discussion.mdwn
@@ -1,4 +1,4 @@
-[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]]
+[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]]
[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
id="license" text="Permission is granted to copy, distribute and/or modify this
@@ -10,7 +10,10 @@ License|/fdl]]."]]"""]]
[[!tag open_issue_documentation open_issue_gnumach]]
-IRC, freenode, #hurd, 2011-08-05:
+[[!toc]]
+
+
+# IRC, freenode, #hurd, 2011-08-05
< neal> braunr: For instance, memory objects are great as they allow you to
specify the mapping policy in user space.
@@ -23,7 +26,8 @@ IRC, freenode, #hurd, 2011-08-05:
< braunr> the kernel eviction policy :)
< neal> that's an implementation detail
-IRC, freenode, #hurd, 2011-09-05:
+
+# IRC, freenode, #hurd, 2011-09-05
<braunr> mach isn't a true modern microkernel, it handles a lot of
resources, such as high level virtual memory and cpu time
@@ -65,3 +69,6 @@ IRC, freenode, #hurd, 2011-09-05:
pages are going to be flushed by themselves
[[open_issues/resource_management_problems]].
+
+
+# [[open_issues/memory_object_model_vs_block-level_cache]]
diff --git a/microkernel/mach/pmap.mdwn b/microkernel/mach/pmap.mdwn
new file mode 100644
index 00000000..6910bfd3
--- /dev/null
+++ b/microkernel/mach/pmap.mdwn
@@ -0,0 +1,74 @@
+[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
+
+[[!tag open_issue_documentation open_issue_gnumach]]
+
+
+# IRC, freenode, #hurd, 2012-02-01
+
+ <sekon> on Hurd what is the difference between kernel memory object and
+ pmap module ??
+ <sekon> pmap is heap/libraries table for each thread while kernel memory
+ object refers to arbitary blobs of data ??
+ <braunr> sekon: pmap is the low level memory mapping module
+ <braunr> i.e. it programs the mmu
+ <braunr> and these aren't hurd-specific, they are mach modules
+ <sekon> braunr: so kernel memonry objects consists of a bunch of pmaps ??
+ <braunr> sekon: memory objects can be various things, be specific please
+ <braunr> (they're certainly not a bunch of pmaps though, no)
+ <braunr> there is one pmap per vm_map, and there is one vm_map per task
+ <braunr> and there is no need for double question marks, is ther ??
+ <sekon> lol then is kernel memory object , please excuse the metaphor
+ something like a base class for pmap
+ <braunr> i don't know what a "kernel memory object" is, be specific please,
+ again
+ <sekon> braunr:
+ http://courses.cs.vt.edu/~cs5204/fall05-gback/presentations/MachOS_Rajesh.ppt
+ <sekon> goto page titled External Memory Management (EMM) on page 15
+ <sekon> Kernel memory object shows up
+ <braunr> you know there are other formats for this document
+ <sekon> nope .. i did not know that
+ <sekon> in page 17 pmamp shows up
+ <braunr> "the problems of external memory management" ?
+ <sekon> braunr: the paper i am also reading is called x15mach_thesis
+ <braunr> ah, that's mine
+ * sekon bows
+ <sekon> :)
+ <braunr> ok i see page 17
+ <sekon> so please good sir explain the relationship between kernel memory
+ object and pmap
+ <sekon> (if any)
+ <sekon> braunr: there is no mention of kernel memory object
+ <braunr> again, i don't see any reference or definition of "kernel memory
+ object"
+ <sekon> but your paper says
+ <sekon> that when page faults occur
+ <sekon> the kernel contact the manager for a kernel reference object
+ <sekon> *memory
+ <braunr> where ?
+ <sekon> in section 2.1.3 (unless i read it wrong)
+ <sekon> no just a sec
+ <sekon> 2.1.5
+ <braunr> i never used the expression "kernel memory object" there :p
+ <braunr> anyway, you're referring simple to memory objects as seen by
+ userspace pagers
+ <braunr> a memory object is a data container
+ <braunr> usually, it's a file
+ <braunr> but it can be anything
+ <braunr> the pager is the task that provides its content and implements the
+ object methods
+ <braunr> as for the relation between them and the pmap module, it's a
+ distant one
+ <braunr> i'll explain it with an example
+ <braunr> page fault -> request content of memory object at a given offset
+ with given length from pager -> ask pmap to establish the mapping in the
+ mmu
+ <sekon> braunr: thank you ver much
+ <sekon> *very
diff --git a/microkernel/viengoos/documentation.mdwn b/microkernel/viengoos/documentation.mdwn
index 52ff7a48..edcc79a7 100644
--- a/microkernel/viengoos/documentation.mdwn
+++ b/microkernel/viengoos/documentation.mdwn
@@ -1,12 +1,12 @@
-[[!meta copyright="Copyright © 2008 Free Software Foundation, Inc."]]
+[[!meta copyright="Copyright © 2008, 2012 Free Software Foundation, Inc."]]
[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
id="license" text="Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License, Version 1.2 or
any later version published by the Free Software Foundation; with no Invariant
Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license
-is included in the section entitled
-[[GNU Free Documentation License|/fdl]]."]]"""]]
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
The most up-to-date documentation is in the source code itself, see in
particular the header files in the hurd directory.
@@ -17,7 +17,8 @@ version of that is available [[here|reference-guide.pdf]]. It is
not, however, automatically regenerated, and thus may not be up to
date.
-Academic Papers:
+
+# Academic Papers
* [Viengoos: A Framework for Stakeholder-Directed Resource
Allocation](http://walfield.org/papers/2009-walfield-viengoos-a-framework-for-stakeholder-directed-resource-allocation.pdf).
@@ -54,3 +55,8 @@ Academic Papers:
argue that only a small static number of scheduling policies are
needed in practice and advocate hierarchical policy specification and
central realization.
+
+
+# Miscellaneous
+
+ * [[IRC_2012-02-23]]
diff --git a/microkernel/viengoos/documentation/irc_2012-02-23.mdwn b/microkernel/viengoos/documentation/irc_2012-02-23.mdwn
new file mode 100644
index 00000000..a3229be9
--- /dev/null
+++ b/microkernel/viengoos/documentation/irc_2012-02-23.mdwn
@@ -0,0 +1,159 @@
+[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
+
+[[!meta title="IRC, freenode, #hurd, 2012-02-23"]]
+
+[[!tag open_issue_documentation open_issue_viengoos]]
+
+ <braunr> neal: i've read a bit about current modern microkernel based
+ systems, and i'm wondering
+ <braunr> neal: can a capability be used for both request and replies, or
+ does messaging need something similar to reply ports ?
+ <neal> braunr: you want a reply port
+ <neal> think about a file server:
+ <neal> the file server publishes a capability to access something
+ <neal> and multiple entities use it
+ <neal> if you wanted just bidirectional caps
+ <braunr> that's the idea i had in mind, i just wondered if it was actually
+ still the case in practice
+ <neal> you'd need to create a new capability every time you delegated the
+ cap
+ <braunr> yes
+ <braunr> thanks
+ <braunr> what about send once rights ?
+ <neal> also, if you send on a cap and then start waiting on it you could
+ get your own reply :)
+ <neal> you can get around send-once rights by using a counter
+ <braunr> no i mean, is their behaviour still needed/useful ?
+ <neal> the counter is kernel implemented
+ <neal> yes
+ <neal> as an optimization
+ <braunr> so they're just a special case of capability
+ <neal> yes
+ <braunr> not a special capability type of their own
+ <neal> but they eliminate the constant create/destroy sequence
+ <braunr> (even if it was already the case at the implementation level in
+ mach, they were named separately which could confuse people)
+ <braunr> hm
+ <braunr> actually, send once rights were used for important notifications
+ such as dead port notifications
+ <braunr> is this still handled at the kernel level in modern ukernels ?
+ <neal> in viengoos, this is called the version field
+ <neal> see chapter 2
+ <neal>
+ http://www.gnu.org/software/hurd/microkernel/viengoos/documentation/reference-guide.pdf
+ <braunr> neal: btw, congratulations for viengoos, it really is a very
+ interesting project: )
+ <neal> thanks
+ <braunr> i don't see the point of rewriting a mach clone after reading
+ about it eh
+ <neal> I would definately do the messenger concept again
+ <neal> but I'd not do persistence
+ <braunr> i don't fully understand how messengers deal with blocking
+ <neal> did you read chapter 4?
+ <braunr> i read all of it but didn't understand everything :)
+ <braunr> it's quite abstract and i didn't make time to read some of the
+ source code
+ <neal> If you have specific questions, I can try to help
+ <braunr> i'll read those chapter again and formulate my questions after
+ <neal> I may have to read them as well :)
+ <braunr> i don't understand how you manage to separate IPC from threading
+ actually
+ <braunr> are messengers queues ?
+ <neal> messengers are super-buffers
+ <neal> they contain a reference to a thread object
+ <neal> to send a message, I use a messenger
+ <neal> I put the data in a buffer
+ <neal> and then I attach the messenger to the target messenger
+ <antrik> braunr: my stance is that we should try to incorporate the ideas
+ from Viengoos into Mach in an evolutionary process...
+ <neal> this causes an activation to be sent to the target messenger's
+ thread object
+ <braunr> neal: which activation ?
+ <neal> an activation is like a CPU interrupt
+ <braunr> neal: is it "allocated" at that moment, or taken from the sending
+ thread ?
+ <braunr> (i'm not sure my question really makes sense to you :/)
+ <antrik> braunr: not sure what you are asking exactly; but the basic idea
+ is that the receiving process preallocates message buffers
+ <braunr> antrik: maybe, i'm not sure
+ <antrik> when someone sends a message, it's stored in one of these buffers,
+ and the process gets a scheduler activation, so it can decide what to do
+ with it
+ <neal> antrik is right
+ <neal> the traget messenger designates a memory buffer
+ <braunr> i'm wondering about the details of this activation
+ <braunr> is it similar to thread migration ?
+ <neal> just before the activation, the data is copied to the messenger's
+ buffer
+ <neal> now someone needs to be notified
+ <neal> (that a message arrived)
+ <neal> that someone is the thread designated in the target messenger's
+ thread field
+ <neal> this is done by an activation
+ <neal> an activation is just an upcall
+ <neal> a thread is forced to a particular IP
+ <neal> an activation isn't a "what" it's a "how"
+ <neal> I never understood thread migration
+ <neal> as it's not really about threads
+ <neal> nor it is about migration
+ <antrik> neal: what happens if another message comes in before the
+ activation handling tread is done with the previous one?...
+ <neal> the messenger is enqueued on the thread object
+ <neal> it is delivered when the thread is in normal mode
+ <neal> part of delivering an activation is putting the thread is activation
+ mode
+ <neal> when in activation mode, it can't receive any activations
+ <braunr> i see
+ <braunr> but then, when a thread receives an activation, does it handle
+ several queued messengers at once (not to loose events/messages) ?
+ <neal> (unless it does a blocking receive on a particular messenger, which
+ is necessary to support memory allocation in activated mode)
+ <neal> it handles one at a time
+ <braunr> ah right
+ <neal> it can't lose events
+ <braunr> activations are sent per messengers/events
+ <neal> well, it can
+ <neal> but it is possible to prevent this
+ <braunr> neal: also, is message passing completely atomic ?
+ <neal> I'm not sure what you mean
+ <neal> which part
+ <braunr> well, all parts of a message :)
+ <braunr> in mach, a message can contain several parts
+ <braunr> data, rights, passing one of them may fail
+ <braunr> only the header is atomically processed
+ <neal> it's not atomic in the sense that a thread can observe the data copy
+ <braunr> that's not what i meant
+ <braunr> is a message completely transferred or not at all in case of
+ failure ?
+ <neal> it may be partially transferred
+ <braunr> or can it be partially transferred
+ <braunr> ok
+ <neal> for instance, if the target thread doesn't provide a memory buffer
+ <neal> then the data can't be copied
+ <neal> I don't recall off hand how I dealt with bad addresses
+ <neal> may be it is not possible
+ <neal> I don't remember
+ <neal> sorry
+ <braunr> but if i read the message structure correctly, there can be one
+ data block, and several capability addresses in a single message, right ?
+ <neal> yes
+ <braunr> ok
+ <braunr> have you considered passing only one object (either data or
+ capability) per message ?
+ <braunr> or is it too inefficient ?
+ <neal> you at least need a reply port
+ <neal> s/port/messenger/
+ <braunr> yes but can't it be passed separately ?
+ <neal> then you have server state
+ <neal> ik
+ <braunr> hm yes
+ <braunr> thanks for your answers: )
+ <neal> no problem
diff --git a/open_issues/boehm_gc.mdwn b/open_issues/boehm_gc.mdwn
index 19bd1b21..e7f849f2 100644
--- a/open_issues/boehm_gc.mdwn
+++ b/open_issues/boehm_gc.mdwn
@@ -1,4 +1,4 @@
-[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]]
+[[!meta copyright="Copyright © 2010, 2012 Free Software Foundation, Inc."]]
[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
id="license" text="Permission is granted to copy, distribute and/or modify this
@@ -281,3 +281,43 @@ It has last been run and compared on 2010-11-10, based on CVS HEAD sources from
Git branches (2010-12-15: last change 2009-09).
* <http://www.hpl.hp.com/personal/Hans_Boehm/gc/#users>
+
+
+## IRC, OFTC, #debian-hurd, 2012-02-05
+
+[[!tag open_issue_porting]]
+
+ <pinotree> youpi: i think i found out the possible cause of the ecl and
+ mono issuess
+ <pinotree> -s
+ <youpi> oh
+ <pinotree> basically, we don't have the realtime signals (so no
+ SIGRTMIN/SIGRTMAX defined), hence things use either SIGUSR1 or
+ SIGUSR2... which are used in libgc to resp. stop/resume threads when
+ "collecting"
+ <pinotree> i just patched ecl to use SIGINFO instead of SIGUSR1 (used when
+ no SIGRTMIN+2 is available), and it seems going on for a while
+ <youpi> uh, why would SIGINFO work better than SIGUSR1?
+ <pinotree> it was a test, i tried the first "not common" signal i saw
+ <pinotree> my test was, use any signal different than USR1/2
+ <youpi> ah, sorry, I hadn't understood
+ <youpi> you mean there's a conflict between ecl and mono using SIGUSR1, as
+ well as libgc?
+ <pinotree> yes
+ <pinotree> for example, in ecl sources see src/c/unixint.d,
+ install_process_interrupt_handler()
+ <youpi> SIGINFO seems a sane choice
+ <youpi> SIGPWR could have been a better choice if it was available :)
+ <pinotree> i would have chose an "unassigned" number, say SIGLOST (the
+ bigger one) + 10, but it would be greater than _NSIG (and thus discarded)
+ <youpi> not a good idea indeed
+ <pinotree> it seems that linux, beside the range for rt signals, has some
+ "free space"
+ <pinotree> i'll start now another ecl build, from scratch this time, with
+ s/SIGUSR1/SIGINFO/ (making sure ctags won't bother), and if it works i'll
+ update svante's bug
+
+ <pinotree> mmap(...PROT_NONE...) failed
+ <pinotree> hmm...
+ <pinotree> apparently enabling MMAP_ANON in mono's libgc copy was a good
+ step, let's see
diff --git a/open_issues/bpf.mdwn b/open_issues/bpf.mdwn
index 98b50430..2a8c897a 100644
--- a/open_issues/bpf.mdwn
+++ b/open_issues/bpf.mdwn
@@ -440,3 +440,125 @@ This is a collection of resources concerning *Berkeley Packet Filter*s.
<braunr> hm, there is a "snoop" source type, using raw sockets
<braunr> too far from the packet source, but i'll try it anyway
<braunr> hm wrong, snoop was the solaris packet filter fyi
+
+
+## IRC, freenode, #hurd, 2012-01-28
+
+ <braunr> nice, i have tcpdump working :)
+ <braunr> let's see if it's as simple with wireshark
+ <pinotree> \o/
+ <braunr> pinotree: it was actually very simple
+ <pinotree> heh, POV ;)
+ <braunr> yep, wireshark works too
+ <braunr> promiscuous mode is harder to test :/
+ <braunr> but that's a start
+
+
+## IRC, freenode, #hurd, 2012-01-30
+
+ <braunr> ok so next step: get tcpreplay working
+ <antrik> braunr: BTW, when you checked the status of the kernel BPF code,
+ did you take zhengda's enhancements/fixes into account?...
+ <braunr> no
+ <braunr> when did i check it ?
+ <antrik> braunr: well, you said the kernel BPF code has serious
+ shortcomings. did you take zhengda's changes into account?
+ <braunr> antrik: ah, when i mention the issues, i considered the userspace
+ translator only
+ <braunr> antrik: and stuff like non blocking io, exporting a selectable
+ file descriptor
+ <braunr> antrik: deb http://ftp.sceen.net/debian-hurd experimental/
+ <braunr> antrik: this is my easy to use repository with a patched
+ libpcap0.8
+ <braunr> and a small and unoptimized pcap-hurd.c module
+ <braunr> it doesn't use devopen yet
+ <braunr> i thought it would be better to have packet filtering working
+ first as a debian patch, then get the new translator+final patch upstream
+ <jkoenig> braunr, tcpdump works great here (awesome!). I'm probably using
+ exactly the same setup and "hardware" as you do, though :-P
+
+
+## IRC, freenode, #hurd, 2012-01-31
+
+ <braunr> antrik: i tend to think we need a bpf translator, or anything
+ between the kernel and libpcap to provide selectable file descriptors
+ <braunr> jkoenig: do you happen to know how mach_msg (as called in a
+ hello.c file without special macros or options) deals with signals ?
+ <braunr> i mean, is it wrapped by the libc in a version that sets errno ?
+ <jkoenig> braunr: no idea.
+ <pinotree> braunr: what's up with it? (not that i have an idea about your
+ actual question, just curious)
+ <braunr> pinotree: i'm improving signal handling in my pcap-hurd module
+ <braunr> i guess checking for MACH_RCV_INTERRUPTED will dio
+ <braunr> -INFO is correctly handled :)
+ <braunr> ok new patch seems fine
+ <antrik> braunr: selectable file descriptors?
+ <braunr> antrik: see pcap_fileno() for example
+ <braunr> it returns a file descriptor matching the underlying object
+ (usually a socket) that can be multiplexed in a select/poll call
+ <braunr> obviously a mach port alone can't do the job
+ <braunr> i've upgraded the libpcap0.8 package with improved signal handling
+ for tests
+ <antrik> braunr: no idea what you are talking about :-(
+
+
+## IRC, freenode, #hurd, 2012-02-01
+
+ <braunr> antrik: you do know about select/poll
+ <braunr> antrik: you know they work with multiple selectable/pollable file
+ descriptors
+ <braunr> on most unix systems, packet capture sources are socket
+ descriptors
+ <braunr> they're selectable/pollable
+ <antrik> braunr: what are packet capture sources?
+ <braunr> antrik: objects that provide applications with packets :)
+ <braunr> antrik: a PF_PACKET socket on Linux for example, or a Mach device,
+ or a BPF file descriptor on BSD
+ <antrik> for a single network device? or all of them?
+ <antrik> AIUI the userspace BPF implementation in libpcap opens this
+ device, waits for packets, and if any arrive, decides depending on the
+ rules whether to pass them to the main program?
+ <braunr> antrik: that's it, but it's not the point
+ <braunr> antrik: the point is that, if programs need to include packet
+ sources in select/poll calls, they need file descriptors
+ <braunr> without a translator, i can't provide that
+ <braunr> so we either decide to stick with the libpcap patch only, and keep
+ this limitation, or we write a translator that enables this feature
+ <pinotree> braunr: are the two options exclusive?
+ <braunr> pinotree: unless we implement a complete bpf translator like i did
+ years ago, we'll need a patch in libpcap
+ <braunr> pinotree: the problem with my early translator implementation is
+ that it's buggy :(
+ <braunr> pinotree: and it's also slower, as packets are small enough to be
+ passed through raw copies
+ <antrik> braunr: I'm not sure what you mean when talking about "programs
+ including packet sources". programs only interact with packet sources
+ through libpcap, right?
+ <antrik> braunr: or are you saying that programs somehow include file
+ descriptors for packet sources (how do they obtain them?) in their main
+ loop, and explicitly pass control to libpcap once something arrives on
+ the respecitive descriptors?
+ <braunr> antrik: that's the idea, yes
+ <antrik> braunr: what is the idea?
+ <braunr> 20:38 < antrik> braunr: or are you saying that programs somehow
+ include file descriptors for packet sources (how do they obtain them?) in
+ their main loop, and explicitly pass control to libpcap once something
+ arrives on the respecitive descriptors?
+ <antrik> braunr: you didn't answer my question though :-)
+ <antrik> braunr: how do programs obtain these FDs?
+ <braunr> antrik: using pcap_fileno() for example
+
+
+## IRC, freenode, #hurd, 2012-02-02
+
+ <antrik> braunr: oh right, you already mentioned that one...
+ <antrik> braunr: so you want some entity that exposes the device as
+ something more POSIXy, so it can be used in standard FS calls, unlike the
+ Mach devices used for pfinet
+ <antrik> this is probably a good sentiment in general... but I'm not in
+ favour of a special solution only for BPF. rather I'd take this as an
+ indication that we probably should expose network interfaces as something
+ file-like in general after all, and adapt pfinet, eth-multiplexer, and
+ DDE accordingly
+ <braunr> antrik: i agree
+ <braunr> antrik: eth-multiplexer would be the right place
diff --git a/open_issues/dde.mdwn b/open_issues/dde.mdwn
index e2cff94f..adb070cd 100644
--- a/open_issues/dde.mdwn
+++ b/open_issues/dde.mdwn
@@ -1,4 +1,5 @@
-[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]]
+[[!meta copyright="Copyright © 2010, 2011, 2012 Free Software Foundation,
+Inc."]]
[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
id="license" text="Permission is granted to copy, distribute and/or modify this
@@ -13,3 +14,350 @@ License|/fdl]]."]]"""]]
[[General Information|/dde]].
Still waiting for interface finalization and proper integration.
+
+[[!toc]]
+
+
+# Upstream Status
+
+
+## IRC, freenode, #hurd, 2012-02-08
+
+At the microkernel davroom at [[community/meetings/FOSDEM_2012]]:
+
+ <antrik> there was quite some talk about DDE. I learnt that there are newer
+ versions in Genode and in Minix (as opposed to the DROPS one we are
+ using)
+ <antrik> but apparently none of the guys involved is interested in creating
+ a proper upstream project with central repository and communication
+ channels :-(
+ <antrik> the original DDE creator was also there, but said he isn't working
+ on it anymore
+ <tschwinge> OK, and the other two projects basically have their own forks.
+ <tschwinge> Or are they actively cooperating?
+ <tschwinge> (If you know about it.)
+ <antrik> well, Genode is also part of the Dresden L4 group; but apart from
+ that, I'd rather call it a fork...
+ <antrik> hm... actually, I'm not sure anymore whether the guy I talked to
+ was from Genode or Nova...
+ <antrik> (both from the Dresdem L4 group)
+
+
+## IRC, freenode, #hurd, 2012-02-19
+
+ <youpi> antrik: do we know exactly which DDE version Zheng Da took as a
+ base ?
+ <youpi> (so as to be able to merge new changes easily)
+ <antrik> youpi: not sure... but from what I gathered at FOSDEM, the version
+ he based on (from DROPS) is not really actively developed right now; if
+ we want to go for newer versions, we probably have to look at other
+ projects (like Genode or Nova or Minix)
+ <youpi> there's no central project for dde ?
+ <youpi> that sucks
+ <antrik> no... and nobody seemed interested in having one :-(
+ <youpi> pff
+ <antrik> which makes me seriously question the original decision to build
+ on DDE...
+ <braunr> ..
+ <antrik> if we have to basically maintain it ourselfs anyways, we could
+ just as well have gone with custom glue
+ <youpi> well, the advantage of DDE is that it already exists now
+ <antrik> on the positive side, one of the projcets (not sure which)
+ apparently have both USB and SATA working with some variant of DDE
+
+
+# IRC, OFTC, #debian-hurd, 2012-02-15
+
+ <pinotree> i have no idea how the dde system works
+ <youpi> gnumach patch to provide access to physical memory and interrupts
+ <youpi> then userland accesses i/o ports by hand to drive things
+ <youpi> but that assumes that no kernel driver is interfering
+ <youpi> so one has to disable kernel drivers
+ <pinotree> how are dde drivers used? can they be loaded on their own
+ automatically, or you have to settrans yourself to setup a device?
+ <youpi> there's no autoloader for now
+ <youpi> we'd need a bus arbitrer that'd do autoprobing
+ <pinotree> i see
+ <pinotree> (you see i'm not really that low level, so pardon the flood of
+ posssibly-noobish questions ;) )
+ <youpi> I haven't set it up yet, but IIRC you need to specify which driver
+ to be used
+ <youpi> well, I mostly have the same questions actually :)
+ <youpi> I just have some guesswork here :)
+ <pinotree> i wonder whether the following could be feasible:
+ <youpi> I'm wondering how we'll manage to make it work in d-i
+ <pinotree> a) you create a package which would b-d on linux-source, build a
+ selection of (network only for now) drivers and install them in
+ /hurd/dde/
+ <youpi> probably through a choice at the boot menu
+ <youpi> I wouldn't dare depending on linux-source
+ <youpi> dde is usually not up-to-date
+ <pinotree> b) add a small utility over the actual fsys_settrans() which
+ would pick the driver from /hurd/dde/
+ <pinotree> ... so you could do `set-dde-driver b43 <device>` (or something
+ like that)
+ <youpi> we can provide something like b) yes
+ <youpi> although documenting the settrans should be fine enough ;)
+ <pinotree> the a) would help/ease with the fact that you need to compile on
+ your own the drivers
+ <pinotree> otherwise we would need to create a new linux-dde-sources-X.Y.Z
+ only with the sources of the drivers we want from linux X.Y.Z
+ <pinotree> (or hurd-dde-linux-X.Y.Z)
+ <CIA-4> samuel.thibault * raccdec3 gnumach/debian/ (changelog
+ patches/70_dde.patch patches/series):
+ <CIA-4> Add DDE experimental support
+ <CIA-4> * debian/patches/70_dde.patch: Add experimental support for irq
+ passing and
+ <CIA-4> physical memory allocation for DDE. Also adds nonetdev boot
+ parameter to
+ <CIA-4> disable network device drivers.
+ <youpi> ok, boots fine with the additional nonetdev option
+ <youpi> now I need to try that dde hurd branch :)
+ <CIA-4> samuel.thibault * rf8b9426 gnumach/debian/patches/70_dde.patch: Add
+ experimental.defs to gnuamch-dev
+
+
+# IRC, freenode, #hurd, 2012-02-19
+
+ * youpi got dde almost working
+ <youpi> it's able to send packets, but apparently not receive them
+ <youpi> (e1000)
+ <youpi> ok, rtl8139 works
+ <youpi> antrik: the wiki instructions are correct
+ <youpi> with e1000 I haven't investigated
+ <antrik> (Archhurd guys also reported problems with e1000 IIRC... the one I
+ built a while back works fine though on my T40p with real e1000 NIC)
+ <antrik> maybe I should try with current versions... something might got
+ broken by later changes :-(
+ <youpi> at least testing could tell the changeset which breaks it
+ <youpi> Mmm, it's very odd
+ <youpi> with the debian package, pfinet's call to device_set_filter returns
+ D_INVALID_OPERATION
+ <youpi> and indeed devnode.c returns that
+ <youpi> ah but it's libmachdev which is supposed to answer here
+ <antrik> youpi: so, regarding the failing device_set_filter... I guess you
+ are using some wrong combination of gnumach and pfinet
+ <youpi> no it's actually that my pfinet was not using bpf
+ <youpi> I've now fixed it
+ <antrik> the DDE drivers rely on zhengda's modified pfinet, which uses
+ devnode, but also switched to using proper BPF filters. so you also need
+ his BPF additions/fixes in gnumach
+ <antrik> OK
+ <youpi> that's the latter
+ <youpi> I had already fixed the devnode part
+ <youpi> but hadn't seen that the filter was different
+ <antrik> err... did I say gnumach? that of course doesn't come into play
+ here
+ <antrik> so yes, you just need a pfinet using BPF
+ <youpi> libmachdev does ;)
+ <antrik> I'm just using pfinet from zhengda's DDE branch... I think devnode
+ and BPF are the only modifications
+ <youpi> there's also a libpcap modification in the incubator
+ <youpi> probably for tcpdump etc.
+ <antrik> libpcap is used by the modified pfinet to compile the filter rule
+ <youpi> why does pfinet need to compile the rule ?
+ <youpi> it's libbpf which is used in the dde driver
+ <antrik> it doesn't strictly need to... but I guess zhengda considered it
+ more elegant to put the source rule in pfinet on compile it live, rather
+ than the compiled blob
+ <antrik> I probably discussed this with him myself a few years back... but
+ my memory on this is rather hazy ;-)
+ <antrik> err... and compile it live
+ <youpi> ah, right, it's only used when asking pfinet to change its filter
+ <youpi> but it does not need it for the default filter
+ <youpi> which is hardcoded
+ <antrik> I see
+ <antrik> when would pfinet change its filter?
+ * youpi now completely converting his hurd box to debian packages with dde
+ support
+ <youpi> on SIOCSIFADDR apparently
+ <youpi> to set "arp or (ip host %s)",
+ <antrik> well, that sounds like the default filter...
+ <youpi> the default filter does not choose an IP
+ <antrik> oh, right... pfinet has to readjust the filter when setting the IP
+ <youpi> arg we lack support for kernel options for gnumach in update-grub
+ <antrik> again, I have a vague recollection of discussing this
+ * youpi crosses fingers
+ <youpi> yay, works
+ <antrik> so we *do* need libpcap in pfinet to set proper rules... though I
+ guess it can also work with a static catchall rule (like it did before
+ zhengda's changes), only less efficient
+ <youpi> well in the past we were already catching everything anyway, so at
+ least it's not a regression :)
+ <antrik> right
+
+
+# IRC, freenode, #hurd, 2012-02-20
+
+ <youpi> I was a bit wary of including the ton of dde headers in hurd-dev
+ <youpi> maybe adding another package for that
+ <youpi> but that would have delayed introducing the dde binaries
+ <youpi> probably we can do that for next upload
+ <pinotree> i can try to work on it, if is feasible (ie if the dde drivers
+ can currently be built from outside the hurd source tree)
+ <youpi> it should be, it's a matter of pointing its makefile to a place
+ where the make scripts and include headers are
+ <youpi> (and the libraries)
+ <pinotree> ok
+ <antrik> youpi: you mean DDEKit headers?
+ <antrik> pinotree: actually it doesn't matter where the dde-ified Linux
+ drivers are built -- libdde_linux26 and the actual drivers use a
+ completetly different build system anyways
+ <antrik> in fact we concluded at some point that they should live in a
+ separate repository -- but that change never happened
+ <antrik> only the base stuff (ddekit, libmachdev etc.) belong in the Hurd
+ source tree
+ <youpi> antrik: yes
+ <youpi> antrik: err, not really completely different
+ <youpi> the actual drivers' Makefile include the libdde_linux26 mk files
+ <youpi> the build itself is separate, though
+ <antrik> youpi: yes, I mean both libdde_linux26 and the drivers use a build
+ system that is completely distinct from the Hurd one
+ <youpi> ah, yes
+ <youpi> libdde_linux26 should however be shipped in the system
+ <antrik> ideally libdde_linux26 and all the drivers should be built in one
+ go I'd say...
+ <youpi> it should be easily feasible to also have a separate driver too
+ <youpi> e.g. to quickly try a 2.6 driver
+ <antrik> youpi: I'm not sure about that. it's not even dynamically linked
+ IIRC?...
+ <youpi> with scripts to build it
+ <youpi> it's not
+ <youpi> but that doesn't mean it can't be separate
+ <youpi> .a files are usually shipped in -dev packages
+ <antrik> youpi: ideally we should try to come with a build system that
+ reuses the original Linux makefile snippets to build all the drivers
+ automatically without any manual per-driver work
+ <youpi> there's usually no modification of the drivers themselves?
+ <youpi> but yeah
+ <youpi> "ideally", when somebody takes the time to do it
+ <antrik> unfortunately, it's necessary to include one particular
+ Hurd/DDE-specific header file in each driver :-(
+ <youpi> can't it be done through gcc's -include option?
+ <antrik> zhengda didn't find a way to avoid this... though I still hope
+ that it must be possible somehow
+ <antrik> I think the problem is that it has to be included *after* the
+ other Linux headers. don't remember the details though
+ <youpi> uh
+ <youpi> well, a good script can add a line after the last occurrence of
+ #include
+ <antrik> yeah... rather hacky, but might work
+ <youpi> even with a bit of grep, tail, cut, and sed it should work :)
+ <antrik> note that this is Hurd-specific; the L4 guys didn't need that
+ <youpi> what is it?
+ <antrik> don't remember off-hand
+
+
+# IRC, freenode, #hurd, 2012-02-22
+
+ <youpi> antrik: AIUI, it should be possible to include all network drivers
+ in just one binary?
+ <youpi> that'd permit to use it in d-i
+ <youpi> and completely replace the mach drivers
+ <youpi> we just need to make sure to include at least what the mach drivers
+ cover
+ <youpi> (all DDE network drivers, I mean)
+ <youpi> of course that doesn't hinder from people to carefully separate
+ drivers in several binaries if they wish
+ <youpi> antrik: it does link at least, I'll give a try later
+ <youpi> yes it works!
+ <youpi> that looks like a plan
+ <youpi> throw all network drivers in a /hurd/dde_net
+ <youpi> settrans it on /dev/dde_net, and settrans devnode on /dev/eth[0-9]
+ <youpi> I'm also uploading a version of hurd which includes headers &
+ libraries, so you just need a make in dde_{e100,e1000,etc,net}
+ <youpi> (uploading it with the dde driver itself :) )
+ <youpi> btw, a nice thing is that we don't really care that all drivers are
+ stuffed into a single binary, since it's a normal process only the useful
+ pages are mapped and actually take memory :)
+ <Tekk_> is that really a nice thing though? compared to other systems I
+ mean
+ <Tekk_> I know on linux it only loads the modules I need, for example. It's
+ definitely a step up for hurd though :D
+ <youpi> that's actually precisely what I mean
+ <youpi> you don't need to load only the modules you need
+ <youpi> you just load them all
+ <youpi> and paging eliminates automatically what's not useful
+ <youpi> even parts of the driver that your device will not need
+ <Tekk_> ooh
+ <Tekk_> awesome
+ <youpi> (actually, it's not even loaded, but the pci tables of the drivers
+ are loaded, then paged out)
+
+
+# IRC, freenode, #hurd, 2012-02-24
+
+ <youpi> antrik_: about the #include <ddekit/timer.h>, I see the issue, it's
+ about jiffies
+ <youpi> it wouldn't be a very good thing to have a jiffies variable which
+ we'd have to update 100times per second
+ <youpi> so ZhengDa preferred to make jiffies a macro which calls a function
+ which reads the mapped time
+ <youpi> however, that break any use of the work "jiffies", e.g. structure
+ members & such
+ <youpi> actually it's not only after headers that the #include has to be
+ done, but after any code what uses the word "jiffies" for something else
+ than the variable
+ <youpi> pb is: it has to be done *before* any code that uses the word
+ "jiffies" for the variable, e.g. inline functions in headers
+ <youpi> in l4dde, there's already the jiffies variable so it's not a
+ problem
+
+
+# IRC, OFTC, #debian-hurd, 2012-02-27
+
+ <tschwinge> I plan to do some light performance testing w.r.t. DDE
+ Ethernet. That is DDE vs. Mach, etc.
+ <youpi> that'd be good, indeed
+ <youpi> I'm getting 4MiB/s with dde
+ <youpi> I don't remember with mach
+ <tschwinge> Yes. It just shouldn't regress too much.
+ <tschwinge> Aha, OK.
+
+
+## IRC, freenode, #hurd, 2012-02-27
+
+ <youpi> tschwinge: nttcp tells me ~80Mbps for mach-rtl8139, ~72Mbps for
+ dde-rtl8139, ~72Mbps for dde-e1000
+ <youpi> civodul: ↑ btw
+ <ArneBab> youpi: so the dde network device is not much slower than the
+ kernel-one?
+ <civodul> youpi: yes, looks good
+ <ArneBab> rather almost the same speed
+ <youpi> apparently
+ <ArneBab> that’s quite a deal.
+ <ArneBab> what speed should it have as maximum?
+ <ArneBab> (means: does the mach version get out all that’s possible?)
+ <ArneBab> differently put: What speed would GNU/Linux get?
+ <youpi> I'm checking that right now
+ <ArneBab> cool!
+ <ArneBab> we need those numbers for the moth after the next
+ <youpi> Mmm, I'm not sure you really want the linux number :)
+ <youpi> 1.6Gbps :)
+ <ArneBab> oh…
+ <youpi> let me check with udp rather than tcp
+ <ArneBab> so the Hurd is a “tiny bit” worse at using the network…
+ <youpi> it might simply be an issue with tcp tuning in pfinet
+ <ArneBab> hm, yes
+ <ArneBab> tcp is not that cheap
+ <ArneBab> and has some pretty advanced stuff for getting to high speeds
+ <youpi> well, I'm not thinking about being cheap
+ <youpi> but using more recent tuning
+ <youpi> that does not believe only 1Mbps network cards exist :)
+ <ArneBab> like adaptive windows and such?
+ <ArneBab> :)
+ <youpi> yes
+ * ArneBab remembers that TCP was invented when the connections went over
+ phone lines - by audio :)
+ <youpi> yep
+ <ArneBab> what’s the system load while doing the test?
+ <youpi> yes, udp seems not so bad
+ <ArneBab> ah, cool!
+ <youpi> it's very variable (300-3000Mbps), but like on linux
+ <ArneBab> that pushing it into user space has so low cost is pretty nice.
+ * ArneBab thinks that that’s a point where Hurd pays off
+ <youpi> that's actually what AST said to fosdem
+ <youpi> he doesn't care about putting an RPC for each and every port i/o
+ <youpi> because hardware is slow anyway
+ <ArneBab> jupp
+ <ArneBab> but it is important to see that in real life
diff --git a/open_issues/default_pager.mdwn b/open_issues/default_pager.mdwn
index 683dd870..9a8e9412 100644
--- a/open_issues/default_pager.mdwn
+++ b/open_issues/default_pager.mdwn
@@ -10,7 +10,10 @@ License|/fdl]]."]]"""]]
[[!tag open_issue_gnumach]]
-IRC, freenode, #hurd, 2011-08-31:
+[[!toc]]
+
+
+# IRC, freenode, #hurd, 2011-08-31
<antrik> braunr: do you have any idea what could cause the paging errors
long before swap is exhausted?
@@ -29,3 +32,6 @@ IRC, freenode, #hurd, 2011-08-31:
<braunr> uvm is too different
<braunr> dragonflybsd maybe, but it's very close to freebsd
<braunr> i didn't look at darwin/xnu
+
+
+# [[trust_the_behavior_of_translators]]
diff --git a/open_issues/glibc_madvise_vs_static_linking.mdwn b/open_issues/glibc_madvise_vs_static_linking.mdwn
index 43b8314f..1f766428 100644
--- a/open_issues/glibc_madvise_vs_static_linking.mdwn
+++ b/open_issues/glibc_madvise_vs_static_linking.mdwn
@@ -26,10 +26,22 @@ case of MADV_DONTNEED), but may influence its performance. The kernel is free
to ignore the advice.* (`man madvise`), so we may simply want to turn it into a
no-op in glibc, avoiding the link-time warning.
-2011-07: This is what Samuel has [done for Debian
-glibc](http://anonscm.debian.org/viewvc/pkg-glibc/glibc-package/trunk/debian/patches/hurd-i386/local-madvise_warn.diff).
-
GCC c5db973fdab3db3e13db575e5650c0bcfd3630f4 (2011-10-17) makes use of this.
As we now export the symbol (and `MADV_DONTNEED`, too), GCC will no longer
`munmap` pages, but will keep them mapped for later re-use. This may increase
memory usage.
+
+2011-07: This is what Samuel has [done for Debian
+glibc](http://anonscm.debian.org/viewvc/pkg-glibc/glibc-package/trunk/debian/patches/hurd-i386/local-madvise_warn.diff).
+
+
+# IRC, freenode, #hurd, 2012-02-16
+
+ <tschwinge> youpi: With Roland's fix the situation will be that just using
+ gcc -static doesn't emit the stub warning, but still will do so in case
+ that the source code itself links in madvise. Is this acceptable for
+ you/Debian/...?
+ <youpi> packages with -Werror will still bug out
+ <youpi> not that I consider -Werror to be a good idea, though :)
+ <tschwinge> youpi: Indeed. Compiler warnings can be caused by all kinds of
+ things. -Werror is good for development, but not for releases.
diff --git a/open_issues/gnumach_memory_management.mdwn b/open_issues/gnumach_memory_management.mdwn
index c34d1200..d29e316c 100644
--- a/open_issues/gnumach_memory_management.mdwn
+++ b/open_issues/gnumach_memory_management.mdwn
@@ -2089,7 +2089,15 @@ There is a [[!FF_project 266]][[!tag bounty]] on this task.
<braunr> ou alors dans les .h ipc
-# IRC, freenode, #hurdfr, 2012-01-18
+# IRC, freenode, #hurd, 2012-01-18
<braunr> does the slab branch need other reviews/reports before being
integrated ?
+
+
+# IRC, freenode, #hurd, 2012-01-30
+
+ <braunr> youpi: do you have some idea about when you want to get the slab
+ branch in master ?
+ <youpi> I was considering as soon as mcsim gets his paper
+ <braunr> right
diff --git a/open_issues/linux_as_the_kernel.mdwn b/open_issues/linux_as_the_kernel.mdwn
new file mode 100644
index 00000000..f14b777e
--- /dev/null
+++ b/open_issues/linux_as_the_kernel.mdwn
@@ -0,0 +1,42 @@
+[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
+
+Instead of attempting a [[history/port_to_another_microkernel]], or writing an
+own one, an implementation of a Hurd system could use another existing
+operating system/kernel, like [[UNIX]], for example, the Linux kernel. This is
+not a [[microkernel]], but that is not an inherent hindrance; depending on what
+the goals are.
+
+There has been an attempt for building a [[Mach_on_top_of_POSIX]].
+
+
+# IRC, freenode, #hurd, 2012-02-08
+
+Richard's X-15 Mach re-implementation:
+
+ <braunr> and in case you didn't notice, it's stalled
+ <braunr> actually i don't intend to work on it for the time being
+ <braunr> i'd rather do as neal suggested: take linux, strip it, and give it
+ a mach interface
+ <braunr> (if your goal really is to get something usable for real world
+ tasks)
+ <antrik> braunr: why would you want to strip down Linux? I think one of the
+ major benefits of building a Linux-Frankenmach would be the ability to
+ use standard Linux functionality alongside Hurd...
+ <braunr> we could have a linux x86_64 based mach replacement in "little"
+ time, with a compatible i386 interface for the hurd
+ <braunr> antrik: well, many of the vfs and network subsystems would be hard
+ to use
+ <antrik> BTW, one of the talks at FOSDEM was about the possibility of using
+ different kernels for Genode, and pariticularily focused on the
+ possibilities with using Linux... unfortunately, I wasn't able to follow
+ the whole talk; but they mentioned similar possibilities to what I'm
+ envisioning here :-)
+
diff --git a/open_issues/memory_object_model_vs_block-level_cache.mdwn b/open_issues/memory_object_model_vs_block-level_cache.mdwn
new file mode 100644
index 00000000..7da5dea4
--- /dev/null
+++ b/open_issues/memory_object_model_vs_block-level_cache.mdwn
@@ -0,0 +1,273 @@
+[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
+
+[[!tag open_issue_documentation open_issue_hurd open_issue_gnumach]]
+
+
+# IRC, freenode, #hurd, 2012-02-14
+
+ <slpz> Open question: what do you think about dropping the memory object
+ model and implementing a simple block-level cache?
+
+[[microkernel/mach/memory_object]].
+
+ <kilobug> slpz: AFAIK the memory object has more purpose than just cache,
+ it's allow used for passing chunk of data between processes, handling
+ swap (which similar to cache, but still slightly different), ...
+ <slpz> kilobug: user processes usually make their way to data with POSIX
+ operations, so memory objects are only needed for mmap'ed files
+ <slpz> kilobug: and swap can be replaced for an in-kernel system or even
+ could still use the memory object
+ <braunr> slpz: memory objects are used for the page cache
+ <kilobug> slpz: translators (especially diskfs based) make heavy use of
+ memory objects, and if "user processes" use POSIX semantics, Hurd process
+ (translators, pagers, ...) shouldn't be bound to POSIX
+ <slpz> braunr: and page cache could be moved to a lower level, near to the
+ devices
+ <braunr> not likely
+ <braunr> well, it could, but then you'd still have the file system overhead
+ <slpz> kilobug: but the use of memory objects it's not compulsory, you can
+ easily write a fs translator without implementing memory objects at all
+ (except to mmap)
+ <braunr> a unified buffer/VM cache as all modern systems have is probably
+ the most efficient approach
+ <slpz> braunr: I agree. I want to look at *BSD/Linux vfs systems to seem
+ how much cache policy depends on the filesystem
+ <slpz> braunr: Are you aware of any good papers on this matter?
+ <braunr> netbsd UVM, the linux virtual memory system
+ <braunr> both a bit old bit still relevant
+ <slpz> braunr: Thanks.
+ <slpz> the problem in our case is that having FS and cache information at
+ different contexts (kernel vs. translator), I find hard to coordinate
+ them.
+ <slpz> that's why I though about a block-level cache that GNU Mach could
+ manage by itself
+ <slpz> I wonder how QNX deals with this
+ <braunr> the point of having a simple page cache is explicitely about not
+ caring if those pages are blocks or files or whatever
+ <braunr> the kernel (at least, mach) normally has all the accounting
+ information it needs to implement its cache policy
+ <braunr> file system translators shouldn't cache much
+ <braunr> the pager interface could be refined, but it looks ok to me as it
+ is
+ <slpz> Mach has the accounting info, but it's not able to purge the cache
+ without coordination with translators
+ <braunr> which is normal
+ <slpz> And this is a big problem when memory pressure increases, as it
+ doesn't know for sure when memory is going to be freed
+ <braunr> Mach flushes its cache when it decides to, and sends back dirty
+ pages if needed by the pager
+ <braunr> that's the case with every paging implementation
+ <braunr> the main difference is security with untrusted pagers
+ <braunr> but that's another issue
+ <slpz> but in a monolithic implementation, the kernel is able for force a
+ chunk of cache memory to be freed without hoping for other process to do
+ the job
+ <braunr> that's not true
+ <braunr> they're not process, they're threads, but the timing issue is the
+ same
+ <braunr> see pdflush on linux
+ <slpz> no, it isn't.
+ <braunr> when memory is scarce, threads that request memory can either wait
+ or immediately fail, and if they wait, they're usually woken by one of
+ the vm threads once flushing is done
+ <slpz> a kernel thread can access all the information in the kernel, and
+ synchronization is pretty easy.
+ <braunr> on mach, synchronization is done with messages, that's even easier
+ than shared kernel locks
+ <slpz> with processes in different spaces, resource coordination becomes
+ really difficult
+ <braunr> and what kind of info would an external pager need when simply
+ asked to take back its dirty pages
+ <braunr> what resources ?
+ <slpz> just take a look at the thread storm problem when GNU Mach needs to
+ clean a bunch of pages
+ <braunr> Mach is big enough to correctly account memory
+ <braunr> there can be thread storms on monolithic systems
+ <braunr> that's a Mach issue, not a microkernel issue
+ <braunr> that's why linux limits the number of pdflush thread instances
+ <slpz> Mach can account memory, but can't assure when be freed by any
+ means, in a lesser degree than a monolithic system
+ <braunr> again i disagree
+ <braunr> no system can guarantee when memory will be freed with paging
+ <slpz> a block level cache can, for most situations
+ <braunr> slpz: why ?
+ <braunr> slpz: or how i mean ?
+ <slpz> braunr: with a block-level page cache, GNU Mach should be able to
+ flush dirty pages directly to the underlaying device without all the
+ complexity and resource cost involved in a m_o_data_return message. It
+ can also throttle the rate at which pages are being cleaned, and do all
+ this while blocking new page allocations to deal with memory exhaustion
+ cases.
+ <slpz> braunr: in the current state, when cleaning dirty pages, GNU Mach
+ sends a bunch on m_o_data_return to the corresponding pagers, hoping they
+ will do their job as soon and as fast as possible.
+ <slpz> memory is not really freed, but transformed from page cache to
+ anonymous memory pertaining to the corresponding translator
+ <slpz> and GNU Mach never knows for sure when this memory is released, if
+ it ever is.
+ <slpz> not being able to flush dirty pages synchronously is a big problem
+ when you need to throttle memory usage
+ <slpz> and needing allocating more memory when you're trying to free (which
+ is the case for the m_o_data_return mechanism) makes the problem even
+ worse
+ <braunr> your idea of a block level cache means in kernel block drivers
+ <braunr> that's not the direction we're taking
+ <braunr> i agree flushing should be a synchronous process, which was one of
+ the proposed improvements in the thread migration papers
+ <braunr> (they didn't achieve it but thought about it for future works, so
+ that the thread at the origin of the fault would handle it itself)
+ <braunr> but it should be possible to have kernel threads similar to
+ pdflush and throttle flush requests too
+ <braunr> again, i really think it's a mach bug, and having a buffer cache
+ would be stepping backward
+ <braunr> the real design issue is allocating memory while trying to free
+ it, yes
+ <slpz> braunr: thread migration doesn't apply to asynchronous IPC, and the
+ entire paging mechanism is implemented this way
+ <slpz> in fact, trying to do a synchronous m_o_data_return will trigger a
+ deadlock for sure
+ <slpz> to achieve synchronous flushing with translators, the entire paging
+ model must be redesigned
+ <slpz> It's true that I'm not very confident of the viability of user space
+ drivers
+ <slpz> at least, not for every device
+ <slpz> I know this is against the current ideas for most ukernel designs,
+ but if we want to achieve real work functionality, I think some
+ sacrifices must be done. Or at least a reasonable compromise.
+ <braunr> slpz: thread migration for paging requests implies synchronous
+ RPC, we don't care much about the IPC layer there
+ <braunr> and it requires large changes of the VM code in addition, yes
+ <braunr> let's not talk about this, we don't have thread migration anyway
+ :p
+ <braunr> except the allocation-on-free-path issue, i really don't see how
+ the current pager interface or the page cache creates problems wrt
+ flushing ..
+ <braunr> monolithic systems also have that problem, with lower impacts
+ though, but still
+ <slpz> braunr: because as it doesn't know when memory is really freed, 1)
+ it just blindly sends a bunch of m_o_data_return to the pagers, usually
+ overloading them (causing thread storms), and 2) it can't properly
+ throttle new page requests to deal with resource exhaustion
+ <braunr> it does know when memory is really freed
+ <braunr> and yes, it blindly sends a bunch of requests, they can and should
+ be trottled
+ <slpz> but dirty pages freed become indistinguishable from common anonymous
+ chunks released, so it doesn't really know if page flushes are really
+ working or not (i.e. doesn't know how fast a device is processing write
+ requests)
+ <braunr> memory is freed when the pager deallocates it
+ <braunr> the speed of the operation is irrelevant
+ <braunr> no system can rely on disk speed to guarantee correct page flushes
+ <braunr> disk or anything else
+ <slpz> requests can't be throttled if Mach doesn't know when they are being
+ processed
+ <braunr> it can easily know it
+ <braunr> they are processed as soon as the request is sent from the kernel
+ <braunr> and processing is done when the pager acknowledges the end of the
+ flush
+ <braunr> memory backing the flushed pages should be released before
+ acknowleding that to avoid starting new requests too soon
+ <slpz> AFAIK pagers doesn't acknowledge the end of the flush
+ <braunr> well that's where the interface should be refined
+ <slpz> Mach just sends the m_o_data_return and continues on its own
+ <braunr> that's why flushing should be synrhconous
+ <braunr> are you sure about that however ?
+ <slpz> so the entire paging system needs a new design... :)
+ <slpz> pretty sure
+ <braunr> not a new design ..
+ <braunr> there is m_o_supply_completed, i don't see how difficult it would
+ be to add m_o_data_return_completed
+ <braunr> it's not a small change, but not a difficult one either
+ <braunr> i'm more worried about the allocation problem
+ <braunr> the default pager should probably be wired in memory
+ <braunr> maybe others
+ <slpz> let's suppose a case in which Mach needs to free memory due to an
+ increase in its pressure. vm_pageout_daemon starts running, clean pages
+ are freed easily, but for each dirty one a m_o_data_return in sent. 1)
+ when should this daemon stop sending m_o_data_return and start waiting
+ for m_o_data_return_completed? 2) what happens if the translator needs to
+ read new blocks to fulfill a write request (pretty common in ext2fs)?
+ <braunr> it should stop after an arbitrary limit is reached
+ <braunr> a reasonable one
+ <braunr> linux limits the number of pdflush threads for that reason as i
+ mentioned (to 8 iirc)
+ <braunr> the problem of reading blocks while flushing is what i'm worried
+ about too, hence the need to wire that code
+ <braunr> well, i'm nto sure it's needed
+ <braunr> again, a reasonable about of free memory should be reserved for
+ that at all times
+ <slpz> but the work for pdflush seems to be a lot easier, as it only deals
+ directly with block devices (if I understood it correctly, I just started
+ looking at it).
+ <braunr> i don't know how other systems compute that, but this is how they
+ seem to do as well
+ <braunr> no, i don't think so
+ <slpz> well, I'll try to invest a few days understanding how pdflush work,
+ to see if some ideas can be borrowed for Hurd
+ <braunr> iirc, freebsd has thresholds in percent for each part of its cache
+ (active, inactive, free, dirty)
+ <slpz> but I still think simple solutions work better, and using the memory
+ object for page cache is tremendously complex.
+ <braunr> the amount of free cache pages is generally sufficient to
+ guarantee much memory can be released at once if needed, without flushing
+ anything
+ <braunr> yes but that's the whole point of the Mach VM
+ <braunr> and its greatest advance ..
+ <slpz> what, memory objects?
+ <braunr> yes
+ <braunr> using physical memory as a cache for anything, not just block
+ buffers
+ <slpz> memory objects work great as a way to provide a shared image of
+ objects between processes, but as page cache they are an overkill (IMHO).
+ <slpz> or, at least, in the way we're using them
+ <braunr> probably
+ <braunr> http://lwn.net/Articles/326552/
+ <braunr> this can help udnerstand the problems we may have without better
+ knowledge of the underlying devices, yes
+ <braunr> (e.g. not being able to send multiple requests to pagers that
+ don't share the same disk)
+ <braunr> slpz: actually i'm not sure it's that overkill
+ <braunr> the linux vm uses struct vm_file to represent memory objects iirc
+ <braunr> there are many links between that structure and some vfs related
+ subsystems
+ <braunr> when a system very actively uses the page cache, the kernel has to
+ maintain a lot of objects to accurately describe the cache content
+ <braunr> you could consider this overkill at first too
+ <braunr> the mach way of doing it just implies some ipc messages instead of
+ function calls, it's not that overkill for me
+ <braunr> the main problems are recursion (allocation while freeing,
+ handling page faults in order to handle flushes, that sort of things)
+ <braunr> struct file and struct address_space actually
+ <braunr> slpz: see struct address_space, it contains a set of function
+ pointers that can help understanding the linux pager interface
+ <braunr> they probably sufferred from similar caveats and worked around
+ them, adjusting that interface on the way
+ <slpz> but their strategy makes them able to treat the relationship between
+ the page cache and the block devices in a really simple way, almost as a
+ traditional storage cache.
+ <slpz> meanwhile on Mach+pager scenario, the relationship between a block
+ in a file and its underlying storage becomes really blurry
+ <slpz> this is a huge advantage when flusing out data, specially when
+ resources are scarce
+ <slpz> I think the idea of using abstract objects for page cache, loses a
+ bit the point that we just want to avoid accessing constantly to a slow
+ device
+ <slpz> and breaking the tight relationship between the device and its
+ cache, makes things a lot harder
+ <slpz> this also manifest itself when flushing clean pages, as things like
+ having an static maximum for cached memory objects
+ <slpz> we shouldn't care about the number of objects, we just need to
+ control the number of pages
+ <slpz> but as we need the pager to flush pages, we need to keep alive a lot
+ of control ports to them
+ <mcsim> slpz: When mo_data_return is called, once the memory manager no
+ longer needs supplied data, it should be deallocated using
+ vm_deallocate. So this way pagers acknowledges the end of flush.
diff --git a/open_issues/select.mdwn b/open_issues/select.mdwn
index 0b69d645..abec304d 100644
--- a/open_issues/select.mdwn
+++ b/open_issues/select.mdwn
@@ -14,9 +14,11 @@ License|/fdl]]."]]"""]]
There are a lot of reports about this issue, but no thorough analysis.
-# `elinks`
+# Short Timeouts
-IRC, unknown channel, unknown date.
+## `elinks`
+
+IRC, unknown channel, unknown date:
<paakku> This is related to ELinks... I've looked at the select()
implementation for the Hurd in glibc and it seems that giving it a short
@@ -31,9 +33,186 @@ IRC, unknown channel, unknown date.
<paakku> Or do I just imagine this problem?
-# dbus
+## [[dbus]]
+
+
+## IRC
+
+### IRC, freenode, #hurd, 2012-01-31
+
+ <braunr> don't you find vim extremely slow lately ?
+ <braunr> (and not because of cpu usage but rather unnecessary sleeps)
+ <jkoenig> yes.
+ <braunr> wasn't there a discussion to add a minimum timeout to mach_msg for
+ select() or something like that during the past months ?
+ <youpi> there was, and it was added
+ <youpi> that could be it
+ <youpi> I don't want to drop it though, some app really need it
+ <braunr> as a debian patch only iirc ?
+ <youpi> yes
+ <braunr> ok
+ <braunr> if i'm right, the proper solution was to fix remote servers
+ instead of client calls
+ <youpi> (no drop, unless the actual bug gets fixed of course)
+ <braunr> so i'm guessing it's just a hack in between
+ <youpi> not only
+ <youpi> with a timeout of zero, mach will just give *no* time for the
+ servers to give an answer
+ <braunr> that's because the timeout is part of the client call
+ <youpi> so the protocol has to be rethought, both server/client side
+ <braunr> a suggested solution was to make it a parameter
+ <braunr> i mean, part of the message
+ <braunr> not a mach_msg parameter
+ <jkoenig> OTOH the servers should probably not be trusted to enforce the
+ timeout.
+ <braunr> why ?
+ <jkoenig> they're not necessarily trusted. (but then again, that's not the
+ only circumstances where that's a problem)
+ <braunr> there is a proposed solution for that too (trust root and self
+ servers only by default)
+ <jkoenig> I'm not sure they're particularily easy to identify in the
+ general case
+ <braunr> "they" ? the solutions you mean ?
+ <braunr> or the servers ?
+ <youpi> jkoenig: you can't trust the servers in general to provide an
+ answer, timeout or not
+ <jkoenig> yes the root/self servers.
+ <braunr> ah
+ <youpi> jkoenig: you can stat the actual node before dereferencing the
+ translator
+ <jkoenig> could they not report FD activity asynchronously to the message
+ port? libc would cache the state
+ <youpi> I don't understand what you mean
+ <youpi> anyway, really making the timeout part of the message is not a
+ problem
+ <braunr> 10:10 < youpi> jkoenig: you can't trust the servers in general to
+ provide an answer, timeout or not
+ <youpi> we already trust everything (e.g. read() ) into providing an answer
+ immediately
+ <braunr> i don't see why
+ <youpi> braunr: put sleep(1) in S_io_read()
+ <youpi> it'll not give you an immediate answer, O_NODELAY being set or not
+ <braunr> well sleep is evil, but let's just say the server thread blocks
+ <braunr> ok
+ <braunr> well fix the server
+ <youpi> so we agree
+ <braunr> ?
+ <youpi> in the current security model, we trust the server into achieve the
+ timeout
+ <braunr> yes
+ <youpi> and jkoenig's remark is more global than just select()
+ <braunr> taht's why we must make sure we're contacting trusted servers by
+ default
+ <youpi> it affects read() too
+ <braunr> sure
+ <youpi> so there's no reason not to fix select()
+ <youpi> that's the important point
+ <braunr> but this doesn't mean we shouldn't pass the timeout to the server
+ and expect it to handle it correctly
+ <youpi> we keep raising issues with things, and not achieve anything, in
+ the Hurd
+ <braunr> if it doesn't, then it's a bug, like in any other kernel type
+ <youpi> I'm not the one to convince :)
+ <braunr> eh, some would say it's one of the goals :)
+ <braunr> who's to be convinced then ?
+ <youpi> jkoenig:
+ <youpi> who raised the issue
+ <braunr> ah
+ <youpi> well, see the irc log :)
+ <jkoenig> not that I'm objecting to any patch, mind you :-)
+ <braunr> i didn't understand it that way
+ <braunr> if you can't trust the servers to act properly, it's similar to
+ not trusting linux fs code
+ <youpi> no, the difference is that servers can be non-root
+ <youpi> while on linux they can't
+ <braunr> again, trust root and self
+ <youpi> non-root fuse mounts are not followed by default
+ <braunr> as with fuse
+ <youpi> that's still to be written
+ <braunr> yes
+ <youpi> and as I said, you can stat the actual node and then dereference
+ the translator afterwards
+ <braunr> but before writing anything, we'd better agree on the solution :)
+ <youpi> which, again, "just" needs to be written
+ <antrik> err... adding a timeout to mach_msg()? that's just wrong
+ <antrik> (unless I completely misunderstood what this discussion was
+ about...)
+
+
+#### IRC, freenode, #hurd, 2012-02-04
+
+ <youpi> this is confirmed: the select hack patch hurts vim performance a
+ lot
+ <youpi> I'll use program_invocation_short_name to make the patch even more
+ ugly
+ <youpi> (of course, we really need to fix select somehow)
+ <pinotree> could it (also) be that vim uses select() somehow "badly"?
+ <youpi> fsvo "badly", possibly, but still
+ <gnu_srs1> Could that the select() stuff be the reason for a ten times
+ slower ethernet too, e.g. scp and apt-get?
+ <pinotree> i didn't find myself neither scp nor apt-get slower, unlike vim
+ <youpi> see strace: scp does not use select
+ <youpi> (I haven't checked apt yet)
+
+
+### IRC, freenode, #hurd, 2012-02-14
+
+ <braunr> on another subject, I'm wondering how to correctly implement
+ select/poll with a timeout on a multiserver system :/
+ <braunr> i guess a timeout of 0 should imply a non blocking round-trip to
+ servers only
+ <braunr> oh good, the timeout is already part of the io_select call
+
+
+### IRC, freenode, #hurdfr, 2012-02-22
+
+ <braunr> le gros souci de notre implé, c'est que le timeout de select est
+ un paramètre client
+ <braunr> un paramètre passé directement à mach_msg
+ <braunr> donc si tu mets un timeout à 0, y a de fortes chances que mach_msg
+ retourne avant même qu'un RPC puisse se faire entièrement (round-trip
+ client-serveur donc)
+ <braunr> et donc quand le timeout est à 0 pour du non bloquant, ben tu
+ bloques pas, mais t'as pas tes évènements ..
+ <abique|work> peut-être que passer le timeout de 10ms à 10 us améliorerait
+ la situation.
+ <abique|work> car 10ms c'est un peut beaucoup :)
+ <braunr> c'est l'interval timer système historique unix
+ <braunr> et mach n'est pas préemptible
+ <braunr> donc c'est pas envisageable en l'état
+ <braunr> ceci dit c'est pas complètement lié
+ <braunr> enfin si, il nous faudrait qqchose de similaire aux high res
+ timers de linux
+ <braunr> enfin soit des timer haute résolution, soit un timer programmable
+ facilement
+ <braunr> actuellement il n'y a que le 8254 qui est programmé, et pour
+ assurer un scheduling à peu près correct, il est programmé une fois, à
+ 10ms, et basta
+ <braunr> donc oui, préciser 1ms ou 1us, ça changera rien à l'interval
+ nécessaire pour déterminer que le timer a expiré
+
+
+### IRC, freenode, #hurd, 2012-02-27
-See [[dbus]].
+ <youpi> braunr: extremely dirty hack
+ <youpi> I don't even want to detail :)
+ <braunr> oh
+ <braunr> does it affect vim only ?
+ <braunr> or all select users ?
+ <youpi> we've mostly seen it with vim
+ <youpi> but possibly fakeroot has some issues too
+ <youpi> it's very little probable that only vim has the issue :)
+ <braunr> i mean, is it that dirty to switch behaviour depending on the
+ calling program ?
+ <youpi> not all select users
+ <braunr> ew :)
+ <youpi> just those which do select({0,0})
+ <braunr> well sure
+ <youpi> braunr: you guessed right :)
+ <braunr> thanks anyway
+ <braunr> it's probably a good thing to do currently
+ <braunr> vim was getting me so mad i was using sshfs lately
+ <youpi> it's better than nothing yes
# See Also
diff --git a/open_issues/trust_the_behavior_of_translators.mdwn b/open_issues/trust_the_behavior_of_translators.mdwn
new file mode 100644
index 00000000..454c638b
--- /dev/null
+++ b/open_issues/trust_the_behavior_of_translators.mdwn
@@ -0,0 +1,181 @@
+[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
+
+[[!tag open_issue_hurd]]
+
+Apart from the issue of [[translators_set_up_by_untrusted_users]], here is
+another problem described.
+
+
+# IRC, freenode, #hurd, 2012-02-17
+
+(Preceded by the [[memory_object_model_vs_block-level_cache]] discussion.)
+
+ <slpz> what should do Mach with a translator that doesn't clean pages in a
+ reasonable amount of time?
+ <slpz> (I'm talking about pages flushed to a pager by the pageout daemon)
+ <braunr> slpz: i don't know what it should do, but currently, it uses the
+ default pager
+
+[[default_pager]].
+
+ <slpz> braunr: I know, but I was thinking about an alternative, for the
+ case in which a translator in not behaving properly
+ <slpz> perhaps freeing the page, removing the map, and letting it die in a
+ segmentation fault could be an option
+ <braunr> slpz: what if the translator can't do it properly because of
+ system resource exhaustion ?
+ <braunr> (i.e. it doesn't have enough cpu time)
+ <slpz> braunr: that's the biggest question
+ <slpz> let's suppose that Mach selects a page, sends it to the pager for
+ cleaning it up, reinjects the page into the active queue, and later it
+ founds the page again and it's still dirty
+ <slpz> but it needs to free some pages because memory it's really, really
+ scarce
+ <slpz> Linux just sits there waiting for I/O completion for that page
+ (trusts its own drivers)
+ <slpz> but we could be dealing with rogue translator...
+ <braunr> yes
+ <braunr> we may need some sort of "authentication" mechanism for pagers
+ <braunr> so that "system pagers" are trusted and others not
+ <braunr> using something like the device master port but for pagers
+ <braunr> a special port passed to trusted pagers only
+ <slpz> hum... that could be used to workaround the untrusted translator
+ crossing problem while walking a directory tree
+
+[[translators_set_up_by_untrusted_users]].
+
+ <slpz> but I think differentiating between trusted and untrusted
+ translators was rejected for philosophical reasons
+ <slpz> (but I'm not sure)
+ <mcsim> slpz: probably there should be something like oom killer?
+ <mcsim> braunr: even if translator is trusted it could have a bug which
+ make it ask more and more memory, so system have something to do with
+ it. Also, this way TCB is increased, so providing port for trusted
+ translators may hurt security.
+ <mcsim> I've read that Genode has "guarded allocators" which help resource
+ accounting by limiting of memory that could be used. Probably something
+ like this could be used in Hurd to limit translators.
+ <antrik> I don't remember how Viengoos deals with this :-(
+
+[[microkernel/Viengoos]].
+
+ <braunr> mcsim: the main feature lacking in mach is resource accounting :p
+
+[[resource_management_problems]].
+
+ <slpz> mcsim: yes, I think there should be a Hurdish oom killer, paying
+ special attention to external pagers
+
+[[microkernel/mach/external_pager_mechanism]].
+
+ <braunr> the oom killer selects untrusted processes by definition (since
+ pagers are in kernel)
+ <mcsim> slpz: and what is better: oom killer or resource accounting?
+ <mcsim> Under resource accounting I mean mechanism when process can't get
+ more resources than it is allowed.
+ <braunr> resource accounting of course
+ <braunr> but it's not just about that
+ <braunr> really, how does the kernel deal when a pager refuses to honor a
+ paging request ?
+ <braunr> whether it is buggy or malicious
+ <braunr> is it really possible to keep all pagers out of the TCB ?
+ <antrik> mcsim: we definitely want proper resource accounting in the long
+ run. the question is how to deal with the situation that resources are
+ reallocated to other tasks, so some pages have to be freed
+ <antrik> I really don't remember how Neal proposed to deal with this
+ <slpz> mcsim: Better: resource accounting (in which resources are accounted
+ to the user tasks which are requesting them, as in the Viengoos
+ model). Good enough an realistic: oom killer
+ <antrik> I'm not sure an OOM killer for non-system pagers is terribly
+ helpful. in typical use, the vast majority of paging is done by trusted
+ pagers...
+ <antrik> and without proper client resource accounting, there are enough
+ other ways a rogue/broken process can eat system resources -- I'm not
+ convinced that untrusted pagers have a major impact on the overall
+ situation
+ <mcsim> If pager can't free resources because of lack, for example, of cpu
+ time it's priority could be increased to give it second chance to free
+ the page. But if it doesn't manage to free resources it could be killed.
+ <antrik> I think the current approach with default pager taking over is
+ good enough for dealing with untrusted pagers. the real problem are even
+ trusted pager frequently failing to deal with the requests
+ <braunr> i agree with antrik
+ <braunr> and i'm opposed to an oom killer
+ <braunr> it's really not a proper fix for our problems
+ <braunr> mcsim: what if needs 3 attempts before succeeding ?
+ <braunr> +it
+ <braunr> and increasing priority without a good reason (e.g. no priority
+ inversion) leads to unfairness
+ <braunr> we don't want to deal with tricky problems like malicious pagers
+ using that to starve other tasks
+ <mcsim> braunr: this is just temporary decision (for example for half of
+ second of user time), to increase probability that task was killed not
+ because of it lacked resources.
+ <braunr> mcsim: tunables should only affect the efficiency of an operation,
+ not its success
+
+
+## IRC, freenode, #hurd, 2012-02-19
+
+ <antrik> neal: the specific question is how to ensure processes free memory
+ fast enough when their allocation becomes lower due to resource pressure
+ <neal> antrik: you can't really.
+ <neal> antrik: the memory manager can act on the application's behalf if
+ the application marks pages as discardable or pagable.
+ <neal> antrik: if the memory manager makes an upcall to the application to
+ free some memory and it doesn't, you have to penalize it.
+ <neal> antrik: You shouldn't the process like exokernel
+ <neal> antrik: It's the developers fault, not the user's
+ <neal> antrik: What you need are controls that ensure that the user stays
+ in control
+ <neal> ...shouldn't *kill* the process...
+ <antrik> neal: well, how can I penalize a process that eats to much
+ physical memory?
+ <neal> in the future, you don't give it as much slack memory
+ <antrik> marking as pagable means a system pager will push them to the swap
+ partition?
+ <antrik> ah, OK
+ <neal> yes
+ <neal> and you page it more aggressively, i.e., you don't give it a chance
+ to free memory
+ <neal> The situation is:
+ <neal> you have memory pressure
+ <neal> you choose a victim process and ask it to free memory
+ <neal> now, you need to wait
+ <neal> if you wait and it doesn't free memory, you give it bad karma
+ <neal> if you wait and it frees memory, you're good
+ <neal> but during that window, a bad process can delay recovery
+ <neal> so, in the future, we give bad processes less time
+ <neal> but, we still send a message asking it to free memory
+ <neal> we just hope it was a bug
+ <antrik> so the major difference to the approach we have in Mach is that
+ instead of just redeclaring some pages as anonymous memory that will be
+ paged to swap by the default pager eventually if the pager in question
+ fails to handle them properly, we wait some time for the process to free
+ (any) memory, and only then start paging out some of it's pages to swap
+ <neal> there's also discardable memory
+ <antrik> hm... there is some notion of "precious" pages in Mach... I wonder
+ whether that is also used to decide about discarding pages instead of
+ pushing them to swap?
+ <neal> antrik: A precious page is ro data that shouldn't be dropped
+ <antrik> ah
+ <antrik> but I guess that means non-precious RO data (such as a cache) can
+ be dropped without asking the pager, right?
+ <neal> yes
+ <antrik> I wonder whether such a karma system can be introduced in Mach as
+ well to deal with problematic pagers
+
+
+## IRC, freenode, #hurd, 2012-02-21
+
+ <neal> antrik: One of the main differences between Mach and Viengoos is
+ that in Mach servers are responsible for managing memory whereas in
+ Viengoos applications are primarily responsible for managing memory.
diff --git a/public_hurd_boxen/xen_handling.mdwn b/public_hurd_boxen/xen_handling.mdwn
index 47d92c43..d4e33ce9 100644
--- a/public_hurd_boxen/xen_handling.mdwn
+++ b/public_hurd_boxen/xen_handling.mdwn
@@ -1,4 +1,4 @@
-[[!meta copyright="Copyright © 2009 Free Software Foundation, Inc."]]
+[[!meta copyright="Copyright © 2009, 2012 Free Software Foundation, Inc."]]
[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
id="license" text="Permission is granted to copy, distribute and/or modify this
@@ -44,6 +44,7 @@ by typing in `host=flubber`, for example, will be enough to get access to
that machine's console.
/!\ TODO: How does one get the environment variables `COLUMNS` and `LINES` set
-properly when using `xm console`? This is relevant for everything using
-`(n)curses` -- for interactive console applications. Using `export COLUMNS=143
-LINES=44` does work, but is a manual process.
+properly when using `xm console`? According to Samuel, *you don't, the xen
+console doesn't have the notion of terminal size*. This is relevant for
+everything using `(n)curses` -- for interactive console applications. Using
+`export COLUMNS=143 LINES=44` does work, but is a manual process.