diff options
23 files changed, 1382 insertions, 18 deletions
diff --git a/faq/binary_compatibility.mdwn b/faq/binary_compatibility.mdwn new file mode 100644 index 00000000..e9dfcdb8 --- /dev/null +++ b/faq/binary_compatibility.mdwn @@ -0,0 +1,33 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation]] + +IRC, freenode, #hurd, 2012-01-13: + + <veganman> sothere's absolutelyno way,evenslowly to run i386 linuxcode + under hurd/i386? Ihave a small app, commercial, which I have to get + running there + <veganman> no source + <braunr> no way + <braunr> you'd need to create a userspace linux server catching linux + system calls and calling hurd specific stuff to implement them + <braunr> it doesn't exist, it may be hard to implement + <braunr> some cases will definitely be hard to implement + <veganman> so, no magic linux lxemu on windows? + <veganman> or linuxemu on plan9 + <pinotree> nope + <veganman> I remember somethingsilly, sonmone hadcompiled linux asauser + applicationon plan9 and inserted his own binaries as + acodeobject,toberunon plan9, for useon ibm hpc hatrdware + <veganman> it was ron minich + <veganman> 5e.iwp9.org/slides/linuxemu.pdf + <veganman> I think that was it + <veganman> google for linux & cnk for additional clues diff --git a/faq/network_transparency.mdwn b/faq/network_transparency.mdwn new file mode 100644 index 00000000..aefaf500 --- /dev/null +++ b/faq/network_transparency.mdwn @@ -0,0 +1,22 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation]] + +IRC, freenode, #hurd, 2012-01-21: + + <chromaticwt> is it possible to transfer servers running on one microkernel + on one machine, to another microkernel running on a different machine? + <chromaticwt> two machines will be running the complete os + <antrik> well, if the code for network-transparent IPC still existed, it + might be possible to move a task to another machine, while keeping the + port associations with the original system... + <antrik> if you mean actually moving it to another system, that's pretty + much impossible in any system that has stateful interfaces diff --git a/hurd/porting/guidelines.mdwn b/hurd/porting/guidelines.mdwn index efc8982a..07807b59 100644 --- a/hurd/porting/guidelines.mdwn +++ b/hurd/porting/guidelines.mdwn @@ -1,5 +1,5 @@ -[[!meta copyright="Copyright © 2002, 2003, 2005, 2007, 2008, 2009, 2010, 2011 -Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2002, 2003, 2005, 2007, 2008, 2009, 2010, 2011, +2012 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -112,6 +112,8 @@ If you get Bad File Descriptor error when trying to read from a file (or accessi ## <a name="PATH_MAX_tt_MAX_PATH_tt_MAXPATHL">`PATH_MAX`, `MAX_PATH`, `MAXPATHLEN`, `_POSIX_PATH_MAX`</a> +<http://pubs.opengroup.org/onlinepubs/009695399/basedefs/limits.h.html> + Every unconditionalized use of `PATH_MAX`, `MAX_PATH` or `MAXPATHLEN` is a POSIX incompatibility. If there is no upper limit on the length of a path (as its the case for GNU), this symbol is not defined in any header file. Instead, you need to either use a different implementation that does not rely on the length of a string or use `sysconf()` to query the length at runtime. If `sysconf()` returns -1, you have to use `realloc()` to allocate the needed memory dynamically. Usually it is thus simpler to just use dynamic allocation. Sometimes the amount is actually known. Else, a geometrically growing loop can be used: for instance, see [Alioth patch](http://alioth.debian.org/tracker/download.php/30628/410472/303735/1575/cpulimit-path-max-fix.patch) or [Pulseaudio patch](http://bugs.debian.org/cgi-bin/bugreport.cgi?msg=5;filename=patch-pulse;att=1;bug=522100). Note that in some cases there are GNU extensions that just work fine: when the `__GLIBC__` macro is defined, `getcwd()` calls can be just replaced by `get_current_dir_name()` calls. Note: constants such as `_POSIX_PATH_MAX` are only the minimum required value diff --git a/hurd/translator/nfs.mdwn b/hurd/translator/nfs.mdwn new file mode 100644 index 00000000..384144dc --- /dev/null +++ b/hurd/translator/nfs.mdwn @@ -0,0 +1,16 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +Translator acting as a NFS client. + + +# See Also + + * [[open_issues/libnfs]] diff --git a/hurd/translator/storeio/discussion.mdwn b/hurd/translator/storeio/discussion.mdwn new file mode 100644 index 00000000..0766e0af --- /dev/null +++ b/hurd/translator/storeio/discussion.mdwn @@ -0,0 +1,16 @@ +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +IRC, freenode, #hurd, 2011-11-10: + + <pinotree> hm, is it normal that st_rdev for storeio translators of + /dev/hd* devices is 0? diff --git a/hurd/translator/tmpfs/discussion.mdwn b/hurd/translator/tmpfs/discussion.mdwn index 486206e3..0409f046 100644 --- a/hurd/translator/tmpfs/discussion.mdwn +++ b/hurd/translator/tmpfs/discussion.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -19,3 +19,267 @@ License|/fdl]]."]]"""]] * [[!GNU_Savannah_bug 26751]] * [[!GNU_Savannah_bug 32755]] + + +# [[Maksym_Planeta]] + +## IRC, freenode, #hurd, 2011-11-29 + + <mcsim> Hello. In seqno_memory_object_data_request I call + memory_object_data_supply and supply one zero filled page, but seems that + kernel ignores this call because this page stays filled in specified + memory object. In what cases kernel may ignore this call? It is written + in documentation that "kernel prohibits the overwriting of live data + pages". But when I called memory_object_lock_request on this page with + should flush and MEMORY_OBJECT_RETURN_ALL nothing change + <braunr> what are you trying to do ? + <mcsim> I think that memory object holds wrong data, so I'm trying to + replace them. This happens when file is truncated, so I should notify + memory object that there is no some data. But since gnumach works only + with sizes that are multiple of vm_page_size, I should manually correct + last page for case when file size isn't multiple of vm_page_size. This is + needed for case when file grows again and that tail of last page, which + wasn't part of file should be filled wit + <mcsim> I've put some printf's in kernel and it seems that page that holds + data which I want replace both absent and busy: + <mcsim> m = vm_page_lookup(object,offset); + <mcsim> ... + <mcsim> if (m->absent && m->busy) { <-- Condition is true + <mcsim> in vm/memory_object.c:169 + <slpz> mcsim: Receiving m_o_data_request means there's no page in the + memory object at that offset, so m_o_data_supply should work + <slpz> are you sure that page is not being installed into the memory + object? + <braunr> it seems normal it's both absent and busy + <braunr> absent because, as sergio said, the page is missing, and busy + because the kernel starts a transfer for its content + <braunr> i don't understand how you determine the kernel ignores your + data_supply + <braunr> "because this page stays filled in specified memory object" + <braunr> please explain this with more detail + <slpz> mcsim: anyway, when truncating a file to a non page-aligned length, + you can just zero fill the rest of the page by mapping the object and + writing to it with memset/bzero + <braunr> (avoid bzero, it's obsolete) + <mcsim> slpz: I'll try try it now. + <braunr> slpz: i think that's what he's trying to do + <mcsim> I don't vm_map it + <braunr> how do you zero it then ? + <braunr> "I call memory_object_data_supply and supply one zero filled page" + <mcsim> First I call mo_lock_request and ask to return this page, than I + memset tail and try to mo_data_supply + <mcsim> I use this function when I try to replace kr = + memory_object_data_supply(reply_to, offset, addr, vm_page_size, FALSE, + VM_PROT_NONE, FALSE, MACH_PORT_NULL); + <mcsim> where addr points to new data, offset points to old data in + object. and reply_to is memory_control which I get as parameter in + mo_data_request + <braunr> why would you want to vm_map it then ? + <mcsim> because mo_data_supply doesn't work. + <braunr> mcsim: i still don't see why you want to vm_map + <mcsim> I just want to try it. + <braunr> but what do you think will happen ? + <mcsim> But seems that it doesn't work too, because I can't vm_map + memory_object from memory_manager of this object. + + +## IRC, freenode, #hurd, 2012-01-05 + + <mcsim> Seems tmpfs works now. The code really needs cleaning, but the main + is that it works. So in nearest future it will be ready for merging to + master branch. BTW, anyone knows good tutorial about refactoring using + git (I have a lot of pointless commits and I want to gather and scatter + them to sensible ones). + <antrik> I wonder whether he actually got the "proper" tmpfs with the + defaul pager working? or only the hack with a private pager? + <mcsim> antrik: with default pager + <antrik> mcsim: wow, that's great :-) + <antrik> how did you fix it? + <mcsim> antrik: The main code I wrote before December, so I forgot some of + what exactly I were doing. So I have to look through my code :) + <mcsim> antrik: old defpager was using old functions like m_o_data_write + instead of m_o_data_return etc. I changed it, mostly because of my + misunderstanding. But I hope that this is not a problem. + + +## IRC, freenode, #hurd, 2012-01-18 + + <antrik> mcsim: did you publish your in-progress work? + <mcsim> there is a branch with working tmpfs in git repository: + http://git.savannah.gnu.org/cgit/hurd/hurd.git/log/?h=mplaneta/tmpfs/defpager + <jd823592> sorry for interrupting the meeting but i wonder what is a + lazyfs? + <mcsim> jd823592: lazyfs is tmpfs which uses own pager + <antrik> mcsim: ah, nice :-) + <antrik> BTW, what exactly did you need to fix to make it work? + <mcsim> most fixes wore in defpager in default_pager_object_set_size. Also, + as i said earlier, I switched to new functions (m_o_data_return instead + of m_o_data_write and so on). I said that this was mostly because of my + misunderstanding, but it turned out that new function provide work with + precious attribute of page. + <mcsim> Also there were some small errors like this: + <mcsim> pager->map = (dp_map_t) kalloc (PAGEMAP_SIZE (new_size)); + <mcsim> memcpy (pager->map, old_mapptr, PAGEMAP_SIZE (old_size)); + <mcsim> where in second line should be new_size too + <mcsim> I removed all warnings in compiling defpager (and this helped to + find an error). + <antrik> great work :-) + <jd823592> tmpfs is nice thing to have :), are there other recent + improvements that were not yet published in previous moth? + <mcsim> BTW, i measured tmpfs speed in it is up to 6 times faster than + ramdisk+ext2fs + <antrik> mcsim: whow, that's quite a difference... didn't expect that + + +## IRC, freenode, #hurd, 2012-01-24 + + <mcsim> braunr: I'm just wondering is there any messages before hurd + breaks. I have quite strange message: memory_object_data_request(0x0, + 0x0, 0xf000, 0x1000, 0x1) failed, 10000003 + <braunr> hm i don't think so + <braunr> usually it either freezes completely, or it panics because of an + exhausted resource + <mcsim> where first and second 0x0 are pager and pager_request for memory + object in vm_fault_page from gnumach/vm_fault.c + <braunr> if you're using the code you're currently working on (which i + assume), then look for a bug there first + <tschwinge> mcsim: Maybe you're running out of swap? + <mcsim> tschwinge: no + <braunr> also, translate the error code + <mcsim> AFAIR that's MACH_INVALID_DEST + <braunr> and what does it mean in this situation ? + <mcsim> I've run fsx as long as possible several times. It runs quite long + but it breaks in different ways. + <mcsim> MACH_SEND_INVALID_DEST + <mcsim> this means that kernel tries to call rpc with pager 0x0 + <mcsim> this is invalid destiantion + <braunr> null port + <braunr> ok + <braunr> did the pager die ? + <mcsim> When I get this message pager dies, but also computer can suddenly + reboot + <braunr> i guess the pager crashing makes mach print this error + <braunr> but then you may have a dead port instead of a null port, i don't + remember the details + <mcsim> braunr: thank you. + <mcsim> btw, for big file sizes fsx breaks on ext2fs + <braunr> could you identify the threshold ? + <braunr> and what's fsx exactly ? + <mcsim> fsx is a testing utility for filesystems + <mcsim> see http://codemonkey.org.uk/projects/fsx/ + <braunr> ah, written by tevanian + <mcsim> threshold seems to be 8Mb + <braunr> fyi, avadis tevanian is the main author of the mach 3 core + services and VM parts + <braunr> well, ext2fs is bugged, we already know that + <braunr> old code maintained as well as possible, but still + <mcsim> hmm, with 6mb it breaks too + <braunr> i guess that it may break on anything larger than a page actually + :p + <mcsim> When I tested with size of 256kb, fsx worked quite long and didn't + break + <braunr> mcsim: without knowing exactly what the test actually does, it's + hard to tell + <mcsim> I see, I just wanted to tell that there are bugs in ext2fs too. But + I didn't debugged it. + <mcsim> fsx performs different operations, like read, write, truncate file, + grow file in random order. + <braunr> in parellel too ? + <braunr> parellel + <braunr> parallel* + <mcsim> no + <mcsim> I run several fsx's parallel on tmpfs, but they break on file with + size 8mb. + <braunr> that must match something in mach + <braunr> s/must/could/ :) + <mcsim> braunr: I've pushed my commits to mplaneta/tmpfs/master branch in + hurd repository, so you could review it. + <braunr> you shouldn't do that just for me :p + <braunr> you should do that regularly, and ask for reviews after + (e.g. during the meetings) + <mcsim> everyone could do that :) + <braunr> i'm quite busy currently unfortunately + <braunr> i'll try when i have time, but the best would be to ask very + specific questions + <braunr> these are usually the faster to answer for people ho have the + necessary expertise to help you + <braunr> fastest* + <mcsim> ok. + <mcsim> braunr: probably, I was doing something wrong, because now parallel + works only for small sizes. Sorry, for disinformation. + + +### IRC, freenode, #hurd, 2012-01-25 + + <antrik> braunr: actually, the paging errors are *precisely* the way my + system tends to die... + <antrik> (it's after about a month of uptime usually though, not a week...) + <antrik> tschwinge: in my case at least, I have still plenty of swap when + this happens. swap usage is generally at about the amount of physical + memory -- I have no idea though whether there is an actual connection, or + it's just coincidence + <braunr> antrik: ok, your hurd dies because of memory issues, my virtual + machines die because of something else (though idk what) + <antrik> before I aquired the habit of running my box 24/7 and thus hitting + this issue, most of the hangs I experienced were also of a different + nature... but very rare in general, except when doing specific + problematic actions + <mcsim> antrik: yes. Do you get messages like that I posted? + <mcsim> here is it: memory_object_data_request(0x0, 0x0, 0xf000, 0x1000, + 0x1) failed, 10000003 + <antrik> mcsim: I can't tell for sure (never noted them down, silly me...) + <antrik> but I definitely get paging errors right before it hangs + <antrik> I guess that was unclear... what I'm trying to say is: I do get + memory_object_data_request() failed; but I'm not sure about the + parameters + <mcsim> antrik: ok. Thank you. + <mcsim> I'll try to find something in defpager, but there should be errors + in mach too. At least because sometimes computer suddenly reboots during + test. + <antrik> mcsim: I don't get sudden reboots + <antrik> might be a different error + <antrik> do you have debugging mode activated in Mach? otherwise it reboots + on kernel panics... + <mcsim> antrik: no. But usually on kernel panics mach waits for some time + showing the error message and only than reboots. + <antrik> OK + <mcsim> how can I know that tmpfs is stable enough? Correcting errors in + kernel to make fsx test work seems to be very complex. + <mcsim> *If errors are in kernel. + <antrik> well, it seems that you tested it already much more thoroughly + than any other code in the Hurd was ever tested :-) + <antrik> of course it would be great if you could pinpoint some of the + problems you see nevertheless :-) + <antrik> but that's not really necessary before declaring tmpfs good enough + I'd say + <mcsim> ok. I'll describe every error I meet on my userpage + <mcsim> but it will take some time, not before weekend. + <antrik> don't worry, it's not urgent + <antrik> the reason I'd really love to see those errors investigated is + that most likely they are the same ones that cause stability problems in + actual use... + <antrik> having an easy method for reproducing them is already a good start + <mcsim> no. they are not the same + <mcsim> every time i get different one + <mcsim> especially when i just start one process fsx and wait error + <antrik> mcsim: have you watched memory stats while running it? if it's + related to the problems I'm experiencing, you will probably see rising + memory use while the test is running + <mcsim> it could be reboot, message, I posted and also fsx could stop + telling that something wrong with data + <antrik> you get all of these also on ext2? + <mcsim> i've done it only once. Here is the log: + http://paste.debian.net/153511/ + <mcsim> I saved "free" output every 30 seconds + <mcsim> no. I'll do it now + <antrik> would be better to log with "vmstat 1" + <mcsim> ok. + <mcsim> as you can see, there is now any leek during work. But near end + free memory suddenly decreases + <antrik> yeah... it's a bit odd, as there is a single large drop, but seems + stable again afterwards... + <antrik> a more detailed log might shed some light + <mcsim> drop at the beginning was when I started translator. + <mcsim> what kind of log do you mean? + <antrik> vmstat 1 I mean + <mcsim> ah... diff --git a/microkernel/mach/gnumach/debugging.mdwn b/microkernel/mach/gnumach/debugging.mdwn index f657e7cc..b57f0393 100644 --- a/microkernel/mach/gnumach/debugging.mdwn +++ b/microkernel/mach/gnumach/debugging.mdwn @@ -1,5 +1,5 @@ -[[!meta copyright="Copyright © 2007, 2008, 2009, 2011 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2007, 2008, 2009, 2011, 2012 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -135,3 +135,11 @@ IRC, freenode, #hurd, 2011-07-14: <braunr> then use info registers to know where the cpu is halted <braunr> and you'll know if you reached that code or not <braunr> (info registers is a qemu command) + + +# Serial Console + +IRC, freenode, #hurd, 2011-11-13: + + <youpi> use console=com0 + <youpi> to activate the console on the first serial port diff --git a/open_issues/anatomy_of_a_hurd_system.mdwn b/open_issues/anatomy_of_a_hurd_system.mdwn index 13599e19..99ef170b 100644 --- a/open_issues/anatomy_of_a_hurd_system.mdwn +++ b/open_issues/anatomy_of_a_hurd_system.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -122,3 +122,147 @@ IRC, OFTC, #debian-hurd, 2011-11-02: system (via libdiskfs). <tschwinge> sekon_: This may help a bit: http://www.gnu.org/software/hurd/hurd/hurd_hacking_guide.html + +--- + +IRC, freenode, #hurd, 2012-01-08: + + <abique> can you tell me how is done in hurd: "ls | grep x" ? + <abique> in bash + <youpi> ls's standard output is a port to the pflocal server, and grep x's + standard input is a port to the pflocal server + <youpi> the connexion between both ports inside the pflocal server being + done by bash when it calls pipe() + <abique> youpi, so STDOUT_FILENO, STDIN_FILENO, STDERR_FILENO still exists + ? + <youpi> sure, hurd is compatible with posix + <abique> so bash 1) creates T1 (ls) and T2 (grep), then create a pipe at + the pflocal server, then connects both ends to T1 and T2, then start(T1), + start(T2) ? + <youpi> not exactly + <youpi> it's like on usual unix, bash creates the pipe before creating the + tasks + <youpi> then forks to create both of them, handling them each side of the + pipe + <abique> ok I see + <youpi> s/handling/handing/ + <abique> but when you do pipe() on linux, it creates a kernel object, this + time it's 2 port on the pflocal ? + <youpi> yes + <abique> how are spawned tasks ? + <abique> with fork() ? + <youpi> yes + <youpi> which is just task_create() and duplicating the ports into the new + task + <abique> ok + <abique> so it's easy to rewrite fork() with a good control of duplicated + fd + <abique> about threading, mutexes, conditions, etc.. are kernel objects or + just userland objects ? + <youpi> just ports + <youpi> (only threads are kernel objects) + <abique> so, about efficiency, are pipes and mutexes efficient ? + <youpi> depends what you call "efficient" + <youpi> it's less efficient than on linux, for sure + <youpi> but enough for a workable system + <abique> maybe hurd is the right place for a userland thread library like + pth or any fiber library + <abique> ? + <youpi> hurd already uses a userland thread library + <youpi> libcthreads + <abique> is it M:N ? + <youpi> libthreads, actually + <youpi> yes + <abique> nice + <abique> is the task scheduler in the kernel ? + <youpi> the kernel thread scheduler, yes, of course + <youpi> there has to be one + <abique> are the posix open()/readdir()/etc... the direct vfs or wraps an + hurd layer libvfs ? + <youpi> they wrap RPCs to the filesystem servers + <antrik> the Bushnell paper is probably the closest we have to a high-level + documentation of these concepts... + <antrik> the Hurd does not have a central VFS component at all. name + lookups are performed directly on the individual FS servers + <antrik> that's probably the most fundamental design feature of the Hurd + <antrik> (all filesystem operations actually, not only lookups) + +IRC, freenode, #hurd, 2012-01-09: + + <braunr> youpi: are you sure cthreads are M:N ? i'm almost sure they're 1:1 + <braunr> and no modern OS is a right place for any thread userspace + library, as they wouldn't have support to run threads on different + processors (unless processors can be handled by userspace servers, but + still, it requires intimate cooperation between the threading library and + the kernel/userspace server in any case + <youpi> braunr: in libthreads, they are M:N + <youpi> you can run threads on different processors by using several kernel + threads, there's no problem in there, a lot of projects do this + <braunr> a pure userspace library can't use kernel threads + <braunr> at least pth was explacitely used on systems like bsd at a time + when they didn't have kernel threads exactly for that reason + <braunr> explicitely* + <braunr> and i'm actually quite surprised to learn that we have an M:N + threading model :/ + <youpi> why do you say "can't" ? + <braunr> but i wanted to reply to abique and he's not around + <youpi> of course you need kernel threads + <youpi> but all you need is to bind them + <braunr> well, what i call a userspace threading library is a library that + completely implement threads without the support of the kernel + <braunr> or only limited support, like signals + <youpi> errr, you can't implement anything with absolutely no support of + the kernel + <braunr> pth used only SIGALRM iirc + <youpi> asking for more kernel threads to use more processors doesn't seem + much + <braunr> it's not + <braunr> but i'm refering to what abique said + <braunr> 01:32 < abique> maybe hurd is the right place for a userland + thread library like pth or any fiber library + <youpi> well, it's indeed more, because the glibc lets external libraries + provide their mutex + <youpi> while on linux, glibc doesn't + <braunr> i believe he meant removing thread support from the kernel :p + <youpi> ah + <braunr> and replying "nice" to an M:N threading model is also suspicious, + since experience seems to show 1:1 models are better + <youpi> "better" ???? + <braunr> yes + <youpi> well + <youpi> I don't have any time to argue about that + <youpi> because that'd be extremely long + <braunr> simpler, so far less bugs, and also less headache concerning posix + conformance + <youpi> but there's no absolute "better" here + <youpi> but less performant + <youpi> less flexible + <braunr> that's why i mention experience :) + <youpi> I mean experience too + <braunr> why less performant ? + <youpi> because you pay kernel transition + <youpi> because you don't know anything about the application threads + <youpi> etc. + <braunr> really ? + <youpi> yes + <braunr> i fail to see where the overhead is + <youpi> I'm not saying m:n is generally better than 1:1 either + <youpi> thread switch, thread creation, etc. + <braunr> creation is slower, i agree, but i'm not sure it's used frequently + enough to really matter + <youpi> it is sometimes used frequently enough + <youpi> and in those cases it would be a headache to avoid it + <braunr> ok + <braunr> i thought thread pools were used in those cases + <youpi> synchronized with kernel mutexes ? + <youpi> that's still slow + <braunr> it reduces to the thread switch overhead + <braunr> which, i agree is slightly slower + <braunr> ok, i's a bit less performant :) + <braunr> well don't futexes exist just for that too ? + <youpi> yes and no + <youpi> in that case they don't help + <youpi> because they do sleep + <youpi> they help only when the threads are living + <braunr> ok + <youpi> now as I said I don't have to talk much more, I have to leave :) diff --git a/open_issues/bpf.mdwn b/open_issues/bpf.mdwn index 73f73093..98b50430 100644 --- a/open_issues/bpf.mdwn +++ b/open_issues/bpf.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2009, 2012 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -70,3 +70,373 @@ This is a collection of resources concerning *Berkeley Packet Filter*s. * [[!GNU_Savannah_patch 6622]] -- pfinet uses the BPF filter * [[!GNU_Savannah_patch 6851]] -- fix a bug in BPF + + +# IRC + +## IRC, freenode, #hurd, 2012-01-13 + + <braunr> hm, i think the bpf code needs a complete redesign :p + <braunr> unless it's actually a true hurdish way to do things + <braunr> antrik: i need your help :) + <braunr> antrik: I need advice on the bpf "architecture" + <braunr> the current implementation uses a translator installed at /dev/bpf + <braunr> which means packets from the kernel are copied to that translator + and then to client applications + <braunr> does that seem ok to you ? + <braunr> couldn't the translator be used to set a direct link between the + kernel and the client app ? + <braunr> which approach seems the more Hurdish to you ? (<= this is what I + need your help on) + <pinotree> braunr: so there would be a roundtrip like kernel → bpf + translator → pfinet? + <antrik> braunr: TBH, I don't see why we need a BPF translator at all... + <braunr> antrik: it handles the ioctls + <braunr> pinotree: pfinet isn't involved (it was merely modified to use the + "new" filter format to specify it used the old packet filter, and not + bpf) + <antrik> braunr: do we really need to emulate the ioctl()s? can't we assume + that all packages using BPF will just use libpcap? + <antrik> (and even if we *do* want to emulate ioctl()s, why can't we handle + this is libc?) + <braunr> antrik: that's what i'm wondering actually + <braunr> even if assuming all packages use libpcap, i'd like our bpf + interface to be close to what bsds have, and most importantly, what + libpcap expects from a bpf interface + <antrik> well, why? if we already have a library handling the abstraction, + I don't see much point in complicating the design and use by adding + another layer :-) + <braunr> so you would advise adapting libpcap to include a hurd specific + module ? + <antrik> there are two reasons for adding translators: more robustness or + more flexibility... so far I don't see how a BPF translator would add + either + <braunr> right + <antrik> yes + <braunr> so we'd end up with a bpf-like interface, the same instructions + and format, with different control calls + <antrik> right + <antrik> note that I had more or less the same desicion to make for KGI + (emulate Linux/BSD ioctl()s, or implement a backend in libggi for + handling Hurd-specific RPC; and after much consideration, I decided on + the latter) + + +## IRC, freenode, #hurd, 2012-01-16 + + <braunr> antrik: is there an existing facility to easily give a send right + to the device master port to a task ? + <braunr> another function of the bpf translator is to handle the /dev/bpf + node, and most importantly its permissions + <braunr> so that users which have read/write access to the node have access + to the packet filter + <braunr> i guess the translator could limit itself to that functionality + <braunr> and then provide a device port on which libpcap operates directly + by means of device_{g,s}et_status/device_set_filter + <antrik> braunr: I don't see the point in seperating permissions for filter + from permissions from general network device access... + <antrik> as for device master port, all root tasks can obtain it from proc + IIRC + <braunr> antrik: yes, but how do we allow non-root users to access that + facility ? + <braunr> on a unix like system, it's a matter of changing the permissions + of /dev/bpf + <antrik> with devnode, non-root users can get access to specific device + nodes, including network devices + <braunr> i can't imagine the hurd being less flexible for that + <braunr> ah devnode + <braunr> good + <antrik> so we can for example make /dev/eth0 accessible by users of some + group + <braunr> what's devnode exactly ? + <antrik> it's a very simple translator that implements an FS node that + looks somewhat like a file, but the only operation it supports is + obtaining a pseudo device master port, giving access to a specific Mach + device + <braunr> is it already part of the hurd ? + <braunr> or hurdextras maybe ? + <antrik> it's only in zhengda's branch + <braunr> ah + <antrik> needed for both eth-multipexer and DDE + <braunr> and bpf soon i guess + <antrik> indeed :-) + <braunr> "obtaining a pseudo device master port", i believe you meant a + pseudo device port + <antrik> I must admit that I don't remember exactly whether devnode proxies + device_open(), so clients direct get a port to the device in question, or + whether it implements a pseudo device master port... + <antrik> but definitely not a pseudo device port :-) + <braunr> i'm almost positive it gives the target device port, otherwise i + don't see the point + <braunr> i don't understand the user of the "pseudo" word here either + <braunr> s/user/use/ + <braunr> aiui, devnode should be started as root (or in any way which gives + it the device master port) + <antrik> the point is that the client doesn't need to know the Mach device + name, and also is not bound to actual kernel devices + <braunr> and when started, implement the required permissions before giving + clients a device port to the specific device it was installed for + <braunr> right + <braunr> but it mustn't be a proxy + <antrik> yes, devnode needs access to either the real master device port + (for kernel devices), or one provided by eth-multiplexer or the DDE + network driver + <braunr> well, a very simple proxy for deviceopen + <braunr> ok + <braunr> that seems exactly what i wanted to do + <braunr> we now need to see if we can integrate it separately + <braunr> create a separate branch that works for the current gnumach code, + and merge dde/other specific code later on + <antrik> you mean independent of eth-multiplexer or DDE? yes, it was + generally agreed that devnode is a good idea in any case. I have no idea + why there are no device nodes for network devices on other UNIX + systems... + <braunr> i've been wondering that for years too :) + <antrik> zhengda's branch has a pfinet modified to a) use devnode, and b) + use BPF + <braunr> why bpf ? + <braunr> for more specific filters maybe ? + <antrik> hm... don't remember whether there was any technical reason for + going with BPF; I guess it just seemed more reasonable to invest new work + in BPF rather than obsolete Mach-specific NPF... + <braunr> cspf could be removed altogether, i agree + <antrik> another plus side of his modified pfinet is that it actually sets + an appropriate filter for TCP/IP and the IP in use, rather than just + setting a dummy filter catching app packets (including those irrelevant + to the specific pfinet instance) + <antrik> err... catching all packets + <braunr> that's what i meant by "for more specific filters maybe ?" + <braunr> he was probably more comfortable with the bpf interface to write + his filter rules + <antrik> well, it would probably be doable with NPF too :-) so by itself + it's not a reason for switching to BPF... + <antrik> it's rather the other way around: as it was necessary to implement + filters in eth-multiplexer, and implementing BPF seemed more reasoable, + pfinet had to be changed to use BPF... + <braunr> antrik: where is zhengda's branch btw ? + <antrik> (I guess using proper filters with eth-multiplexer is not strictly + necessary; but it would be a major performance hit not to) + <antrik> it's in incubator.git + <antrik> but it's very messy + <braunr> ok + <antrik> at some point I asked him to provide cleaned up branches, and I'm + pretty sure he said he did, but I totally fail to remember where he + published them :-( + <braunr> hm, i don't like how devnode is "architectured" :/ + <braunr> but it makes things a little more easy to get working i guess + <LarstiQ> antrik: any idea what to grep the logs on for that? + <braunr> ok never mind, devnode is fine + <braunr> exactly what i need + <braunr> i wonder however if it shouldn't be improved to better handle + permissions + <braunr> ok, never mind either, permission handling is fine + <braunr> so what are we waiting for ? :) + <antrik> I remember that there were some issues with permission handling, + but I don't remember whether all were fixed :-( + <antrik> LarstiQ: hm... good question... + <braunr> ah ? + <braunr> hm actually, there could be issues for packet filters, yes + <braunr> i guess we want to allow e.g. read-only opens for capture only + <antrik> braunr: that would have to be handled by the actual BPF + implementation I'd say + <braunr> it should already be the case + <antrik> what's the problem then? + <braunr> but when the actual device_open() is performed, the appropriate + permissions must be provided + <braunr> and checking those is the responsibility of the proxy, devnode in + this case + <antrik> and it doesn't do that? + <braunr> apparently not + <braunr> the only check is against the device name + <braunr> i'll begin playing with that first + <antrik> I vaguely remember that there has been discussion about the + relation of underlying device open mode and devnode open mode... but I + don't remember the outcome. in fact it was probably one of the + discussions I never got around to follow up on... :-( + <antrik> before you begin playing, take a look at the relevant messages in + the ML archive :-) + <antrik> must have been around two years ago + <braunr> ok + <antrik> some thread with me and scolobb (Sergiu Ivanov +- spelling) and + probably zhengda + <antrik> there might also be some outstanding patch(es) from scolobb, not + sure + + +## IRC, freenode, #hurd, 2012-01-17 + + <braunr> antrik: i think i found the thread you mentioned about devnode + <braunr> neither sergiu nor zhengda considered the use of a read-only + device for packet filtering + <braunr> leading to assumptions such as "only receiving packets + <braunr> is not terribly useful, in view of the fact that you have to at + least + <braunr> request them, which implies *sending* packets :-) + <braunr> " + <braunr> IMO, devnode should definitely check its node permissions to build + the device open flags + <braunr> good news is that it doesn't depend on anything specific to other + incubator projects + <braunr> making it almost readily mergeable in the hurd + <braunr> i'm not sure devnode is an appropriate name though + <braunr> maybe something like device, or devproxy + <braunr> proxy-devopen maybe + <antrik> braunr: well, I don't remember the details of the disucssion; but + as I mentioned in some mail, I did actually want to write a followup, + just didn't get around to it... so I was definitely not in agreement with + some of the statements made by others. I just don't remember on which + point :-) + <antrik> which thread was it? + <antrik> anyways, this should in no way be specific to network + devices... the idea is simply that if the client has only read + permissions on the device node, it should only get to open the underlying + device for read. it's up to the kernel to handle the read-only status for + the device once it's opened + <antrik> as for the naming, the idea is that devnode simply makes Mach + devices accessible through FS nodes... so the name seemed appropriate + <antrik> you may be right though that just "device" might be more + straightforward... I don't agree on the other variants + <braunr> antrik: + http://lists.gnu.org/archive/html/bug-hurd/2009-12/msg00155.html + <braunr> antrik: i agree with the general idea behind permission handling, + i was just referring to their thoughts about it, which probably led to + the hard coded READ | WRITE flags + <antrik> braunr: unfortunately, I don't remember the context of the + discussion... would take me a while to get into this again :-( + <antrik> the discussion seems to be about eth-multiplexer as much as about + devnode (if not more), and I don't remember the exact interaction + + +## IRC, freenode, #hurd, 2012-01-18 + + <braunr> so, does anyone have an objection to getting devnode into the hurd + and calling it something else like e.g. device ? + <youpi> braunr: it's Zhengda's work, right? + <braunr> yes + <youpi> I'm completely for it, it just perhaps needs some cleanup + <braunr> i have a few changes to add to what already exists + <braunr> ok + <braunr> well i'm assigning myself to the task + <antrik> braunr: I'm still not convinced just "device" is preferable + <antrik> perhaps machdevice ;-) + <antrik> but otherwise, I'd LOVE to see it in :-) + <braunr> i don't know .. what if the device is actually eth-multiplexer or + a dde one ? + <braunr> it's not really "mach", is it ? + <braunr> or do we only refer to the interface ? + <youpi> that translator is only for mach devices + <braunr> so you consider dde devices as being mach devices too ? + <braunr> it's a simple proxy for device_open really + <youpi> will these devices use that translator? + <youpi> ah + <youpi> I thought it was using a mach-specific RPC + <braunr> so we can consider whatever we want + <antrik> braunr: yes, the translator is for Mach device interface only. it + might be provided by other servers, but it's still Mach devices + <youpi> then drop the mach, yes + <braunr> i'd tend to agree with antrik + <youpi> antrik: I'd say the device interface is part of the hur dinterfaces + <braunr> then machdev :p + <braunr> no, it's really part of the mach interface + <youpi> it's part of the mach interface, yes + <youpi> but also of the Hurd, no? + <antrik> DDE network servers also use the Mach device interface + <braunr> no + <youpi> can't we say it's part of it? + <youpi> I mean + <youpi> even if we change the kernel + <braunr> dde is the only thing that implements it besides the kernel that i + know of + <youpi> we will probably want to keep the same interface + <braunr> yes but that's a mach thing + <youpi> what we have now is not necessarily a reason + <antrik> as for other DDE drivers, I for my part believe they should export + proper Hurd (UNIX) device nodes directly... but for some reason zhengda + insisted on implementing it as Mach devices too :-( + <braunr> antrik: i agree with you on that too + <braunr> i was a bit surprised to see the same interface was reused + <braunr> youpi: we can, we just have to agree on what we'll do + <braunr> what do you mean by "even if we change the kernel" ? + <antrik> the problem with "machdev" is that it might suggest the translator + actually implements the device... not sure whether this would cause + serious confusion + <antrik> "devopen" might be another option + <antrik> or "machdevopen" to be entirely verbose ;-) + <braunr> an option i suggested earlier which you disagreed on :p + <braunr> but devopen is the one i'd choose + <antrik> youpi: as I already mentioned in the libburn thread, I don't + actually think the Mach device interface is very nice; IMHO we should get + rid of it as soon as we can, rather than port it to other + architectures... + <antrik> but even *if* we decided to reuse it after all, it would still be + the Mach device interface :-) + <braunr> actually, zheng da already suggested that name a long time ago + <braunr> http://lists.gnu.org/archive/html/bug-hurd/2008-08/msg00005.html + <braunr> no actually antrik did eh + <braunr> ok let's use devopen + <antrik> braunr: you suggested proxy-devopen, which I didn't like because + of the "proxy" part :-) + <braunr> not only, but i don't have the logs any more :p + <antrik> oh, I already suggested devopen once? didn't expect myself to be + that consistent... ;-) + <antrik> braunr: you suggested device, devproxy or proxy-devopen + <braunr> ah, ok + <braunr> devopen is better + <antrik> I wonder whether it's more important for clarity to have "mach" in + there or "open"... or whether it's really too unweildy to have both + + +## IRC, freenode, #hurd, 2012-01-21 + + <braunr> oh btw, i made devopen run today, it shouldn't be hard getting it + in properly + <braunr> patching libpcap will be somewhat trickier + <braunr> i don't even really need it, but it allows having user access to + mach devices, which is nice for the libpcap patch and tcpdump tests + <braunr> permission checking is actually its only purpose + <braunr> well, no, not really, it also allows opening devices implemented + by user space servers transparently + + +## IRC, freenode, #hurd, 2012-01-27 + + <braunr> hmm, bpf needs more work :( + <braunr> or we can use the userspace bpf filter in libpcap, so that it + works with both gnumach and dde drivers + <antrik> braunr: there is a userspace BPF implementation in libpcap? I'm + surprised that zhengda didn't notice it, and ported the one from gnumach + instead... + <antrik> what is missing in the kernel implementation? + <braunr> antrik: filling the bpf header + <braunr> frankly, i'm not sure we want to bother with the kernel + implementation + <braunr> i'd like it to work with both gnumach and dde drivers + <braunr> and in the long run, we'll be using userspace drivers anyway + <braunr> the bpf header was one of the things the defunct translator did + <braunr> which involved ugly memcpy()s :p + <antrik> braunr: well, if you want to get rid of the kernel implementation, + basically you would have to take up eth-multiplexer and get it into + mainline + <antrik> (and make sure it's used by default in Debian) + <antrik> I frankly believe it's the better design anyways... but quite a + major change :-) + <braunr> not that major to me + <braunr> in the meantime i'll use the libpcap embedded implementation + <braunr> we'll have something useful faster, with minimum work when + eth-multiplexer is available + <antrik> eth-multiplexer is ready for use, it just needs to go upstream + <antrik> though it's probably desirable to switch it to the BPF + implementation from libpcap + <braunr> using the libpcap implementation in libpcap and in eth-multiplexer + are two different things + <braunr> the latter is preferrable + <braunr> (and yes, by available, i meant upstream ofc) + <antrik> eth-mulitplexer is already using libpcap anyways (for compiling + the filters); I'm sure zhengda just didn't realize it has an actual BPF + implementation too... + <braunr> we want the filter implementation as close to the packet source as + possible + <antrik> I have been using eth-multiplexer for at least two years now + <braunr> hm, there is a "snoop" source type, using raw sockets + <braunr> too far from the packet source, but i'll try it anyway + <braunr> hm wrong, snoop was the solaris packet filter fyi diff --git a/open_issues/code_analysis/discussion.mdwn b/open_issues/code_analysis/discussion.mdwn new file mode 100644 index 00000000..f8a0657d --- /dev/null +++ b/open_issues/code_analysis/discussion.mdwn @@ -0,0 +1,44 @@ +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation]] + +[[!toc]] + + +# IRC, freenode, #hurd, 2011-12-04 + + <mcsim> has anyone used splice on hurd? + <mcsim> splice -> splint + <youpi> not that I know of + <mcsim> this is tool for statically checking C programs + <mcsim> seems I made it work + <braunr> hm i realli i personnally dislike such tools a lot, but sometimes + it might help + <braunr> hello hurd people + <mcsim> braunr: hello + <braunr> mcsim: duma may be helpful as replacement for the memcheck part of + valgrind + <mcsim> defpager uses it's own dynamic memory allocator, which uses + vm_allocate/vm_deallocate as backing store? Am I able to use duma in such + case? + <braunr> you will have to adapt it + <braunr> but it's already designed to handle custom allocators + <braunr> iirc + <braunr> btw, are there special flags for that memory which the pager + allocates ? + <braunr> e.g. to use wired memory ? + <mcsim> yes, wired memory + <braunr> you'll have to change that in duma then + <braunr> but apart from such details, it should be straightforward + <antrik> braunr: I have no idea about duma; but if you think it's a useful + tool, please add it to open_issues/code_analysis.mdwn + <antrik> (I guess we should have a "proper" page listing useful debugging + tools...) diff --git a/open_issues/dbus.mdwn b/open_issues/dbus.mdwn new file mode 100644 index 00000000..ec39e063 --- /dev/null +++ b/open_issues/dbus.mdwn @@ -0,0 +1,82 @@ +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +IRC, freenode, #hurd, 2011-11-26: + + <antrik> BTW, how much effort is necessary to fix dbus? + <pinotree> basically, have pflocal know who's the sender + (pid/uid/gid/groups) in the socket send op + +IRC, freenode, #hurd, 2011-12-16: + + <braunr> pinotree: what's the problem with dbus ? + <pinotree> braunr: select() returning 0 changed fd's with very short (eg < + 1ms) timeouts when there are actually events; + +[[select]]. + + <pinotree> and missing socket credentials + +[[sendmsg_scm_creds]]. + + <braunr> oh + <braunr> which socket creds interface ? + <pinotree> bsd, i.e. with SCM_CREDENTIALS payload for cmsg on + {recv,send}msg() + <braunr> ok + <braunr> SCM_RIGHTS too ? + <braunr> the select issue seems weird though + <pinotree> hm no, that's for passing fd's to other processes + <braunr> is it specific to pflocal or does dbus use pfinet too ? + <pinotree> iirc on very short timeouts the application has no time waiting + for the reply of servers + <braunr> i see + <pinotree> braunr: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=79358 + <braunr> thanks + <pinotree> (the interesting messages are from #53 and on) + <braunr> 2000 eh ... :) + <braunr> hm i agree with neal, i don't understand why the timeout is given + to the kernel as part of the mach_msg call + +IRC, freenode, #hurd, 2011-12-20: + + <braunr> hm, i don't see any occurrence of SCM_CREDENTIALS in dbus + <braunr> only SCM_RIGHTS + <pinotree> braunr: yes, that one + <braunr> oh + <braunr> i thought you said the opposite last time + <pinotree> dbus/dbus-sysdeps-unix.c, write_credentials_byte and + _dbus_read_credentials_socket (with #define HAVE_CMSGCRED) + <braunr> hm + <braunr> which version ? + <braunr> i don't see anything in 1.4.16 + <pinotree> 1.4.16 + <braunr> grmbl + <braunr> ah, i see + <braunr> SCM_CREDS + <pinotree> if you want, i have a simplier .c source with it + <braunr> no i'm just gathering info + <pinotree> ok + <braunr> what's the deal with SCM_CREDS and SCM_CREDENTIALS ? + <braunr> bsd vs sysv ? + <braunr> oh, http://lists.debian.org/debian-hurd/2002/03/msg00135.html + <braunr> so we actually do want both SCM_CREDS and SCM_RIGHTS for debus + <braunr> dbus + <pinotree> SCM_RIGHTS is a different matter, it is for passing fd's + <braunr> yes + <braunr> but it's used by dbus + <braunr> so if we can get it, it should help too + <pinotree> there's a preliminary patch for it done by emilio time ago, and + iirc it's applied to debian's glibc + <braunr> ah, he changed the libc + <braunr> right, that's the only sane way + <pinotree> iirc roland didn't like one or more parts of it (but i could be + wrong) + <braunr> ok diff --git a/open_issues/default_pager.mdwn b/open_issues/default_pager.mdwn index 18670c75..683dd870 100644 --- a/open_issues/default_pager.mdwn +++ b/open_issues/default_pager.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -19,7 +19,7 @@ IRC, freenode, #hurd, 2011-08-31: <antrik> (and also I/O performance steadily dropping before that point is reached?) -[[performance/degradation]] (?). +[[performance/degradation]], [[ext2fs_page_cache_swapping_leak]]. <antrik> hm <braunr> there could too many things diff --git a/open_issues/ext2fs_page_cache_swapping_leak.mdwn b/open_issues/ext2fs_page_cache_swapping_leak.mdwn index 075533e7..7c4cf52d 100644 --- a/open_issues/ext2fs_page_cache_swapping_leak.mdwn +++ b/open_issues/ext2fs_page_cache_swapping_leak.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -8,7 +8,7 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -[[!tag open_issue_hurd]] +[[!tag open_issue_gnumach open_issue_hurd]] There is a [[!FF_project 272]][[!tag bounty]] on this task. @@ -27,6 +27,7 @@ There is a [[!FF_project 272]][[!tag bounty]] on this task. <youpi> yes <youpi> the disk content, basicallyt :) + # IRC, freenode, #hurd, 2011-04-18 <antrik> damn, a cp -a simply gobbles down swap space... @@ -257,3 +258,107 @@ There is a [[!FF_project 272]][[!tag bounty]] on this task. <antrik> the only way to free the memory seems to be terminating the FS server <youpi> uh :/ + + +# IRC, freenode, #hurd, 2011-11-30 + + <antrik> slpz: basically, whenever free memory goes below the paging + threshold (which seems to be around 6 MiB) while there is other I/O + happening, swap usage begins to increase continuously; and only gets + freed again when the filesystem translator in question exits + <antrik> so it sounds *very* much like pages go to swap because the + filesystem isn't quick enough to properly page them out + <antrik> slpz: I think it was you who talked about double paging a while + back? + <slpz> antrik: probably, sounds like me :-) + <antrik> slpz: I have some indication that the degenerating performance and + ultimate hang issues I'm seeing are partially or entirely caused by + double paging... + <antrik> slpz: I don't remember, did you propose some possible fix? + <slpz> antrik: hmm... perhaps it wasn't me, because I don't remember trying + to fix that problem... + <slpz> antrik: at which point do you think pages get duplicated? + <antrik> slpz: it was a question. I don't remember whether you proposed + something or not :-) + <antrik> slpz: basically, whenever free memory goes below the paging + threshold (which seems to be around 6 MiB) while there is other I/O + happening, swap usage begins to increase continuously; and only gets + freed again when the filesystem translator in question exits + <antrik> so it sounds *very* much like pages go to swap because the + filesystem isn't quick enough to properly page them out + <tschwinge> + http://www.bddebian.com:8888/~hurd-web/open_issues/ext2fs_page_cache_swapping_leak/ + <slpz> tschwinge: thanks + <slpz> antrik: I see + <tschwinge> Always at your service. ;-) + <slpz> antrik: I didn't addressed this problem directly, but when I've + modified the pageout mechanism to provide a special treatment for + external pages, I also removed the possibility of sending them to the + default pager + <slpz> antrik: this was in my experimental environment, of course + <antrik> slpz: oh, nice... so it may fix the issues I'm seeing? :-) + <antrik> anything testable yet? + <slpz> antrik: yes, only anonymous memory could be swapped with that + <slpz> antrik: it works, but is ugly as hell + <antrik> tschwinge: these is also your observation about compilations + getting slower on further runs, and my followups... I *suspect* it's the + same issue + +[[performance/degradation]]. + + <slpz> antrik: I'm thinking about establishing a repository for these + experimental versions, so they don't get lost with the time + <antrik> slpz: please do :-) + <slpz> antrik: perhaps in savannah's HARD project + <antrik> even if it's not ready for upstream, it would be nice if I could + test it -- right now it's bothering me more than any other Hurd issues I + think... + <slpz> also, there's another problem which causes performance degradation + with the simple use of the system + <tschwinge> slpz: Please just push to Savannah Hurd. Under your + slpz/... or similar. + <tschwinge> antrik: Might very well be, yes. + <slpz> and I almost sure it is the fragmentation of the task map + <slpz> tschwinge: ok + <slpz> after playing a bit with a translator, it can easily get more than + 3000 entries in its map + <antrik> slpz: yeah, other issues might play a role here as well. I + observed that terminating the problematic FS servers does free most of + the memory and remove most of the performance degradation, but in some + cases it's still very slow + <slpz> that makes vm_map_lookup a lot slower + <antrik> on a related note: any idea what can cause paging errors and a + system hang even when there is plenty of free swap? + <antrik> (I'm not entirely sure, but my impression is that it *might* be + related to the swap usage and performance degradation problems) + <slpz> I think this degree of fragmentation has something to do with the + reiterative mapping of memory objects which is done in pager-memcpy.c + <slpz> antrik: which kind of paging errors? + <antrik> hm... I don't think I ever noted down the exact message; but I + think it's the same you get when actually running out of swap + <slpz> antrik: that could be the default pager dying for some internal bug + <antrik> well, but it *seems* to go along with the performance degradation + and/or swap usage + <slpz> I also have the impression that we're using memory objects the wrong + way + <antrik> basically, once I get to a certain level of swap use and slowness + (after about a month of use), the system eventually dies + <slpz> antrik: I never had a system running for that time, so it could be a + completely different problem from what I've seen before :-/ + <slpz> Anybody has experience with block-level caches on microkernel + environments? + <antrik> slpz: yeah, it typically happens after about a month of my normal + use... but I can significantly accellerate it by putting some problematic + load on it, such as large apt-get runs... + <slpz> I wonder if it would be better to put them in kernel or in user + space. And in the latter, if it would be better to have one per-device + shared for all accesing translators, or just each task should have its + own cache... + <antrik> slpz: + http://lists.gnu.org/archive/html/bug-hurd/2011-09/msg00041.html is where + I described the issue(s) + <antrik> (should send another update for the most recent findings I + guess...) + <antrik> slpz: well, if we move to userspace drivers, the kernel part of + the question is already answered ;-) + <antrik> but I'm not sure about per-device cache vs. caching in FS server diff --git a/open_issues/gcc.mdwn b/open_issues/gcc.mdwn index a3b3a2d2..71b4b2ea 100644 --- a/open_issues/gcc.mdwn +++ b/open_issues/gcc.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2007, 2008, 2009, 2010, 2011 Free Software +[[!meta copyright="Copyright © 2007, 2008, 2009, 2010, 2011, 2012 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -206,6 +206,20 @@ Last reviewed up to the [[Git mirror's 3d83581faf4eaf52c1cf52cc0d11cc7dd1264275 `mkdir` the directory for now, but what is really going on? GCC has *use `/usr/include` patch*, but glibc still installs into `/include/`? + * `__GLIBC__` + + IRC, freenode, #hurd, 2012-01-05: + + <civodul> on GNU/kFreeBSD, it's GCC that defines __GLIBC__, funny + <youpi> ?? + <youpi> not from features.h ? + <civodul> in gcc/config/kfreebsd-gnu.h + <civodul> :-) + <pinotree> correct, it's enabled in gcc's config + <pinotree> i discovered that after banging my head on the wall trying + to find out why some stuff wasn't compiling even after kfreebsd + porting patches adding preprocessors checks for __GLIBC__ + # Build diff --git a/open_issues/gnumach_memory_management.mdwn b/open_issues/gnumach_memory_management.mdwn index c9c3e64f..c34d1200 100644 --- a/open_issues/gnumach_memory_management.mdwn +++ b/open_issues/gnumach_memory_management.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -2012,3 +2012,84 @@ There is a [[!FF_project 266]][[!tag bounty]] on this task. <mcsim> braunr: everything is ok <braunr> good <braunr> i think the branch is ready for integration + + +# IRC, freenode, #hurd, 2011-12-17 + + <braunr> in the slab branch, there now is no use for the defines in + kern/mach_param.h + <braunr> should the file be removed or left empty as a placeholder for + future arbitrary limits ? + <braunr> (i'd tend ro remove it as a way of indicating we don't want + arbitrary limits but there may be a good reason to keep it around .. :)) + <youpi> I'd just drop it + <braunr> ok + <braunr> hmm maybe we do want to keep that one : + <braunr> #define IMAR_MAX (1 << 10) /* Max number of + msg-accepted reqs */ + <antrik> whatever that is... + <braunr> it gets returned in ipc_marequest_info + <braunr> but the mach_debug interface has never been used on the hurd + <braunr> there now is a master-slab branch in the gnumach repo, feel free + to test it + + +# IRC, freenode, #hurd, 2011-12-22 + + <youpi> braunr: does the new gnumach allocator has profiling features? + <youpi> e.g. to easily know where memory leaks reside + <braunr> youpi: you mean tracking call traces to allocated blocks ? + <youpi> not necessarily traces + <youpi> but at least means to know what kind of objects is filling memory + <braunr> it's very close to the zone allocator + <braunr> but instead of zones, there are caches + <braunr> each named after the type they store + <braunr> see http://www.sceen.net/~rbraun/slabinfo.out + <youpi> ok, so we can know, per-type, how much memory is used + <braunr> yes + <youpi> good + <braunr> if backtraces can easily be forged, it wouldn't be hard to add + that feature too + <youpi> does it dump such info when memory goes short? + <braunr> no but it can + <braunr> i've done this during tests + <youpi> it'd be good + <youpi> because I don't know in advance when a buildd will crash due to + that :) + <braunr> each time slab_collect() is called for example + <youpi> I mean not on collect, but when it's too late + <youpi> and thus always enabled + <braunr> ok + <youpi> (because there's nothing better to do than at least give infos) + <braunr> you just have to define "when it's too late", and i can add that + <youpi> when there is no memory left + <braunr> you mean when the number of free pages strictly reaches 0 ? + <youpi> yes + <braunr> ok + <youpi> i.e. just before crashing the kernel + <braunr> i see + + +# IRC, freenode, #hurdfr, 2012-01-02 + + <youpi> braunr: le code du slab allocator, il est écrit from scratch ? + <youpi> il y a encore du copyright carnegie mellon + <youpi> (dans slab_info.h du moins) + <youpi> ipc_hash_global_size = 256; + <youpi> il faudrait mettre 256 comme constante dans un header + <youpi> sinon c'est encore une valeur arbitraire cachée dans du code + <youpi> de même pour ipc_marequest_size etc. + <braunr> youpi: oui, from scratch + <braunr> slab_info.h est à l'origine zone_info.h + <braunr> pour les valeurs fixes, elles étaient déjà présentes de cette + façon, j'ai pensé qu'il valait mieux laisser comme ça pour faciliter la + lecture des diffs + <braunr> je ferai des macros à la place + <braunr> du coup il faudra peut-être remettre mach_param.h + <braunr> ou alors dans les .h ipc + + +# IRC, freenode, #hurdfr, 2012-01-18 + + <braunr> does the slab branch need other reviews/reports before being + integrated ? diff --git a/open_issues/libnfs.mdwn b/open_issues/libnfs.mdwn new file mode 100644 index 00000000..8cadefa4 --- /dev/null +++ b/open_issues/libnfs.mdwn @@ -0,0 +1,28 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +IRC, freenode, #hurd, 2012-01-09: + + <pinotree> https://github.com/sahlberg/libnfs ← maybe it could be used for + nfs support, instead of the rpc stuff "removed" in newer glibc versions + <antrik> pinotree: sounds like it could do much more than just the RPC + stuff -- definitely interesting :-) + <pinotree> hm but it seems to be an abstraction over either classic rpc or + tirpc + <pinotree> (anyway, it is packaged already in debian) + <antrik> good licensing too + <antrik> I guess I'll modify the GSoC task to "rework the Hurd NFS client + to use libnfs" :-) + <pinotree> the nfs translator? + <antrik> yes + +[[hurd/translator/nfs]] diff --git a/open_issues/mach-defpager_debugging.mdwn b/open_issues/mach-defpager_debugging.mdwn new file mode 100644 index 00000000..33e717d9 --- /dev/null +++ b/open_issues/mach-defpager_debugging.mdwn @@ -0,0 +1,22 @@ +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_gdb]] + +IRC, freenode, #hurd, 2011-11-10: + + <mcsim> hello. Is there any way to debug mach-defpager? When I set + breakpoint to any function in it, pager never breaks. + +IRC, freenode, #hurd, 2011-11-11: + + <mcsim> hello. I've read that hde tried to debug defpager and wrote some + patches for debugging defpager, but I couldn't find them. Does anybody + know are these patches in public? diff --git a/open_issues/mmap_write-only.mdwn b/open_issues/mmap_write-only.mdwn new file mode 100644 index 00000000..467274c5 --- /dev/null +++ b/open_issues/mmap_write-only.mdwn @@ -0,0 +1,56 @@ +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_glibc]] + +# IRC, freenode, #hurd, 2011-12-14 + + <pinotree> hm, interesting mmap bug + <youpi> ? + <pinotree> youpi: http://paste.debian.net/149252/ + #include <sys/types.h> + #include <sys/mman.h> + #include <fcntl.h> + #include <stdio.h> + #include <errno.h> + #include <stdlib.h> + #include <string.h> + #include <unistd.h> + + void die(int x, const char *s) + { + perror(s); + exit(x); + } + + static const char s_file[] = "foo-mmaptest"; + + int main() + { + int fd; + void *p; + + fd = creat(s_file, 0777); + if (fd < 0) die(1, "creat"); + errno = 0; + p = mmap(NULL, 1, PROT_READ, MAP_SHARED, fd, 0); + printf("> %p vs %p, %d (%s)\n", p, MAP_FAILED, errno, strerror(errno)); + unlink(s_file); + return (p != MAP_FAILED); + } + <pinotree> on linux it returns 0 and fails with EACCESS (as it seems it + should, by reading the mmap posix docs), on hurd it returns 1 and the + mmap succeeds + <pinotree> (taken from llvm's configure) + <youpi> why should it? file size extension ? + <pinotree> creat creates a o_wronly file, while the mmap specifies only + read protection + <youpi> oh, craet is always wo + <youpi> I didn't know that diff --git a/open_issues/open_symlink.mdwn b/open_issues/open_symlink.mdwn new file mode 100644 index 00000000..20e4a4fe --- /dev/null +++ b/open_issues/open_symlink.mdwn @@ -0,0 +1,18 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_glibc]] + +# IRC, freenode, #hurd, 2012-01-02 + + <pinotree> hm, is it a known issue that open("somesymlink", O_RDONLY | + O_NOFOLLOW) does not fail with ELOOP? + <youpi> pinotree: iirc there is code for it, maybe not the same behavior as + on linux diff --git a/open_issues/performance/degradation.mdwn b/open_issues/performance/degradation.mdwn index 64542af3..e520ed6a 100644 --- a/open_issues/performance/degradation.mdwn +++ b/open_issues/performance/degradation.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -46,3 +46,6 @@ IRC, freenode, #hurd, 2011-09-02: # IRC, freenode, #hurd, 2011-09-22 See [[/open_issues/pagers]], IRC, freenode, #hurd, 2011-09-22. + + +# [[ext2fs_page_cache_swapping_leak]] diff --git a/open_issues/select.mdwn b/open_issues/select.mdwn index 0f750631..0b69d645 100644 --- a/open_issues/select.mdwn +++ b/open_issues/select.mdwn @@ -1,4 +1,5 @@ -[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011, 2012 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -29,6 +30,12 @@ IRC, unknown channel, unknown date. respond in time? <paakku> Or do I just imagine this problem? ---- + +# dbus + +See [[dbus]]. + + +# See Also See also [[select_bogus_fd]] and [[select_vs_signals]]. diff --git a/open_issues/sendmsg_scm_creds.mdwn b/open_issues/sendmsg_scm_creds.mdwn index c613e21c..cf0103df 100644 --- a/open_issues/sendmsg_scm_creds.mdwn +++ b/open_issues/sendmsg_scm_creds.mdwn @@ -1,4 +1,5 @@ -[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011, 2012 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -96,4 +97,5 @@ IRC, unknown channel, unknown date. --- -See also [[pflocal_socket_credentials_for_local_sockets]] and [[pflocal_reauth]]. +See also [[dbus]], [[pflocal_socket_credentials_for_local_sockets]] and +[[pflocal_reauth]]. diff --git a/open_issues/servers_default-pager_permissions.mdwn b/open_issues/servers_default-pager_permissions.mdwn new file mode 100644 index 00000000..58dba1cb --- /dev/null +++ b/open_issues/servers_default-pager_permissions.mdwn @@ -0,0 +1,27 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="/servers/default-pager permissions"]] + +[[!tag open_issue_hurd]] + +IRC, freenode, #hurd, 2012-01-14: + + <youpi> antrik: what are the permissions that are supposed to be given to + /servers/default-pager ? + <antrik> olaf@alien:~$ ls -l /servers/default-pager + <antrik> crw-rw-rw- 1 root root 0, 0 Sep 17 2004 /servers/default-pager + <antrik> oh, interesting... in the other system it's different + <antrik> olaf@alien:~$ ls -l /sub/servers/default-pager + <antrik> crw-r--r-- 1 root root 0, 0 Jul 10 2006 + /sub/servers/default-pager + <antrik> both are Debian, the latter installed with crosshurd + <antrik> (and native-install run in a chroot or subhurd, don't remember + which...) |