From bcfc058a332da0a2bd2e09e13619be3e2eb803a7 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Tue, 11 Dec 2012 11:04:26 +0100 Subject: IRC. --- open_issues/anatomy_of_a_hurd_system.mdwn | 111 +++++++++++++-- open_issues/fakeroot_eagain.mdwn | 216 +++++++++++++++++++++++++++++ open_issues/gnumach_memory_management.mdwn | 49 +++++++ open_issues/libpthread.mdwn | 92 ++++++++++++ open_issues/netstat.mdwn | 34 +++++ open_issues/performance.mdwn | 103 ++++++++++++++ open_issues/robustness.mdwn | 65 +++++++++ open_issues/select.mdwn | 128 +++++++++++++++++ 8 files changed, 783 insertions(+), 15 deletions(-) create mode 100644 open_issues/fakeroot_eagain.mdwn create mode 100644 open_issues/netstat.mdwn (limited to 'open_issues') diff --git a/open_issues/anatomy_of_a_hurd_system.mdwn b/open_issues/anatomy_of_a_hurd_system.mdwn index 99ef170b..3e585876 100644 --- a/open_issues/anatomy_of_a_hurd_system.mdwn +++ b/open_issues/anatomy_of_a_hurd_system.mdwn @@ -13,7 +13,10 @@ License|/fdl]]."]]"""]] A bunch of this should also be covered in other (introductionary) material, like Bushnell's Hurd paper. All this should be unfied and streamlined. -IRC, freenode, #hurd, 2011-03-08: +[[!toc]] + + +# IRC, freenode, #hurd, 2011-03-08 I've a question on what are the "units" in the hurd project, if you were to divide them into units if they aren't, and what are the @@ -38,9 +41,8 @@ IRC, freenode, #hurd, 2011-03-08: no servers often depend on other servers for certain functionality ---- -IRC, freenode, #hurd, 2011-03-12: +# IRC, freenode, #hurd, 2011-03-12 when mach first starts up, does it have some basic i/o or fs functionality built into it to start up the initial hurd translators? @@ -72,24 +74,24 @@ IRC, freenode, #hurd, 2011-03-12: it also does some bootstrapping work during startup, to bring the rest of the system up ---- + +# Source Code Documentation Provide a cross-linked sources documentation, including generated files, like RPC stubs. * ---- -[[Hurd_101]]. +# [[Hurd_101]] + ---- +# [[hurd/IO_path]] -More stuff like [[hurd/IO_path]]. +Need more stuff like that. ---- -IRC, freenode, #hurd, 2011-10-18: +# IRC, freenode, #hurd, 2011-10-18 what happens @ boot. and which translators are started in what order? @@ -97,9 +99,8 @@ IRC, freenode, #hurd, 2011-10-18: ext2; ext2 starts exec; ext2 execs a few other servers; ext2 execs init. from there on, it's just standard UNIX stuff ---- -IRC, OFTC, #debian-hurd, 2011-11-02: +# IRC, OFTC, #debian-hurd, 2011-11-02 is __dir_lookup a RPC ?? where can i find the source of __dir_lookup ?? @@ -123,9 +124,8 @@ IRC, OFTC, #debian-hurd, 2011-11-02: sekon_: This may help a bit: http://www.gnu.org/software/hurd/hurd/hurd_hacking_guide.html ---- -IRC, freenode, #hurd, 2012-01-08: +# IRC, freenode, #hurd, 2012-01-08 can you tell me how is done in hurd: "ls | grep x" ? in bash @@ -187,7 +187,8 @@ IRC, freenode, #hurd, 2012-01-08: that's probably the most fundamental design feature of the Hurd (all filesystem operations actually, not only lookups) -IRC, freenode, #hurd, 2012-01-09: + +## IRC, freenode, #hurd, 2012-01-09 youpi: are you sure cthreads are M:N ? i'm almost sure they're 1:1 and no modern OS is a right place for any thread userspace @@ -266,3 +267,83 @@ IRC, freenode, #hurd, 2012-01-09: they help only when the threads are living ok now as I said I don't have to talk much more, I have to leave :) + + +# IRC, freenode, #hurd, 2012-12-06 + + spiderweb: have you read + http://www.gnu.org/software/hurd/hurd-paper.html ? + I'll have a look. + and also the beginning of + http://ftp.sceen.net/mach/mach_a_new_kernel_foundation_for_unix_development.pdf + these two should provide a good look at the big picture the hurd + attemtps to achieve + I can't help but wonder though, what advantages were really + achieved with early mach? + weren't they just running a monolithic unix server like osx does? + most mach-based systems were + but thanks to that, they could provide advanced features over + other well established unix systems + while also being compatible + so basically it was just an ease of development thing + well that's what mach aimed at being + same for the hurd + making things easy + but as a side effect hurd actually delivers on the advantages of + microkernels aside from that, but the older systems wouldn't, correct? + that's how there could be network file systems in very short time + and very scarce resources (i.e. developers working on it), while on other + systems it required a lot more to accomplish that + no, it's not a side effect of the microkernel + the hurd retains and extends the concept of flexibility introduced + by mach + the improved stability, etc. isn't a side effect of being able to + restart generally thought of as system-critical processes? + no + you can't restart system critical processes on the hurd either + that's one feature of minix, and they worked hard on it + ah, okay. so that's currently just the domain of minix + okay + spiderweb: well, there's 1 advantage of minix for you :P + the main idea of mach is to make it easy to extend unix + without having hundreds of system calls + the hurd keeps that and extends it by making many operations + unprivileged + you don't need special code for kernel modules any more + it's easy + you don't need special code to handle suid bits and other ugly + similar hacks, + it's easy + you don't need fuse + easy + etc.. + + +# IRC, freenode, #hurd, 2012-12-06 + + what is the #1 feature that distinguished hurd from other + operating systems. the concept of translators. (will read more when I get + more time). + yes, translators + using the VFS as a service directory + and the VFS permissions to control access to those services + + +# IRC, freenode, #hurd, 2012-12-10 + + I want to work on hurd, but I think I'm going to start with + minix, I own the minix book 3rd ed. it seems like a good intro to + operating systems in general. like I don't even know what a semaphore is + yet. + well, enjoy learning :) + once I finish that book, what reading do you guys recommend? + other than the wiki + i wouldn't recommend starting with a book that focuses on one + operating system anyway + you tend to think in terms of what is done in that specific + implementation and compare everything else to that + tannenbaum is not only the main author or minix, but also the one + of the book http://en.wikipedia.org/wiki/Modern_Operating_Systems + + http://en.wikipedia.org/wiki/List_of_important_publications_in_computer_science#Operating_systems + should be a pretty good list :) diff --git a/open_issues/fakeroot_eagain.mdwn b/open_issues/fakeroot_eagain.mdwn new file mode 100644 index 00000000..6b684a04 --- /dev/null +++ b/open_issues/fakeroot_eagain.mdwn @@ -0,0 +1,216 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_glibc open_issue_porting]] + + +# IRC, freenode, #hurd, 2012-12-05 + + rbraun 18813 R 2hrs ln -sf ../af_ZA/LC_NUMERIC + debian/locales-all/usr/lib/locale/en_BW/LC_NUMERIC + when building glibc + is this a known issue ? + braunr: No. Can you get a backtrace? + tschwinge: with gdb you mean ? + Yes. If you have any debugging symbols (glibc?). + or the build log leading to that ? + ok, i will next time i have it + OK. + (i regularly had it when working on the pthreads port) + tschwinge: + http://www.sceen.net/~rbraun/hurd_glibc_build_deadlock_trace + youpi: ^ + Mmm, there's not so much we can do about this one + youpi: what do you mean ? + the problem is that it's really a reentrency issue of the libc + locale + it would happen just the same on linux + sure + but hat doesn't mean we can't report and/or fix it :) + (the _nl_state_lock) + do you have any workaround in mind ? + no + actually that's what I meant by "there's not so much we can do + about this" + ok + because it's a bad interaction between libfakeroot and glibc + glibc believe fxtstat64 would never call locale functions + but with libfakeroot it does + i see + only because we get an EAGAIN here + but hm, doesn't it happen on linux ? + EAGAIN doesn't happen on linux for fxstat64, no :) + why does it happen on the hurd ? + I mean for fakeroot stuff + probably because fakeroot uses socket functions + for which we probably don't properly handleEAGAIN + I've already seen such kind of issue + in buildd failures + ok + (so the actual bug here is EAGAIN + ) + yes, so we can do something about it + worth a look + (implement sysv semaphores) + pinotree: if we could also solve all these buildd EAGAIN issues + that'd be nice :) + that EAGAIN error might also be what makes exim behave badly and + loop forever + possibly + i've updated the trace with debugging symbols + it fails on connect + like http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=563342 ? + it's EAGAIN, not ECONNREFUSED + ah ok + might be an error in tcp_v4_get_port + + +## IRC, freenode, #hurd, 2012-12-06 + + hmm, tcp_v4_get_port sometimes fails indeed + braunr: may I ask how you found out, adding print statements in + pfinet, or? + yes + OK, so that's the only (easy) way to debug. + that's the last resort + gdb is easy too + i could have added a breakpoint too + but i didn't want to block pfinet while i was away + is it possible to force the use of fakeroot-tcp on linux ? + the problem seems to be that fakeroot doesn't close the sockets + that it connected to faked-tcp + which, at some point, exhauts the port space + braunr: sure + change the fakeroot dpkg alternative + ok + calling it explicitly `fakeroot-tcp command` or + `dpkg-buildpackage -rfakeroot-tcp ...` should work too + fakeroot-tcp looks really evil :p + hum, i don't see any faked-tcp process on linux :/ + not even with `fakeroot-tcp bash -c "sleep 10"`? + pinotree: now yes + but, does it mean faked-tcp is started for *each* process loading + fakeroot-tcp ? + (the lib i mean) + i think so + well the hurd doesn't seem to do that at all + or maybe it does and i don't see it + the stale faked-tcp processes could be those that failed something + only + yes, there's also that issue: sometimes there are stake + faked-tcp processes + hum no, i see one faked-tcp that consumes cpu when building glibc + *stale + it's the same process for all commands + but, does it mean faked-tcp is started for *each* + process loading fakeroot-tcp ? + → everytime you start fakeroot, there's a new faked-xxx for it + it doesn't look that way + again, on the hurd, i see one faked-tcp, consuming cpu while + building so i assume it services libfakeroot-tcp requests + yes + which means i probably won't reproduce the problem on linux + it serves that fakeroot under which the binary(-arch) target is + run + or perhaps it's the normal fakeroot-tcp behaviour on sid + pinotree: a faked-tcp that is started for each command invocation + will implicitely make the network stack close all its sockets when + exiting + pinotree: as our fakeroot-tcp uses the same instance of faked-tcp, + it's a lot more likely to exhaust the port space + i see + i'll try on sid and see how it behaves + pinotree: on the other hand, forking so many processes at each + command invocation may make exec leak a lot :p + or rather, a lot more + (or maybe not, since it leaks only in some cases) + +[[exec_leak]]. + + pinotree: actually, the behaviour under linux is the same with the + alternative correctly set, whereas faked-tcp is restarted (if used at + all) with -rfakeroot-tcp + hm no, even that isn't true + grr + pinotree: i think i found a handy workaround for fakeroot + pinotree: the range of local ports in our networking stack is a + lot more limited than what is configured in current systems + by extending it, i can now build glibc \o/ + braunr: what are the current ours and the usual one? + see pfinet/linux-src/net/ipv4/tcp_ipv4.c + the modern ones are the ones suggested in the comment + sysctl_local_port_range is the symbol storing the range + i see + what's the current range on linux? + 20:44 < braunr> the modern ones are the ones suggested in the + comment + i see + $ cat /proc/sys/net/ipv4/ip_local_port_range + 32768 61000 + so, i'm not sure why we have the problem, since even on linux, + netstat doesn't show open bound ports, but it does help + the fact faked-tcp can remain after its use is more problematic + (maybe pfinet could grow a (startup-only?) option to change it, + similar to that sysctl) + but it can also stems from the same issue gnu_srs found about + closed sockets that haven't been shut down + perhaps + but i don't see the point actually + we could simply change the values in the code + + youpi: first, in pfinet, i increased the range of local ports to + reduce the likeliness of port space exhaustion + so we should get a lot less EAGAIN after that + (i've not committed any of those changes) + range of local ports? + see pfinet/linux-src/net/ipv4/tcp_ipv4.c, tcp_v4_get_port function + and sysctl_local_port_range array + oh + EAGAIN is caused by tcp_v4_get_port failing at + /* Exhausted local port range during search? */ + if (remaining <= 0) + goto fail; + interesting + so it's not a hurd bug after all + just a problem in fakeroot eating a lot of ports + maybe because of the same issue gnu_srs worked on (bad socket + close when no clean shutdown) + maybe, maybe not + but increasing the range is effective + and i compared with what linux does today, which is exactly what + is in the comment above sysctl_local_port_range + so it looks safe + so that means that the pfinet just uses ports 1024- 4999 for + auto-allocated ports? + i guess so + the linux pfinet I meant + i haven't checked the whole code but it looks that way + ./sysctl_net_ipv4.c:static int ip_local_port_range_min[] = { 1, 1 + }; + ./sysctl_net_ipv4.c:static int ip_local_port_range_max[] = { 65535, + 65535 }; + looks like they have increased it since then :) + hum :) + $ cat /proc/sys/net/ipv4/ip_local_port_range + 32768 61000 + yep, same here + ./inet_connection_sock.c: .range = { 32768, 61000 }, + so there are two things apparently + but linux now defaults to 32k-61k + braunr: please just push the port range upgrade to 32Ki-61K + ok, will do + there's not reason not to do it + + +## IRC, freenode, #hurd, 2012-12-11 + + youpi: at least, i haven't had any failure building eglibc since + the port range patch + good :) diff --git a/open_issues/gnumach_memory_management.mdwn b/open_issues/gnumach_memory_management.mdwn index 9feb30c8..e5e9d2c5 100644 --- a/open_issues/gnumach_memory_management.mdwn +++ b/open_issues/gnumach_memory_management.mdwn @@ -2133,3 +2133,52 @@ There is a [[!FF_project 266]][[!tag bounty]] on this task. do you want to review ? I don't think there is any need to ok + + +# IRC, freenode, #hurd, 2012-12-08 + + braunr: hi. Do I understand correct that merely the same technique + is used in linux to determine the slab where, the object to be freed, + resides? + yes but it's faster on linux since it uses a direct mapping of + physical memory + it just has to shift the virtual address to obtain the physical + one, whereas x15 has to walk the pages tables + of course it only works for kmalloc, vmalloc is entirely different + btw, is there sense to use some kind of B-tree instead of AVL to + decrease number of cache misses? AFAIK, in modern processors size of L1 + cache line is at least 64 bytes, so in one node we can put at least 4 + leafs (key + pointer to data) making search faster. + that would be a b-tree + and yes, red-black trees were actually developed based on + properties observed on b-trees + but increasing the size of the nodes also increases memory + overhead + and code complexity + that's why i have a radix trees for cases where there are a large + number of entries with keys close to each other :) + a radix-tree is basically a b-tree using the bits of the key as + indexes in the various arrays it walks instead of comparing keys to each + other + the original avl tree used in my slab allocator was intended to + reduce the average height of the tree (avl is better for that) + avl trees are more suited for cases where there are more lookups + than inserts/deletions + they make the tree "flatter" but the maximum complexity of + operations that change the tree is 2log2(n), since rebalancing the tree + can make the algorithm reach back to the tree root + red-black trees have slightly bigger heights but insertions are + limited to 2 rotations and deletions to 3 + there should be not much lookups in slab allocators + which explains why they're more generally found in generic + containers + or do I misunderstand something? + well, there is a lookup for each free() + whereas there are insertions/deletions when a slab becomes + non-empty/empty + I see + so it was very efficient for caches of small objects, where slabs + have many of them + also, i wrote the implementation in userspace, without + functionality pmap provides (although i could have emulated it + afterwards) diff --git a/open_issues/libpthread.mdwn b/open_issues/libpthread.mdwn index 81f1a382..befc1378 100644 --- a/open_issues/libpthread.mdwn +++ b/open_issues/libpthread.mdwn @@ -1234,3 +1234,95 @@ There is a [[!FF_project 275]][[!tag bounty]] on this task. of a "message server" à la dmesg [[translator_stdout_stderr]]. + + +### IRC, freenode, #hurd, 2012-12-10 + + braunr: unable to adjust libports thread priority: (ipc/send) + invalid destination port + I'll see what package brought that + (that was on a buildd) + wow + mkvtoolnix_5.9.0-1: + shouldn't that code be done in pthreads and then using such + pthread api? :p + pinotree: you've already asked that question :p + i know :p + the semantics of pthreads are larger than what we need, so that + will be done "later" + but this error shouldn't happen + it looks more like a random mach bug + youpi: anything else on the console ? + nope + i'll add traces to know which step causes the error + + +## IRC, freenode, #hurd, 2012-12-05 + + tschwinge: i'm currently working on a few easy bugs and i have + planned improvements for libpthreads soon + wotwot, which ones? + pinotree: first, fixing pthread_cond_timedwait (and everything + timedsomething actually) + pinotree: then, fixing cancellation + pinotree: and last but not least, optimizing thread wakeup + i also want to try replacing spin locks and see if it does what i + expect + which fixes do you plan applying to cond_timedwait? + see sysdeps/generic/pt-cond-timedwait.c + the FIXME comment + ah that + well that's important :) + did you have something else in mind ? + hm, __pthread_timedblock... do you plan fixing directly there? i + remember having seem something related to that (but not on conditions), + but wasn't able to see further + it has the same issue + i don't remember the details, but i wrote a cthreads version that + does it right + in the io_select_timeout branch + see + http://git.savannah.gnu.org/cgit/hurd/hurd.git/tree/libthreads/cancel-cond.c?h=rbraun/select_timeout + for example + * pinotree looks + what matters is the msg_delivered member used to synchronize + sleeper and waker + the waker code is in + http://git.savannah.gnu.org/cgit/hurd/hurd.git/tree/libthreads/cprocs.c?h=rbraun/select_timeout + never seen cthreads' code before :) + soon you shouldn't have any more reason to :p + ah, so basically the cthread version of the pthread cleanup + stack + cancelation (ie the cancel hook) broadcasts the condition + yes + so a similar fix would be needed in all the places using + __pthread_timedblock, that is conditions and mutexes + and that's what's missing in glibc that prevents deploying a + pthreads based hurd currently + no that's unrelated + ok + the problem is how __pthread_block/__pthread_timedblock is + synchronized with __pthread_wakeup + libpthreads does exactly the same thing as cthreads for that, + i.e. use messages + but the message alone isn't enough, since, as explained in the + FIXME comment, it can arrive too late + it's not a problem for __pthread_block because this function can + only resume after receiving a message + but it's a problem for __pthread_timedblock which can resume + because of a timeout + my solution is to add a flag that says whether a message was + actually sent, and lock around sending the message, so that the thread + resume can accurately tell in which state it is + and drain the message queue if needed + i see, race between the "i stop blocking because of timeout" and + "i stop because i got a message" with the actual check for the real cause + locking around mach_msg may seem overkill but it's not in + practice, since there can only be one message at most in the message + queue + and i checked that in practice by limiting the message queue size + and check for such errors + but again, it would be far better with mutexes only, and no spin + locks + i wondered for a long time why the load average was so high on the + hurd under even "light" loads + now i know :) diff --git a/open_issues/netstat.mdwn b/open_issues/netstat.mdwn new file mode 100644 index 00000000..b575ea7f --- /dev/null +++ b/open_issues/netstat.mdwn @@ -0,0 +1,34 @@ +[[!meta copyright="Copyright © 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_glibc open_issue_hurd open_issue_porting]] + + +# IRC, freenode, #hurd, 2012-12-06 + + we need a netstat command + wouldn't that require rpcs and notifications in pfinet to get + info on the known sockets? + depends on the interface + netstat currently uses /proc/net/* so that's out of the question + but a bsd netstat using ioctls could do the job + i'm not sure if it's done that way + i don't see why it would require notifications though + if add such rpcs to pfinet, you could show the sockets in procfs + yes + that's the clean way :p + but why notifications ? + to get changes on data of sockets (status change, i/o activity, + etc) + (possibly i'm forgetting some already there features to know + that) + the socket state is centralized in pfinet + netstat polls it + (or asks it once) diff --git a/open_issues/performance.mdwn b/open_issues/performance.mdwn index 8147e5eb..ae05e128 100644 --- a/open_issues/performance.mdwn +++ b/open_issues/performance.mdwn @@ -83,6 +83,109 @@ call|/glibc/fork]]'s case. ouch +## [[!message-id "20121202101508.GA30541@mail.sceen.net"]] + + +## IRC, freenode, #hurd, 2012-12-04 + + why do some people think hurd is slow? i find it works well even + under heavy load inside a virtual machine + damo22: the virtual machine actually assists the hurd a lot :p + but even with that, the hurd is a slow system + i would have thought it would have the potential to be very fast, + considering the model of the kernel + the design implies by definition more overhead, but the true cause + is more than 15 years without optimization on the core components + how so ? + since there are less layers of code between the hardware bare + metal and the application that users run + how so ? :) + it's the contrary actually + VFS -> IPC -> scheduler -> device drivers -> hardware + that is monolithic + well, it's not really meaningful + and i'd say the same applies for a microkernel system + if the application can talk directly to hardware through the + kernel its almost like plugging directly into the hardware + you never talk directly to hardware + you talk to servers instead of the kernel + ah + consider monolithic kernel systems like systems with one big + server + the kernel + whereas a multiserver system is a kernel and many servers + you still need the VFS to identify your service (and thus your + server) + you need much more IPC, since system calls are "replaced" with RPC + the scheduler is basically the same + okay + device drivers are similar too, except they run in thread context + (which is usually a bit heavier) + but you can do cool things like report when an interrupt line is + blocked + and there are many context switches between all that + you can do all that in a monolithic kernel too, and faster + but it's far more elegant, and (when well done) easy to do on a + microkernel based system + yes + i like elegant, makes coding easier if you know the basics + there are only two major differences between a monolilthic kernel + and a multiserver microkernel system + * damo22 listens + 1/ independence of location (your resources could be anywhere) + 2/ separation of address spaces (your servers have their own + addresses) + wow + these both imply additional layers of indirection, making the + system as a whole slower + but it would be far more secure though i suspect + yes + and reliable + that's why systems like qnx were usually adopted for critical + tasks + security and reliability are very important, i would switch to the + hurd if it supported all the hardware i use + so would i :) + but performance matters too + not to me + it should :p + it really does matter a lot in practice + i mean, a 2x slowdown compared to linux would not affect me + if it had all the benefits we mentioned above + but the hurd is really slow for other reasons than its additional + layers of indrection unfortunately + is it because of lack of optimisation in the core code? + we're working on these issues, but it's not easy and takes a lot + of time :p + like you said + yes + and also because of some fundamental design choices related to the + microkernel back in the 80s + what about the darwin system + it uses a mach kernel? + yes + what is stopping someone taking the MIT code from darwin and + creating a monster free OS + what for ? + because it already has hardware support + and a mach kernel + in kernel drivers ? + it has kernel extensions + you can do things like kextload module + first, being a mach kernel doesn't make it compatible or even + easily usable with the hurd, the interfaces have evolved independantly + and second, we really do want more stuff out of the kernel + drivers in particular + may i ask why you are very keen to have drivers out of kernel? + for the same reason we want other system services out of the + kernel + security, reliability, etc.. + ease of debugging + the ability to restart drivers separately, without restarting the + kernel + i see + + # IRC, freenode, #hurd, 2012-09-13 {{$news/2011-q2#phoronix-3}}. diff --git a/open_issues/robustness.mdwn b/open_issues/robustness.mdwn index d32bd509..1f8aa0c6 100644 --- a/open_issues/robustness.mdwn +++ b/open_issues/robustness.mdwn @@ -62,3 +62,68 @@ License|/fdl]]."]]"""]] well, I'm not aware of the Minix implementation working across reboots. the one I have in mind based on a generic session management infrastructure should though :-) + + +## IRC, freenode, #hurd, 2012-12-06 + + out of curiosity, would it be possible to strap on a resurrection + server to hurd? + in the future, that is + sure + cool :) + but this requires things like persistence + like a reincarnation server? + it's a lot of works, with non negligible overhead + spiderweb: yes, exactly. I didn't remember tanenbaum's wording on + that + i'm pretty sure most people would be against that + braunr: why so? + it was actually the feature that convinced me that ukernels were a + good idea + spiderweb: because then you need a process that keeps track of all + the other servers + and they have to be replying to "useless" pings to see if they're + still alive + spiderweb: the hurd community isn't looking for a system reliable + in critical environments + just a general purpose system + and persistence requires regular data saves + it's expensive + as well as that + we already have performance problems because of the nature of the + system, adding more without really looking for the benefits is useless + so you can't theoretically have both? + persistence and performance ? + it's hard + spiderweb: you need to modify the other translators to be + persistent + only the ones you care about actually + but it's just better to make the critical servers very stable + so it's not just turning on and off the reincarnation + (there isn't that much code there) + and the other servers restartable + braunr: I think that if there will be aim to make something like + resurrection server than it will be needed rewrite most servers to make + them stateless, isn't it? + that's a lot easier and already works with non essential passive + translators + mcsim: pretty much + mcsim: only those you care about + mcsim: the proc auth exec servers for example, perhaps the file + system servers that can act as root fs, but the others would simply be + restarted by the passive translator mechanism + what about restarting device drivers, that would be simple + right? + that's perfectly doable, yes + (being an OS newbie) - it does seem to me that the whole + reincarnation server concept could quite possibly be a band aid. + spiderweb: no it really works + many systems do that actually + let me give you a link + + http://ftp.sceen.net/curios_improving_reliability_through_operating_system_structure.pdf + it's a bit old, but there is a review of systems aiming at + resilience and how they achieve part of it + neat, thanks + actually it's not that old at all + around 2007 diff --git a/open_issues/select.mdwn b/open_issues/select.mdwn index 12807e11..778af530 100644 --- a/open_issues/select.mdwn +++ b/open_issues/select.mdwn @@ -1503,6 +1503,134 @@ IRC, unknown channel, unknown date: [[Term_blocking]]. +# IRC, freenode, #hurd, 2012-12-05 + + well if i'm unable to build my own packages, i'll send you the one + line patch i wrote that fixes select/poll for the case where there is + only one descriptor + (the current code calls mach_msg twice, each time with the same + timeout, doubling the total wait time when there is no event) + + +## IRC, freenode, #hurd, 2012-12-06 + + damn, my eglibc patch breaks select :x + i guess i'll just simplify the code by using the same path for + both single fd and multiple fd calls + at least, the patch does fix the case i wanted it to .. :) + htop and ping act at the right regular interval + my select patch is : + /* Now wait for reply messages. */ + - if (!err && got == 0) + + if (!err && got == 0 && firstfd != -1 && firstfd != lastfd) + basically, when there is a single fd, the code calls io_select + with a timeout + and later calls mach_msg with the same timeout + effectively making the maximum wait time twice what it should be + ouch + which is why htop and ping are "laggy" + and perhaps also why fakeroot is when building libc + well + when building packages + my patch avoids entering the mach_msg call if there is only one fd + (my failed attempt didn't have the firstfd != -1 check, leading to + the 0 fd case skipping mach_msg too, which is wrong since in that case + there is just no wait, making applications use select/poll for sleeping + consume all cpu) + + the second is a fix in select (yet another) for the case where a + single fd is passed + in which case there is one timeout directly passed in the + io_select call, but then yet another in the mach_msg call that waits for + replies + this can account for the slowness of a bunch of select/poll users + + +## IRC, freenode, #hurd, 2012-12-07 + + finally, my select patch works :) + + +## IRC, freenode, #hurd, 2012-12-08 + + for those interested, i pushed my eglibc packages that include + this little select/poll timeout fix on my debian repository + deb http://ftp.sceen.net/debian-hurd experimental/ + reports are welcome, i'm especially interested in potential + regressions + + +## IRC, freenode, #hurd, 2012-12-10 + + I have verified your double timeout bug in hurdselect.c. + Since I'm also working on hurdselect I have a few questions + about where the timeouts in mach_msg and io_select are implemented. + Have a big problem to trace them down to actual code: mig magic + again? + yes + see hurd/io.defs, io_select includes a waittime timeout: + natural_t; parameter + waittime is mig magic that tells the client side not to wait more + than the timeout + and in _hurd_select, you can see these lines : + err = __io_select (d[i].io_port, d[i].reply_port, + /* Poll only if there's a single + descriptor. */ + (firstfd == lastfd) ? to : 0, + to being the timeout previously computed + "to" + and later, when waiting for replies : + while ((msgerr = __mach_msg (&msg.head, + MACH_RCV_MSG | options, + 0, sizeof msg, portset, to, + MACH_PORT_NULL)) == + MACH_MSG_SUCCESS) + the same timeout is used + hope it helps + Additional stuff on io-select question is at + http://paste.debian.net/215401/ + Sorry, should have posted it before you comment, but was + disturbed. + 14:13 < braunr> waittime is mig magic that tells the client side + not to wait more than the timeout + the waittime argument is a client argument only + that's one of the main source of problems with select/poll, and + the one i fixed 6 months ago + so there is no relation to the third argument of the client call + and the third argument of the server code? + no + the 3rd argument at server side is undoubtedly the 4th at client + side here + but for the fourth argument there is? + i think i've just answered that + when in doubt, check the code generated by mig when building glibc + as I said before, I have verified the timeout bug you solved. + which code to look for RPC_*? + should be easy to guess + is it the same with mach_msg()? No explicit usage of the timeout + there either. + in the code for the function I mean. + gnu_srs: mach_msg is a low level system call + see + http://www.gnu.org/software/hurd/gnumach-doc/Mach-Message-Call.html#Mach-Message-Call + found the definition of __io_select in: RPC_io_select.c, thanks. + so the client code to look for wrt RPC_ is in hurd/*.defs? what + about the gnumach/*/include/*.defs? + a final question: why use a timeout if there is a single FD for + the __io_select call, not when there are more than one? + well, the code is obviously buggy, so don't expect me to justify + wrong code + but i suppose the idea was : if there is only one fd, perform a + classical synchronous RPC, whereas if there are more use a heavyweight + portset and additional code to receive replies + + exim4 didn't get fixed by the libc patch, unfortunately + yes i noticed + gdb can't attach correctly to exim, so it's probably something + completely different + i'll try the non intrusive mode + + # See Also See also [[select_bogus_fd]] and [[select_vs_signals]]. -- cgit v1.2.3