[[!meta copyright="Copyright © 2010, 2011, 2012, 2013 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] [[!tag open_issue_glibc]] There are a lot of reports about this issue, but no thorough analysis. # Short Timeouts ## `elinks` IRC, unknown channel, unknown date: This is related to ELinks... I've looked at the select() implementation for the Hurd in glibc and it seems that giving it a short timeout could cause it not to report that file descriptors are ready. It sends a request to the Mach port of each file descriptor and then waits for responses from the servers. Even if the file descriptors have data for reading or are ready for writing, the server processes might not respond immediately. So if I want ELinks to check which file descriptors are ready, how long should the timeout be in order to ensure that all servers can respond in time? Or do I just imagine this problem? ## [[dbus]] ## IRC ### IRC, freenode, #hurd, 2012-01-31 don't you find vim extremely slow lately ? (and not because of cpu usage but rather unnecessary sleeps) yes. wasn't there a discussion to add a minimum timeout to mach_msg for select() or something like that during the past months ? there was, and it was added that could be it I don't want to drop it though, some app really need it as a debian patch only iirc ? yes ok if i'm right, the proper solution was to fix remote servers instead of client calls (no drop, unless the actual bug gets fixed of course) so i'm guessing it's just a hack in between not only with a timeout of zero, mach will just give *no* time for the servers to give an answer that's because the timeout is part of the client call so the protocol has to be rethought, both server/client side a suggested solution was to make it a parameter i mean, part of the message not a mach_msg parameter OTOH the servers should probably not be trusted to enforce the timeout. why ? they're not necessarily trusted. (but then again, that's not the only circumstances where that's a problem) there is a proposed solution for that too (trust root and self servers only by default) I'm not sure they're particularily easy to identify in the general case "they" ? the solutions you mean ? or the servers ? jkoenig: you can't trust the servers in general to provide an answer, timeout or not yes the root/self servers. ah jkoenig: you can stat the actual node before dereferencing the translator could they not report FD activity asynchronously to the message port? libc would cache the state I don't understand what you mean anyway, really making the timeout part of the message is not a problem 10:10 < youpi> jkoenig: you can't trust the servers in general to provide an answer, timeout or not we already trust everything (e.g. read() ) into providing an answer immediately i don't see why braunr: put sleep(1) in S_io_read() it'll not give you an immediate answer, O_NODELAY being set or not well sleep is evil, but let's just say the server thread blocks ok well fix the server so we agree ? in the current security model, we trust the server into achieve the timeout yes and jkoenig's remark is more global than just select() taht's why we must make sure we're contacting trusted servers by default it affects read() too sure so there's no reason not to fix select() that's the important point but this doesn't mean we shouldn't pass the timeout to the server and expect it to handle it correctly we keep raising issues with things, and not achieve anything, in the Hurd if it doesn't, then it's a bug, like in any other kernel type I'm not the one to convince :) eh, some would say it's one of the goals :) who's to be convinced then ? jkoenig: who raised the issue ah well, see the irc log :) not that I'm objecting to any patch, mind you :-) i didn't understand it that way if you can't trust the servers to act properly, it's similar to not trusting linux fs code no, the difference is that servers can be non-root while on linux they can't again, trust root and self non-root fuse mounts are not followed by default as with fuse that's still to be written yes and as I said, you can stat the actual node and then dereference the translator afterwards but before writing anything, we'd better agree on the solution :) which, again, "just" needs to be written err... adding a timeout to mach_msg()? that's just wrong (unless I completely misunderstood what this discussion was about...) #### IRC, freenode, #hurd, 2012-02-04 this is confirmed: the select hack patch hurts vim performance a lot I'll use program_invocation_short_name to make the patch even more ugly (of course, we really need to fix select somehow) could it (also) be that vim uses select() somehow "badly"? fsvo "badly", possibly, but still Could that the select() stuff be the reason for a ten times slower ethernet too, e.g. scp and apt-get? i didn't find myself neither scp nor apt-get slower, unlike vim see strace: scp does not use select (I haven't checked apt yet) ### IRC, freenode, #hurd, 2012-02-14 on another subject, I'm wondering how to correctly implement select/poll with a timeout on a multiserver system :/ i guess a timeout of 0 should imply a non blocking round-trip to servers only oh good, the timeout is already part of the io_select call ### IRC, freenode, #hurdfr, 2012-02-22 le gros souci de notre implé, c'est que le timeout de select est un paramètre client un paramètre passé directement à mach_msg donc si tu mets un timeout à 0, y a de fortes chances que mach_msg retourne avant même qu'un RPC puisse se faire entièrement (round-trip client-serveur donc) et donc quand le timeout est à 0 pour du non bloquant, ben tu bloques pas, mais t'as pas tes évènements .. peut-être que passer le timeout de 10ms à 10 us améliorerait la situation. car 10ms c'est un peut beaucoup :) c'est l'interval timer système historique unix et mach n'est pas préemptible donc c'est pas envisageable en l'état ceci dit c'est pas complètement lié enfin si, il nous faudrait qqchose de similaire aux high res timers de linux enfin soit des timer haute résolution, soit un timer programmable facilement actuellement il n'y a que le 8254 qui est programmé, et pour assurer un scheduling à peu près correct, il est programmé une fois, à 10ms, et basta donc oui, préciser 1ms ou 1us, ça changera rien à l'interval nécessaire pour déterminer que le timer a expiré ### IRC, freenode, #hurd, 2012-02-27 braunr: extremely dirty hack I don't even want to detail :) oh does it affect vim only ? or all select users ? we've mostly seen it with vim but possibly fakeroot has some issues too it's very little probable that only vim has the issue :) i mean, is it that dirty to switch behaviour depending on the calling program ? not all select users ew :) just those which do select({0,0}) well sure braunr: you guessed right :) thanks anyway it's probably a good thing to do currently vim was getting me so mad i was using sshfs lately it's better than nothing yes ### IRC, freenode, #hurd, 2012-07-21 damn, select is actually completely misdesigned :/ iiuc, it makes servers *block*, in turn :/ can't be right ok i understand it better yes, timeouts should be passed along with the other parameters to correctly implement non blocking select (or the round-trip io_select should only ask for notification requests instead of making a server thread block, but this would require even more work) adding the timeout in the io_select call should be easy enough for whoever wants to take over a not-too-complicated-but-not-one-liner-either task :) braunr: why is a blocking server thread a problem? antrik: handling the timeout at client side while server threads block is the problem the timeout must be handled along with blocking obviously so you either do it at server side when async ipc is available, which is the case here or request notifications (synchronously) and block at client side, waiting forthose notifications braunr: are you saying the client has a receive timeout, but when it elapses, the server thread keeps on blocking?... antrik: no i'm referring to the non-blocking select issue we have antrik: the client doesn't block in this case, whereas the servers do which obviously doesn't work .. see http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=79358 this is the reason why vim (and probably others) are slow on the hurd, while not consuming any cpu the current work around is that whenevever a non-blocking select is done, it's transformed into a blocking select with the smallest possible timeout whenever* braunr: well, note that the issue only began after fixing some other select issue... it was fine before apparently, the issue was raised in 2000 also, note that there is a delay between sending the io_select requests and blocking on the replies when machines were slow, this delay could almost guarantee a preemption between these steps, making the servers reply soon enough even for a non blocking select the problem occurs when sending all the requests and checking for replies is done before servers have a chance the send the reply braunr: I don't know what issue was raised in 2000, but I do know that vim worked perfectly fine until last year or so. then some select fix was introduced, which in turn broke vim antrik: could be the timeout rounding, Aug 2 2010 hum but, the problem wasn't with vim vim does still work fine (in fact, glibc is patched to check some well known process names and selectively fix the timeout) which is why vim is fast and view isn't the problem was with other services apparently and in order to fix them, that workaround had to be introduced i think it has nothing to do with the timeout rounding it must be the time when youpi added the patch to the debian package braunr: the problem is that with the patch changing the timeout rounding, vim got extremely slow. this is why the ugly hacky exception was added later... after reading the report, I agree that the timeout needs to be handled by the server. at least the timeout=0 case. vim uses often 0-time selects to check whether there's input client-side handling might still be OK for other timeout settings I guess I'm a bit ambivalent about that I tend to agree with Neal though: it really doesn't make much sense to have a client-side watchdog timer for this specific call, while for all other ones we trust the servers not to block... or perhaps not. for standard sync I/O, clients should expect that an operation could take long (though not forever); but they might use select() precisely to avoid long delays in I/O... so it makes some sense to make sure that select() really doesn't delay because of a busy server OTOH, unless the server is actually broken (in which anything could happen), a 0-time select should never actually block for an extended period of time... I guess it's not wrong to trust the servers on that pinotree: hm... that might explain a certain issue I *was* observing with Vim on Hurd -- though I never really thought about it being an actual bug, as opposed to just general Hurd sluggishness... but it makes sense now antrik: http://patch-tracker.debian.org/patch/series/view/eglibc/2.13-34/hurd-i386/local-select.diff so I guess we all agree that moving the select timeout to the server is probably the most reasonably approach... braunr: BTW, I wouldn't really consider the sync vs. async IPC cases any different. the client blocks waiting for the server to reply either way... the only difference is that in the sync IPC case, the server might want to take some special precaution so it doesn't have to block until the client is ready to receive the reply but that's optional and not really select-specific I'd say (I'd say the only sane approach with sync IPC is probably for the server never to wait -- if the client fails to set up for receiving the reply in time, it looses...) and with the receive buffer approach in Viengoos, this can be done really easy and nice :-) #### IRC, freenode, #hurd, 2012-07-22 antrik: you can't block in servers with sync ipc so in this case, "select" becomes a request for notifications whereas with async ipc, you can, so it's less efficient to make a full round trip just to ask for requests when you can just do async requests (doing the actual blocking) and wait for any reply after braunr: I don't understand. why can't you block in servers with async IPC? braunr: err... with sync IPC I mean antrik: because select operates on more than one fd braunr: and what does that got to do with sync vs. async IPC?... maybe you are thinking of endpoints here, which is a whole different story traditional L4 has IPC ports bound to specific threads; so implementing select requires a separate client thread for each server. but that's not mandatory for sync IPC. Viengoos has endpoints not bound to threads antrik: i don't know what "endpoint" means here but, you can't use sync IPC to implement select on multiple fds (and thus possibly multiple servers) by blocking in the servers you'd block in the first and completely miss the others braunr: I still don't see why... or why async IPC would change anything in that regard antrik: well, you call select on 3 fds, each implemented by different servers antrik: you call a sync select on the first fd, obviously you'll block there antrik: if it's async, you don't block, you just send the requests, and wait for any reply like we do braunr: I think you might be confused about the meaning of sync IPC. it doesn't in any way imply that after sending an RPC request you have to block on some particular reply... antrik: what does sync mean then? braunr: you can have any number of threads listening for replies from the various servers (if using an L4-like model); or even a single thread, if you have endpoints that can listen on replies from different sources (which was pretty much the central concern in the Viengoos IPC design AIUI) antrik: I agree with your "so it makes some sense to make sure that select() really doesn't delay because of a busy server" (for blocking select) and "OTOH, unless the server is actually broken (in which anything could happen), a 0-time select should never actually block" (for non-blocking select) youpi: regarding the select, I was thinking out loud; the former statement was mostly cancelled by my later conclusions... and I'm not sure the latter statement was quite clear do you know when it was? after rethinking it, I finally concluded that it's probably *not* a problem to rely on the server to observe the timout. if it's really busy, it might take longer than the designated timeout (especially if timeout is 0, hehe) -- but I don't think this is a problem and if it doens't observe the timout because it's broken/malicious, that's not more problematic that any other RPC the server doesn't handle as expected ok did somebody wrote down the conclusion "let's make select timeout handled at server side" somewhere? youpi: well, neal already said that in a followup to the select issue Debian bug... and after some consideration, I completely agree with his reasoning (as does braunr) #### IRC, freenode, #hurd, 2012-07-23 antrik: i was meaning sync in the most common meaning, yes, the client blocking on the reply braunr: I think you are confusing sync IPC with sync I/O ;-) braunr: by that definition, the vast majority of Hurd IPC would be sync... but that's obviously not the case synchronous IPC means that send and receive happen at the same time -- nothing more, nothing less. that's why it's called synchronous antrik: yes antrik: so it means the client can't continue unless he actually receives in a pure sync model such as L4 or EROS, this means either the sender or the receiver has to block, so synchronisation can happen. which one is server and which one is client is completely irrelevant here -- this is about individual message transfer, not any RPC model on top of it i the case of select, i assume sender == client in Viengoos, the IPC is synchronous in the sense that transfer from the send buffer to the receive buffer happens at the same time; but it's asynchronous in the sense that the receiver doesn't necessarily have to be actively waiting for the incoming message ok, i was talking about a pure sync model (though it most cases it will still do so...) braunr: BTW, in the case of select, the sender is *not* the client. the reply is relevant here, not the request -- so the client is the receiver (the select request is boring) sorry, i don't understand, you seem to dismiss the select request for no valid reason I still don't see how sync vs. async affects the select reply receive though... blocking seems the right approach in either case blocking is required but you either block in the servers, or in the client (and if blocking in the servers, the client also blocks) i'll explain how i see it again there are two approaches to implementing select 1/ send requests to all servers, wait for any reply, this is what the hurd does but it's possible because you can send all the requests without waiting for the replies 2/ send notification requests, wait for a notification this doesn't require blocking in the servers (so if you have many clients, you don't need as many threads) i was wondering which approach was used by the hurd, and if it made sense to change TBH I don't see the difference between 1) and 2)... whether the message from the server is called an RPC reply or a notification is just a matter of definition I think I see though what you are getting at with sync IPC, if the client sent all requests and only afterwards started to listen for replies, the servers might need to block while trying to deliver the reply because the client is not ready yet that's one thing yes but even in the sync case, the client can immediately wait for replies to each individual request -- it might just be more complicated, depending on the specifics of the IPC design what i mean by "send notification requests" is actually more than just sending, it's a complete RPC and notifications are non-blocking, yes (with L4, it would require a separate client thread for each server contacted... which is precisely why a different mechanism was designed for Viengoos) seems weird though don't they have a portset like abstraction ? braunr: well, having an immediate reply to the request and a separate notification later is just a waste of resources... the immediate reply would have no information value no, in original L4 IPC is always directed to specific threads antrik: some could see the waste of resource as being the duplication of the number of client threads in the server you could have one thread listening to replies from several servers -- but then, replies can get lost i see (or the servers have to block on the reply) so, there are really no capabilities in the original l4 design ? though I guess in the case of select() it wouldn't really matter if replies get lost, as long as at least one is handled... would just require the listener thread by separate from the thread sending the requests braunr: right. no capabilities of any kind that was my initial understanding too thanks so I partially agree: in a purely sync IPC design, it would be more complicated (but not impossible) to make sure the client gets the replies without the server having to block while sending replies arg, we need hurd_condition_timedwait (and possible condition_timedwait) to cleanly fix io_select luckily, i still have my old patch for condition_timedwait :> bddebian: in order to implement timeouts in select calls, servers now have to use a hurd_condition_timedwait function is it possible that a thread both gets canceled and timeout on a wait ? looks unlikely to me hm, i guess the same kind of compatibility constraints exist for hurd interfaces so, should we have an io_select1 ? braunr: I would use a more descriptive name: io_select_timeout() antrik: ah yes well, i don't really like the idea of having 2 interfaces for the same call :) because all select should be select_timeout :) but ok antrik: actually, having two select calls may be better oh it's really minor, we do'nt care actually braunr: two select calls? antrik: one with a timeout and one without the glibc would choose at runtime right. that was the idea. like with most transitions, that's probably the best option there is no need to pass the timeout value if it's not needed, and it's easier to pass NULL this way oh nah, that would make the transition more complicated I think ? ok :) this way, it becomes very easy the existing io_select call moves into a select_common() function the old variant doesn't know that the server has to return immediately; changing that would be tricky. better just use the new variant for the new behaviour, and deprecate the old one and the entry points just call this common function with either NULL or the given timeout no need to deprecate the old one that's what i'm saying and i don't understand "the old variant doesn't know that the server has to return immediately" won't the old variant block indefinitely in the server if there are no ready fds? yes it will oh, you mean using the old variant if there is no timeout value? yes well, I guess this would work well of course, the question is rather if we want this or not :) hm... not sure we need something to improve the process of changing our interfaces it's really painful currnelty inside the servers, we probably want to use common code anyways... so in the long run, I think it simplifies the code when we can just drop the old variant at some point a lot of the work we need to do involves changing interfaces, and we very often get to the point where we don't know how to do that and hardly agree on a final version : :/ ok but how do you tell the server you don't want a timeout ? a special value ? like { -1; -1 } ? hm... good point i'll do it that way for now it's the best way to test it which way you mean now? keeping io_select as it is, add io_select_timeout yeah, I thought we agreed on that part... the question is just whether io_select_timeout should also handle the no-timeout variant going forward, or keep io_select for that. I'm really not sure maybe I'll form an opinion over time :-) but right now I'm undecided i say we keep io_select anyway it won't change much we can just change that at the end if we decide otherwise right even passing special values is ok with a carefully written hurd_condition_timedwait, it's very easy to add the timeouts :) antrik, braunr: I'm wondering, another solution is to add an io_probe, i.e. the server has to return an immediate result, and the client then just waits for all results, without timeout that'd be a mere addition in the glibc select() call: when timeout is 0, use that, and otherwise use the previous code the good point is that it looks nicer in fs.defs are there bad points? (I don't have the whole issues in the mind now, so I'm probably missing things) youpi: the bad point is duplicating the implementation maybe what duplication ? ah you mean for the select case yes although it would be pretty much the same that is, if probe only, don't enter the wait loop could that be just some ifs here and there? (though not making the code easier to read...) hm i'm not sure it's fine in that case oi_select_timeout looks ncier ideed :) my problem with the current implementation is having the timeout at the client side whereas the server side is doing the blocking I wonder how expensive a notification is, compared to blocking a blocking indeed needs a thread stack (and kernel thread stuff) with the kind of async ipc we have, it's still better to do it that way and all the code already exists having the timeout at the client side also have its advantage has* latency is more precise so the real problem is indeed the non blocking case only isn't it bound to kernel ticks anyway ? uh, not if your server sucks or is loaded for whatever reason ok, that's not what I understood by "precision" :) I'd rather call it robustness :) hm right there are several ways to do this, but the io_select_timeout one looks fine to me and is already well on its way and it's reliable (whereas i'm not sure about reliability if we keep the timeout at client side) btw make the timeout nanoseconds ?? pselect uses timespec, not timeval do we want pselect ? err, that's the only safe way with signals not only, no and poll is timespec also not only?? you mean ppol ppoll no, poll too by "the only safe way", I mean for select calls i understand the race issue ppoll is a gnu extension int poll(struct pollfd *fds, nfds_t nfds, int timeout); ah, right, I was also looking at ppoll any way we can use nanosecs most event loops use a pipe or a socketpair there's no reason not to youpi: I briefly considered special-casisg 0 timeouts last time we discussed this; but I concluded that it's probably better to handle all timeouts server-side I don't see why we should even discuss that and translate signals to writes into the pipe/socketpair antrik: ok you can't count on select() timout precision anyways a few ms more shouldn't hurt any sanely written program braunr: "most" doesn't mean "all" there *are* applications which use pselect well mach only handles millisedonds seconds and it's not going out of the standard mach is not the hurd if we change mach, we can still keep the hurd ipcs anyway agagin I reallyt don't see the point of the discussion is there anything *against* using nanoseconds? i chose the types specifically because of that :p but ok i can change again becaus what?? i chose to use mach's native time_value_t because it matches timeval nicely but it doesn't match timespec nicely no it doesn't should i add a hurd specific time_spec_t then ? "how do you tell the server you don't want a timeout ? a special value ? like { -1; -1 } ?" you meant infinite blocking? youpi: yes oh right, pselect is posix actually posix says that there can be limitations on the maximum timeout supported, which should be at least 31 days