From 2603401fa1f899a8ff60ec6a134d5bd511073a9d Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Tue, 7 Aug 2012 23:25:26 +0200 Subject: IRC. --- open_issues/libpthread.mdwn | 524 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 524 insertions(+) (limited to 'open_issues/libpthread.mdwn') diff --git a/open_issues/libpthread.mdwn b/open_issues/libpthread.mdwn index c5054b7f..03a52218 100644 --- a/open_issues/libpthread.mdwn +++ b/open_issues/libpthread.mdwn @@ -42,3 +42,527 @@ There is a [[!FF_project 275]][[!tag bounty]] on this task. there'll still be the issue that only one will be initialized and one that provides libc thread safety functions, etc. that's what i wanted to knew, thanks :) + + +## IRC, freenode, #hurd, 2012-07-23 + + So I am not sure what to do with the hurd_condition_wait stuff + i would also like to know what's the real issue with cancellation + here + because my understanding is that libpthread already implements it + does it look ok to you to make hurd_condition_timedwait return an + errno code (like ETIMEDOUT and ECANCELED) ? + braunr: that's what pthread_* function usually do, yes + i thought they used their own code + no + thanks + well, first, do you understand what hurd_condition_wait is ? + it's similar to condition_wait or pthread_cond_wait with a subtle + difference + it differs from the original cthreads version by handling + cancellation + but it also differs from the second by how it handles cancellation + instead of calling registered cleanup routines and leaving, it + returns an error code + (well simply !0 in this case) + so there are two ways + first, change the call to pthread_cond_wait + Are you saying we could fix stuff to use pthread_cond_wait() + properly? + it's possible but not easy + because you'd have to rewrite the cancellation code + probably writing cleanup routines + this can be hard and error prone + and is useless if the code already exists + so it seems reasonable to keep this hurd extension + but now, as it *is* a hurd extension noone else uses + braunr: BTW, when trying to figure out a tricky problem with the + auth server, cfhammer digged into the RPC cancellation code quite a bit, + and it's really a horrible complex monstrosity... plus the whole concept + is actually broken in some regards I think -- though I don't remember the + details + antrik: i had the same kind of thoughts + antrik: the hurd or pthreads ones ? + not sure what you mean. I mean the RPC cancellation code -- which + is involves thread management too + ok + I don't know how it is related to hurd_condition_wait though + well i found two main entry points there + hurd_thread_cancel and hurd_condition_wait + and it didn't look that bad + whereas in the pthreads code, there are many corner cases + and even the standard itself looks insane + well, perhaps the threading part is not that bad... + it's not where we saw the problems at any rate :-) + rpc interruption maybe ? + oh, right... interruption is probably the right term + yes that thing looks scary + :)) + the migration thread paper mentions some things about the problems + concerning threads controllability + I believe it's a very strong example for why building around + standard Mach features is a bad idea, instead of adapting the primitives + to our actual needs... + i wouldn't be surprised if the "monstrosities" are work arounds + right + + +## IRC, freenode, #hurd, 2012-07-26 + + Uhm, where does /usr/include/hurd/signal.h come from? + head -n4 /usr/include/hurd/signal. + h + Ohh glibc? + That makes things a little more difficult :( + why ? + Hurd includes it which brings in cthreads + ? + the hurd already brings in cthreads + i don't see what you mean + Not anymore :) + the system cthreads header ? + well it's not that difficult to trick the compiler not to include + them + signal.h includes cthreads.h I need to stop that + just define the _CTHREADS_ macro before including anything + remember that header files are normally enclosed in such macros to + avoid multiple inclusions + this isn't specific to cthreads + converting hurd from cthreads to pthreads will make hurd and + glibc break source and binary compatibility + Of course + reminds me of the similar issues of the late 90s + Ugh, why is he using _pthread_self()? + maybe because it accesses to the internals + "he" ? + Thomas in his modified cancel-cond.c + well, you need the internals to implement it + hurd_condition_wait is similar to pthread_condition_wait, except + that instead of stopping the thread and calling cleanup routines, it + returns 1 if cancelled + not that i looked at it, but there's really no way to implement + it using public api? + Even if I am using glibc pthreads? + unlikely + God I had all of this worked out before I dropped off for a + couple years.. :( + this will come back :p + that makes you the perfect guy to work on it ;) + I can't find a pt-internal.h anywhere.. :( + clone the hurd/libpthread.git repo from savannah + Of course when I was doing this libpthread was still in hurd + sources... + So if I am using glibc pthread, why can't I use pthread_self() + instead? + that won't give you access to the internals + OK, dumb question time. What internals? + the libpthread ones + that's where you will find if your thread has been cancelled or + not + pinotree: But isn't that assuming that I am using hurd's + libpthread? + if you aren't inside libpthread, no + pthread_self is normally not portable + you can only use it with pthread_equal + so unless you *know* the internals, you can't use it + and you won't be able to do much + so, as it was done with cthreads, hurd_condition_wait should be + close to the libpthread implementation + inside, normally + now, if it's too long for you (i assume you don't want to build + glibc) + you can just implement it outside, grabbing the internal headers + for now + another "not that i looked at it" question: isn't there no way + to rewrite the code using that custom condwait stuff to use the standard + libpthread one? + and once it works, it'll get integrated + pinotree: it looks very hard + braunr: But the internal headers are assuming hurd libpthread + which isn't in the source anymore + from what i could see while working on select, servers very often + call hurd_condition_wait + and they return EINTR if canceleld + so if you use the standard pthread_cond_wait function, your thread + won't be able to return anything, unless you push the reply in a + completely separate callback + i'm not sure how well mig can cope with that + i'd say it can't :) + no really it looks ugly + it's far better to have this hurd specific function and keep the + existing user code as it is + bddebian: you don't need the implementation, only the headers + the thread, cond, mutex structures mostly + I should turn to "pt-internal.h" and just put it + in libshouldbelibc, no? + no, that header is not installed + Obviously not the "best" way + pinotree: ?? + pinotree: what does it change ? + braunr: it == ? + bddebian: you could even copy it entirely in your new + cancel-cond.C and mention where it was copied from + pinotree: it == pt-internal.H not being installed + that he cannot include it in libshouldbelibc sources? + ah, he wants to copy it? + yes + i want him to copy it actually :p + it may be hard if there are a lot of macro options + the __pthread struct changes size and content depending on other + internal sysdeps headers + well he needs to copy those too :p + Well even if this works we are going to have to do something + more "correct" about hurd_condition_wait. Maybe even putting it in + glibc? + sure + but again, don't waste time on this for now + make it *work*, then it'll get integrated + Like it has already? This "patch" is only about 5 years old + now... ;-P + but is it complete ? + Probably not :) + Hmm, I wonder how many undefined references I am going to get + though.. :( + Shit, 5 + One of which is ___pthread_self.. :( + Does that mean I am actually going to have to build hurds + libpthreads in libshouldbeinlibc? + Seriously, do I really need ___pthread_self, __pthread_self, + _pthread_self and pthread_self??? + I'm still unclear what to do with cancel-cond.c. It seems to me + that if I leave it the way it is currently I am going to have to either + re-add libpthreads or still all of the libpthreads code under + libshouldbeinlibc. + then add it in libc + glib + glibc + maybe under the name __hurd_condition_wait + Shouldn't I be able to interrupt cancel-cond stuff to use glibc + pthreads? + interrupt ? + Meaning interject like they are doing. I may be missing the + point but they are just obfuscating libpthreads thread with some other + "namespace"? (I know my terminology is wrong, sorry). + they ? + Well Thomas in this case but even in the old cthreads code, + whoever wrote cancel-cond.c + but they use internal thread structures .. + Understood but at some level they are still just getting to a + libpthread thread, no? + absolutely not .. + there is *no* pthread stuff in the hurd + that's the problem :p + Bah damnit... + cthreads are directly implement on top of mach threads + implemeneted* + implemented* + Sure but hurd_condition_wait wasn't + of course it is + it's almost the same as condition_wait + but returns 1 if a cancelation request was made + Grr, maybe I am just confusing myself because I am looking at + the modified (pthreads) version instead of the original cthreads version + of cancel-cond.c + well if the modified version is fine, why not directly use that ? + normally, hurd_condition_wait should sit next to other pthread + internal stuff + it could be renamed __hurd_condition_wait, i'm not sure + that's irrelevant for your work anyway + I am using it but it relies on libpthread and I am trying to use + glibc pthreads + hum + what's the difference between libpthread and "glibc pthreads" ? + aren't glibc pthreads the merged libpthread ? + quite possibly but then I am missing something obvious. I'm + getting ___pthread_self in libshouldbeinlibc but it is *UND* + bddebian: with unmodified binaries ? + braunr: No I added cancel-cond.c to libshouldbeinlibc + And some of the pt-xxx.h headers + well it's normal then + i suppose + braunr: So how do I get those defined without including + pthreads.c from libpthreads? :) + pinotree: hm... I think we should try to make sure glibc works + both whith cthreads hurd and pthreads hurd. I hope that shoudn't be so + hard. + breaking binary compatibility for the Hurd libs is not too + terrible I'd say -- as much as I'd like that, we do not exactly have a + lot of external stuff depending on them :-) + bddebian: *sigh* + bddebian: just add cancel-cond to glibc, near the pthread code :p + braunr: Wouldn't I still have the same issue? + bddebian: what issue ? + is hurd_condition_wait() the name of the original cthreads-based + function? + antrik: the original is condition_wait + I'm confused + is condition_wait() a standard cthreads function, or a + Hurd-specific extension? + antrik: as standard as you can get for something like cthreads + braunr: Where hurd_condition_wait is looking for "internals" as + you call them. I.E. there is no __pthread_self() in glibc pthreads :) + hurd_condition_wait is the hurd-specific addition for cancelation + bddebian: who cares ? + bddebian: there is a pthread structure, and conditions, and + mutexes + you need those definitions + so you either import them in the hurd + braunr: so hurd_condition_wait() *is* also used in the original + cthread-based implementation? + or you write your code directly where they're available + antrik: what do you call "original" ? + not transitioned to pthreads + ok, let's simply call that cthreads + yes, it's used by every hurd servers + virtually + if not really everyone of them + braunr: That is where you are losing me. If I can just use + glibc pthreads structures, why can't I just use them in the new pthreads + version of cancel-cond.c which is what I was originally asking.. :) + you *have* to do that + but then, you have to build the whole glibc + * bddebian shoots himself + and i was under the impression you wanted to avoid that + do any standard pthread functions use identical names to any + standard cthread functions? + what you *can't* do is use the standard pthreads interface + no, not identical + but very close + bddebian: there is a difference between using pthreads, which + means using the standard posix interface, and using the glibc pthreads + structure, which means toying with the internale implementation + you *cannot* implement hurd_condition_wait with the standard posix + interface, you need to use the internal structures + hurd_condition_wait is actually a shurd specific addition to the + threading library + hurd* + well, in that case, the new pthread-based variant of + hurd_condition_wait() should also use a different name from the + cthread-based one + so it's normal to put it in that threading library, like it was + done for cthreads + 21:35 < braunr> it could be renamed __hurd_condition_wait, i'm not + sure + Except that I am trying to avoid using that threading library + what ? + If I am understanding you correctly it is an extention to the + hurd specific libpthreads? + to the threading library, whichever it is + antrik: although, why not keeping the same name ? + braunr: I don't think having hurd_condition_wait() for the cthread + variant and __hurd_condition_wait() would exactly help clarity... + I was talking about a really new name. something like + pthread_hurd_condition_wait() or so + braunr: to avoid confusion. to avoid accidentally pulling in the + wrong one at build and/or runtime. + to avoid possible namespace conflicts + ok + well yes, makes sense + braunr: Let me state this as plainly as I hope I can. If I want + to use glibc's pthreads, I have no choice but to add it to glibc? + and pthread_hurd_condition_wait is a fine name + bddebian: no + bddebian: you either add it there + bddebian: or you copy the headers defining the internal structures + somewhere else and implement it there + but adding it to glibc is better + it's just longer in the beginning, and now i'm working on it, i'm + really not sure + add it to glibc directly :p + That's what I am trying to do but the headers use pthread + specific stuff would should be coming from glibc's pthreads + yes + well it's not the headers you need + you need the internal structure definitions + sometimes they're in c files for opacity + So ___pthread_self() should eventually be an obfuscation of + glibcs pthread_self(), no? + i don't know what it is + read the cthreads variant of hurd_condition_wait, understand it, + do the same for pthreads + it's easy :p + For you bastards that have a clue!! ;-P + I definitely vote for adding it to the hurd pthreads + implementation in glibc right away. trying to do it externally only adds + unnecessary complications + and we seem to agree that this new pthread function should be + named pthread_hurd_condition_wait(), not just hurd_condition_wait() :-) + + +## IRC, freenode, #hurd, 2012-07-27 + + OK this hurd_condition_wait stuff is getting ridiculous the way + I am trying to tackle it. :( I think I need a new tactic. + bddebian: what do you mean ? + braunr: I know I am thick headed but I still don't get why I + cannot implement it in libshouldbeinlibc for now but still use glibc + pthreads internals + I thought I was getting close last night by bringing in all of + the hurd pthread headers and .c files but it just keeps getting uglier + and uglier + youpi: Just to verify. The /usr/lib/i386-gnu/libpthread.so that + ships with Debian now is from glibc, NOT libpthreads from Hurd right? + Everything I need should be available in glibc's libpthreads? (Except for + hurd_condition_wait obviously). + 22:35 < antrik> I definitely vote for adding it to the hurd + pthreads implementation in glibc right away. trying to do it externally + only adds unnecessary complications + bddebian: yes + same as antrik + fuck + libpthread *already* provides some odd symbols (cthread + compatibility), it can provide others + bddebian: don't curse :p it will be easier in the long run + * bddebian breaks out glibc :( + but you should tell thomas that too + braunr: I know it just adds a level of complexity that I may not + be able to deal with + we wouldn't want him to waste too much time on the external + libpthread + which one ? + glibc for one. hurd_condition_wait() for another which I don't + have a great grasp on. Remember my knowledge/skillsets are limited + currently. + bddebian: tschwinge has good instructions to build glibc + keep your tree around and it shouldn't be long to hack on it + for hurd_condition_wait, i can help + Oh I was thinking about using Debian glibc for now. You think I + should do it from git? + no + debian rules are even more reliable + (just don't build all the variants) + `debian/rules build_libc` builds the plain i386 variant only + So put pthread_hurd_cond_wait in it's own .c file or just put it + in pt-cond-wait.c ? + i'd put it in pt-cond-wait.C + youpi or braunr: OK, another dumb question. What (if anything) + should I do about hurd/hurd/signal.h. Should I stop it from including + cthreads? + it's not a dumb question. it should probably stop, yes, but there + might be uncovered issues, which we'll have to take care of + Well I know antrik suggested trying to keep compatibility but I + don't see how you would do that + compability between what ? + and source and/or binary ? + hurd/signal.h implicitly including cthreads.h + ah + well yes, it has to change obviously + Which will break all the cthreads stuff of course + So are we agreeing on pthread_hurd_cond_wait()? + that's fine + Ugh, shit there is stuff in glibc using cthreads?? + like what ? + hurdsig, hurdsock, setauth, dtable, ... + it's just using the compatibility stuff, that pthread does provide + but it includes cthreads.h implicitly + s/it/they in many cases + not a problem, we provide the functions + Hmm, then what do I do about signal.h? It includes chtreads.h + because it uses extern struct mutex ... + ah, then keep the include + the pthread mutexes are compatible with that + we'll clean that afterwards + arf, OK + that's what I meant by "uncover issues" + + +## IRC, freenode, #hurd, 2012-07-28 + + Well crap, glibc built but I have no symbol for + pthread_hurd_cond_wait in libpthread.so :( + Hmm, I wonder if I have to add pthread_hurd_cond_wait to + forward.c and Versions? (Versions obviously eventually) + bddebian: most probably not about forward.c, but definitely you + have to export public stuff using Versions + + +## IRC, freenode, #hurd, 2012-07-29 + + braunr: http://paste.debian.net/181078/ + ugh, inline functions :/ + "Tell hurd_thread_cancel how to unblock us" + i think you need that one too :p + ?? + well, they work in pair + one cancels, the other notices it + hurd_thread_cancel is in the hurd though, iirc + or uh wait + no it's in glibc, hurd/thread-cancel.c + otherwise it looks like a correct reuse of the original code, but + i need to understand the pthreads internals better to really say anything + + +## IRC, freenode, #hurd, 2012-08-03 + + pinotree: what do you think of + condition_implies/condition_unimplies ? + the work on pthread will have to replace those + + +## IRC, freenode, #hurd, 2012-08-06 + + bddebian: so, where is the work being done ? + braunr: Right now I would just like to testing getting my glibc + with pthread_hurd_cond_wait installed on the clubber subhurd. It is in + /home/bdefreese/glibc-debian2 + we need a git branch + braunr: Then I want to rebuild hurd with Thomas's pthread + patches against that new libc + Aye + i don't remember, did thomas set a git repository somewhere for + that ? + He has one but I didn't have much luck with it since he is using + an external libpthreads + i can manage the branches + I was actually patching debian/hurd then adding his patches on + top of that. It is in /home/bdefreese/debian-hurd but he has updateds + some stuff since then + Well we need to agree on a strategy. libpthreads only exists in + debian/glibc + it would be better to have something upstream than to work on a + debian specific branch :/ + tschwinge: do you think it can be done + ? + + +## IRC, freenode, #hurd, 2012-08-07 + + braunr: You mean to create on Savannah branches for the + libpthread conversion? Sure -- that's what I have been suggesting to + Barry and Thomas D. all the time. + + braunr: OK, so I installed my glibc with + pthread_hurd_condition_wait in the subhurd and now I have built Debian + Hurd with Thomas D's pthread patches. + bddebian: i'm not sure we're ready for tests yet :p + braunr: Why not? :) + bddebian: a few important bits are missing + braunr: Like? + like condition_implies + i'm not sure they have been handled everywhere + it's still interesting to try, but i bet your system won't finish + booting + Well I haven't "installed" the built hurd yet + I was trying to think of a way to test a little bit first, like + maybe ext2fs.static or something + Ohh, it actually mounted the partition + How would I actually "test" it? + git clone :p + building a debian package inside + removing the whole content after + that sort of things + Hmm, I think I killed clubber :( + Yep.. Crap! :( + ? + how did you do that ? + Mounted a new partition with the pthreads ext2fs.static then did + an apt-get source hurd to it.. + what partition, and what mount point ? + I added a new 2Gb partition on /dev/hd0s6 and set the translator + on /home/bdefreese/part6 + shouldn't kill your hurd + Well it might still be up but killed my ssh session at the very + least :) + ouch + braunr: Do you have debugging enabled in that custom kernel you + installed? Apparently it is sitting at the debug prompt. -- cgit v1.2.3