diff options
Diffstat (limited to 'open_issues/implementing_hurd_on_top_of_another_system.mdwn')
| -rw-r--r-- | open_issues/implementing_hurd_on_top_of_another_system.mdwn | 420 |
1 files changed, 420 insertions, 0 deletions
diff --git a/open_issues/implementing_hurd_on_top_of_another_system.mdwn b/open_issues/implementing_hurd_on_top_of_another_system.mdwn new file mode 100644 index 00000000..cf3db0ab --- /dev/null +++ b/open_issues/implementing_hurd_on_top_of_another_system.mdwn @@ -0,0 +1,420 @@ +[[!meta copyright="Copyright © 2010, 2011, 2012, 2013 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation]] + +It is possible to run Hurd stuff on top of another system instead of on Mach. +One obvious variant is [[emulation]] (using [[hurd/running/QEMU]], for +example), but +doing that does not really integratable the Hurd guest into the host system. +There is also a more direct way, more powerful, but it also has certain +requirements to do it effectively. + +See also [[Mach_on_top_of_POSIX]]. + + +# IRC, freenode, #hurd, August / September 2010 + + <marcusb> silver_hook: the Hurd can also refer to the interfaces of the + filesystems etc, and a lot of that is really just server/client APIs that + could be implemented on any system that has transferable rights to + message capabilities. + <marcusb> silver_hook: it's surprising how few systems *have* transferable + rights, though! + <marcusb> silver_hook: usually it is added as an afterthought + <marcusb> and comes with restriction + <youpi> marcusb: there's SCM_RIGHTS to transfer fds, which is quite often + available + <marcusb> youpi: yes, I know this as "fdpassing" + <marcusb> youpi: it's described in the Stevens series even + [...] + <marcusb> ArneBab: well, let me put it this way. the Linux kernel has no + interface to manipulate another tasks's virtual address space, ie you + can't map/unmap stuff in another process + <marcusb> ArneBab: you would have to use ptrace and load some stub code in + that process to make that happen. + <marcusb> ArneBab: so for complete transparent manipulation, you need a + kernel module + <marcusb> that is what the User Mode Linux kernel module does + <marcusb> ArneBab: so say you use the User Mode Linux kernel module for + that one feature. Then you can do everything that User Mode Linux can + do, which, I assure you, includes running subhurds :) + <marcusb> it can be a bit tricky to implement those features, but it is not + harder than writing a kernel in the first place + <ArneBab> So, if I got an admin to install User Mode Linux and Mach + emulation, I’d get the flexibility (and independence from admin + decisions) I have in the Hurd? + <marcusb> ArneBab: one problem is that you still use Linux. For those who + want to get rid of Linux for political reasons, that would mean complete + failure + <marcusb> ArneBab: if you have UML kernel module, you can implement Mach in + user space + <marcusb> ArneBab: in fact, John Tobey did this a couple of years ago, or + started it + +[[Mach_on_top_of_POSIX]]. + + <marcusb> ArneBab: or you can just implement parts of it and relay to Linux + for the rest + <marcusb> the point is, that if you don't care for kernel improvements, and + are sufficiently happy with the translator stuff, it's not hard to bring + the Hurd to Linux or BSD + +Continue reading about the [[benefits_of_a_native_Hurd_implementation]]. + + +# IRC, freenode, #hurd, 2010-12-28 + + <antrik> kilobug: there is no real requirement for the Hurd to run on a + microkernel... as long as the important mechanisms are provided (most + notably external pagers and Mach IPC), the Hurd could run an top of + pretty much any kernel... + <antrik> whether it makes sense is another question of course :-) + <antrik> though I must say that I'm more and more convinced running the + Hurd on top of a monolithic kernel would actually be a useful approach + for the time being... + + +# IRC, freenode, #hurd, 2011-02-11 + + <neal> marcus and I were discussing how to add Mach to Linux + <neal> one could write a module to implement Mach IPC + <neal> and another to implement Mach VM + <neal> the big thing missing with Mach VM is the ability for a tracing + process to easily map or unmap an inferior process's memory + <antrik> neal: why would a tracing process need to map the inferior's + memory? + <neal> the simple answer is that is how it is done on Mach + <antrik> neal: is it? not sure we are talking about the same thing + here. GDB uses vm_read()/vm_write() to access the inferior's memory AFAIK + <neal> on linux? + <neal> I think it use /proc/pid/mem + <antrik> on Hurd + <neal> I'm talking about adding Mach to Linux + <neal> by adding some functionality to Linux + <neal> and then implementing a bunch in user space + <antrik> yeah, but I don't understand the point about mapping inferior's + memory :-( + <antrik> what would be in user space? + <neal> there are a number of different cut points + <neal> one could imagine just using Linux's device drivers, CPU scheduler, + memory management, etc. + <neal> another possibility would be something higher where Hurd processes + just use some Hurdish servers + <antrik> neal: yeah, these are all options I have been considering... too + bad I wasn't able to come to FOSDEM -- I'd love to have participated in + this discussion :-( + <antrik> neal: BTW, were you just discussing this as a hypothetical idea, + or something you are seriously considering? + <neal> I'm unlikely to work on it, sorry + <antrik> didn't really expect that :-) + <antrik> would be nice though if you could write up your conclusions... + + +# IRC, freenode, #hurd, 2012-10-12 + + <peo-xaci> do hurd system libraries make raw system calls ever + (i.e. inlined syscall() / raw assembly)? + <braunr> sure + <peo-xaci> hmm, so a hurd emulation layer would need to use ptrace if it + should be fool proof? :/ + <braunr> there is no real need for raw assembly, and the very syscalls are + all available through macros + <braunr> hum what are you trying to say ? + <peo-xaci> well, if they are done through syscall, as a function, not a + macro, then they can be intercepted with LD_PRELOAD + <peo-xaci> so applications that do Hurd (Mach?) syscalls could work on + f.e. Linux, if a special libc is injected into the program with + LD_PRELOAD + <peo-xaci> same thing with making standard Linux-applications go through + the Hurd emulation layer + <peo-xaci> without recompilation + <mel-_> peo-xaci: the second direction is implemented in glibc. + <mel-_> for the other direction, I personally see little use for it + <braunr> peo-xaci: ok i misunderstood + <braunr> peo-xaci: i don't think there is any truely direct syscall usage + in the hurd + <peo-xaci> hmm, I'm not sure I understand what directions you are referring + to mel-_ + <braunr> peo-xaci: what are you trying to achieve ? + <peo-xaci> I want to make the Hurd design more accessible by letting Hurd + application run on the Linux kernel, preferably without + recompilation. This would be done with a daemon that implements Mach and + which all syscalls would go to. + <peo-xaci> then, I also want so that standard Linux applications can go + through that Mach daemon as well, if a special libc is preloaded + <braunr> you might want to discuss this with antrik + <peo-xaci> what I'm trying to figure out specifically is if there is some + library/interface that glue Hurd with Mach and would be better suited to + emulate than Mach? Mach seems to be more of an implementation detail to + the hurd and not something an application would directly use. + <braunr> yes, the various hurd libraries (libports and libpager mostly) + <peo-xaci> From [http://www.gnu.org/software/hurd/hurd/libports.html]: + "libports is not (at least, not for now) a generalization / abstraction + of Mach ports to the functionality the Hurd needs, that is, it is not + meant to provide an interface independently of the underlying + microkernel." + <peo-xaci> Is this still true? + <peo-xaci> Does libpager abstract the rest? + <peo-xaci> (and the other hurd libraries) + <braunr> there is nothing that really abstracts the hurd from mach + <braunr> for example, reference counting often happens here and there + <braunr> and core libraries like glibc and libpthread heavily rely on it + (through sysdeps specific code though) + <braunr> libports and libpager are meant to simplify object manipulation + for the former, and pager operations for the latter + <peo-xaci> and applications, such as translators, often use Mach interfaces + directly? + <peo-xaci> correct? + <braunr> depends on what often means + <braunr> let's say they do + <peo-xaci> :/ then it probably is better to emulate Mach after all + <braunr> there was a mach on posix port a long time ago + <peo-xaci> I thought applications were completely separated from the + microkernel in use by the Hurd + <braunr> that level of abstraction is pretty new + <braunr> genode is the only system i know which does that + +[[microkernel/Genode]]. + + <braunr> and it's still for "l4 variants" + <pinotree> ah, thanks (i forgot that name) + <antrik> braunr: Genode also runs on Linux and a few other non-L4 + environments IIRC + <antrik> peo-xaci: I'm not sure binary emulation is really useful. rather, + I'd recompile stuff as "regular" Linux executables, only using a special + glibc + <antrik> where the special glibc could be basically a port of the Hurd + glibc communicating with the Mach emulation instead of real Mach; or it + could do emulation at a higher level + <antrik> a higher level emulation would be more complicated to implement, + but more efficient, and allow better integration with the ordinary + GNU/Linux environment + <antrik> also note that any regular program could be recompiled against the + HELL glibc to run in the Hurdish environment... + <antrik> (well, glibc + hurd server libraries) + <peo-xaci> I'm willing to accept that Hurd-application would need to be + recompiled to work on the HELL + <peo-xaci> but not Linux-applications :) + <antrik> peo-xaci: if you happen to understand German, there is a fairly + good overview in my thesis report ;-) + <antrik> peo-xaci: there are no "Hurd applications" or "Linux applications" + <peo-xaci> well, let me define what I mean by the terms: Hurd applications + use Hurd-specific interfaces/syscalls, and Linux applications use + Linux-specific interfaces/syscalls + <antrik> a few programs use Linux-specific interfaces (and we probably + can't run them in HELL just as we can't run them on actual Hurd); but all + other programs work in any glibc environment + <antrik> (usually in any POSIX environment in fact...) + <antrik> peo-xaci: no sane application uses syscalls + <peo-xaci> they do under the hood + <peo-xaci> I have read about inlined syscalls + <antrik> again, there are *some* applications using Linux-specific + interfaces (sometimes because they are inherently bound to Linux + features, sometimes unnecessarily) + <antrik> so far there are no applications using Hurd-specific interfaces + <peo-xaci> translators do? + <peo-xaci> they are standard executables are they not? + <peo-xaci> I would like so that translators also can be run in the HELL + <antrik> I wouldn't consider them applications. all existing translators + are pretty much components of the Hurd itself + <peo-xaci> okay, it's a question about semantics, perhaps I should use + another word than "applications" :) + <peo-xaci> for me, applications are what have a main-function, or similar + single entry point + <braunr> hum + <braunr> that's not a good enough definition + <antrik> anyways, as I said, I think recompiling translators against a + Hurdish glibc and ported translator libraries seems the most reasonable + approach to me + <braunr> let's say applications are userspace processes that make use of + services provided by the operating system + <braunr> translators being part of the operating system here + <antrik> braunr: do you know whether the Mach-on-POSIX was actually + functional, or just an abandoned experiment?... + <antrik> (I don't remember hearing of it before...) + <braunr> incomplete iirc + <peo-xaci> braunr: still, when I've explained what I meant, even if I used + the wrong term, then my previous statements should come in another light + <peo-xaci> antrik / braunr: are you still interested in hearing my + thoughts/ideas about HELL? + <antrik> oh, there is more to come? ;-) + <peo-xaci> yes! I don't think I have made myself completely understood :/ + <peo-xaci> what I envision is a HELL system that works on as low level as + feasible, to make it possible to do almost anything that can be done on + the real Hurd (except possibly testing hardware drivers and such very low + level stuff). + <braunr> sure + <peo-xaci> I want it to be more than just allowing programs to access a + virtual filesystem à la FUSE. My idea is that all user space system + libraries/programs of the Hurd should be inside the HELL as well, and + they should not be emulated. + <peo-xaci> The system should at the very least be API compatible, so at the + very most a recompilation is necessary. + <peo-xaci> I also want so that GNU/Linux-programs can access the features + of the HELL with little effort on the user. At most perhaps a script that + wraps LD_PRELOADing has to be run on the binary. Best would be if it + could work also with insane assembly programs using raw system calls, or + if glibc happens to have some well hidden syscall being inlined to raw + assembly code. + <peo-xaci> And I think I have an idea on how an implementation could + satisfy these things! + <peo-xaci> By modifying the kernel and replace those syscalls that make + sense for the Hurd/Mach + <peo-xaci> with "the kernel", I meant Linux + <braunr> it's possible but tedious and not very useful so better do that + later + <braunr> mach did something similar at its time + <braunr> there was a syscall emulation library + <peo-xaci> but isn't it about as much work as emulating the interface on + user-level? + <braunr> and the kernel cooperated so that unmodified unix binaries + performing syscalls would actually jump to functions provided by that + library, which generally made an RPC + <peo-xaci> instead of a bunch of extern-declerations, one would put the + symbols in the syscall table + <braunr> define what "those syscalls that make sense for the Hurd/Mach" + actually means + <peo-xaci> open/close, for example + <braunr> otherwise i don't see another better way than what the old mach + folks did + <braunr> well, with that old, but existing support, your open would perform + a syscall + <braunr> the kernel would catch it and redirect the caller to its syscall + emulation library + <braunr> which would call the open RPC instead + <peo-xaci> wait, so this "existing support" you're talking about; is this a + module for the Linux kernel (or a fork, or something else)? + <peo-xaci> where can I find it? + <braunr> no + <braunr> it was for mach + <braunr> in order to run unmodified unix binaries + <braunr> the opposite of what you're trying to do + <peo-xaci> ah okay + <braunr> well + <braunr> not really either :) + <peo-xaci> does posix/unix define a standard for how a syscall table should + look like, to allow binary syscall compatibility? + <braunr> absolutely not + <peo-xaci> so how could this mach module run any unmodified unix binary? if + they expected different sys calls at different offsets? + <braunr> posix specifically (and very early) states that it almost forbids + itself to deal with anything regarding to ABIs + <braunr> depends + <braunr> since it was old, there weren't that many unix systems + <braunr> and even today, there are techniques like those used by netbsd + (and many other actually) + <braunr> that are able to inspect the binary and load a syscall emulation + environment depending on its exposed ABI + <braunr> e.g. file on an executable states which system it's for + <peo-xaci> hmm, I'm not sure how a kernel would implement that in + practice.. I thought these things were so hard coded and dependent on raw + memory reads that it would not be possible + <braunr> but i really think it's not worth the time for your project + <peo-xaci> to be honest I have virtually no experience of practical kernel + programming + <braunr> with an LDT on x86 for example + <braunr> no, there is really not that much hardcoded + <braunr> quite the contrary + <braunr> there is a lot of runtime detection today + <peo-xaci> well I mean how the syscall table is read + <braunr> it's not read + <peo-xaci> it's read to find the function pointer to the syscall handler in + the kernel? + <braunr> no + <braunr> that's the really basic approach + <braunr> (and in practice it can happen of course) + <braunr> what really happens is that, for example, on linux, the user space + system call code is loaded as a virtual shared library + <braunr> use ldd on an executable to see it + <braunr> this virtual object provides code that, depending on what the + kernel has detected, will use the appropriate method to perform a system + call + <peo-xaci> but this user space system calls need to make some kind of cpu + interupt to communicate with the kernel, right? + <braunr> the glibc itself has no idea how a system call will look like in + the end + <braunr> yes + <peo-xaci> an assembler programmer would be able to get around this glue + code? + <braunr> that's precisely what is embedded in this virtual library + <braunr> it could yes + <braunr> i think even when sysenter/sysexit is supported, legacy traps are + still implemented to support old binaries + <braunr> but then all these different entry points will lead to the same + code inside the kernel + <peo-xaci> but when the glue code is used, then its API compatible, and + then I can understand that the kernel can allow different syscall + implementations for different executables + <braunr> what glue code ? + <peo-xaci> what you talked about above "the user space system call code is + loaded as a virtual shared library" + <braunr> let's call it vdso + +[[vDSO]] + + <peo-xaci> thanks, I looked it up on Wikipedia and understand immediately + :P + <peo-xaci> so VDSOs are provided by the kernel, not a regular library file, + right? + <vdox2> What does HELL stand for :) ? + <dardevelin> vdox2, Hurd Emulation Layer for Linux + <vdox2> dardevelin: thanks + <braunr> peo-xaci: yes + <antrik> peo-xaci: I believe your goals are conflicting. a low-level + implementation makes it basically impossible to interact between the HELL + environment and the GNU/Linux environment in any meaningful way. to allow + such interaction, you *have* to have some glue at a higher semantic level + <braunr> agreed + <antrik> peo-xaci: BTW, if you want regular Linux binaries to get somehow + redirected to access HELL facilities, there is already a framework (don't + remember the name right now) that allows this kind of system call + redirection on Linux + <antrik> (it can run both through LD_PRELOAD or as a kernel module -- where + obviously only the latter would allow raw system call redirection... but + TBH, I don't think that's worthwhile anyways. the rare cases where + programs use raw system calls are usually for extremely system-specific + stuff anyways...) + <antrik> ViewOS is the name + <antrik> err... View-OS I mean + <antrik> or maybe View OS ? ;-) + <antrik> whatever, you'll find it :-) + +[[Virtual_Square_View-OS]]. + + <antrik> I'm not sure it's really worthwhile to use this either + though... the most meaningful interaction is probably at the FS level, + and that can be done with FUSE + <antrik> OHOH, View-OS probably allows doing more interesting stuff that + FUSE, such as modyfing the way the VFS works... + <antrik> OTOH + <antrik> so it could expose more of the Hurd features, at least in theory + + +## IRC, freenode, #hurd, 2012-10-13 + + <peo-xaci> antrik / braunr: thanks for your input! I'm not entirely + convinced though. :) I will probably return to this project once I have + acquired a lot more knowledge about low level stuff. I want to see for + myself whether a low level HELL is not feasible. :P + <braunr> peo-xaci: what's the point of a low level hell ? + <peo-xaci> more Hurd code can be tested in the hell, if the hell is at a + low level + <peo-xaci> at a higher level, some Hurd code cannot run, because the + interfaces they use would not be accessible from the higher level + emulation + <antrik> peo-xaci: I never said it's not possible. I actually said it would + be easier to do. I just said you can't do it low level *and* have + meaningful interaction with the host system + <peo-xaci> I don't understand why + <braunr> peo-xaci: i really don't see what you want to achieve with low + level support + <braunr> what would be unavailable with a higher level approach ? |