From 49a086299e047b18280457b654790ef4a2e5abfa Mon Sep 17 00:00:00 2001 From: Samuel Thibault Date: Wed, 18 Feb 2015 00:58:35 +0100 Subject: Revert "rename open_issues.mdwn to service_solahart_jakarta_selatan__082122541663.mdwn" This reverts commit 95878586ec7611791f4001a4ee17abf943fae3c1. --- open_issues/anatomy_of_a_hurd_system.mdwn | 1430 +++++++++++++++++++++++++++++ 1 file changed, 1430 insertions(+) create mode 100644 open_issues/anatomy_of_a_hurd_system.mdwn (limited to 'open_issues/anatomy_of_a_hurd_system.mdwn') diff --git a/open_issues/anatomy_of_a_hurd_system.mdwn b/open_issues/anatomy_of_a_hurd_system.mdwn new file mode 100644 index 00000000..496d2a06 --- /dev/null +++ b/open_issues/anatomy_of_a_hurd_system.mdwn @@ -0,0 +1,1430 @@ +[[!meta copyright="Copyright © 2011, 2012, 2013, 2014 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!taglink open_issue_documentation]] + +A bunch of this should also be covered in other (introductionary) material, +like Bushnell's Hurd paper. All this should be unfied and streamlined. + +[[!toc]] + + +# IRC, freenode, #hurd, 2011-03-08 + + I've a question on what are the "units" in the hurd project, if + you were to divide them into units if they aren't, and what are the + dependency relations between those units(roughly, nothing too pedantic + for now) + there is GNU Mach (the microkernel); there are the server + libraries in the Hurd package; there are the actual servers in the same; + and there is the POSIX implementation layer in glibc + relations are a bit tricky + Mach is the base layer which implements IPC and memory management + hmm I'll probably allocate time for dependency graph generation, + in the worst case + on top of this, the Hurd servers, using the server libraries, + implement various aspects of the system functionality + client programs use libc calls to use the servers + (servers also use libc to communicate with other servers and/or + Mach though) + so every server depends solely on mach, and no other server? + s/mach/mach and/or libc/ + I think these things should be pretty clear one you are somewhat + familiar with the Hurd architecture... nothing really tricky there + no + servers often depend on other servers for certain functionality + + +# Bootstrap + +## [[hurd_init]] + +## IRC, freenode, #hurd, 2011-03-12 + + when mach first starts up, does it have some basic i/o or fs + functionality built into it to start up the initial hurd translators? + I/O is presently completely in Mach + filesystems are in userspace + the root filesystem and exec server are loaded by grub + o I see + so in order to start hurd, you would have to start mach and + simultaneously start the root filesystem and exec server? + not exactly + GRUB loads all three, and then starts Mach. Mach in turn starts + the servers according to the multiboot information passed from GRUB + ok, so does GRUB load them into ram? + I'm trying to figure out in my mind how hurd is initially started + up from a low-level pov + yes, as I said, GRUB loads them + ok, thanks antrik...I'm new to the idea of microkernels, but a + veteran of monolithic kernels + although I just learned that windows nt is a hybrid kernel which I + never knew! + note there's a /hurd/ext2fs.static + I belive that's what is used initially... right? + yes + loading the shared libraries in addition to the actual server + would be unweildy + so the root FS server is linked statically instead + what does the root FS server do? + well, it serves the root FS ;-) + it also does some bootstrapping work during startup, to bring the + rest of the system up + + +## IRC, freenode, #hurd, 2014-01-03 + + hmpf, the hurd bootstrapping process is complicated and fragile, + maybe to the point that it is to be considered broken + aiui the hurd uses the filesystem for service lookup + older mach documentation suggests that there once existed a name + server instead for this purpose + the hurd approach is elegant and plan9ish + the problem is in the early bootstrapping + what if the root filesystem is r/o and there is no /servers or + /servers/exec ? + e. g. rm /servers/exec && reboot -> the rootfs dies early in the + hurd server bootstrap :/ + well yes + it's normal to have such constraints + uh no + at the same time, the boot protocol must be improved, if only to + support userspace disk drivers + totally unacceptable + why not ? + b/c my box just died and lost it's exec node + so ? + loosing the exec node is unacceptable + well, linux dies too if you don't have /dev populated at least a + bit + not being able to boot without the "exec" service is pretty normal + the hurd turns the vfs into a service directory + the exec service is there, only the lookup mechanism is broken + replacing the name server you mentioned earlier + yes + if you don't have services, you don't have them + i don't see the problem + the problem is the lookup mechanism getting broken + ... that easily + imagine a boot protocol based on a ramfs filled from a cpio + i do actually ;) + there would be no reason at all the lookup mechanism would break + yes + but the current situation is not acceptable + i agree + ^^ + ext2fs is too unreliable for that + but using the VFS as a directory is more than acceptable + it's probably the main hurd feature + yes + i see it rather as a circular dependency problem + and if you have good ideas, i'm all ear for propel ... :> + antrik already talked about some of them for the bootstrap + protocol + we should sum them up somewhere if not done already + i've been pondering how to install a tmpfs translator as root + translator + braunr: we could create a special translator for /servers + maybe + very much like fakeroot, it just proxies messages to a real + translator + but if operations like settrans fail, we handle them + transparently, like an overlay + i consider /servers to be very close to /dev + yes + so something like devfs seems obvious yes + i don't even think there needs to be an overlay + y not ? + why does /servers need real nodes ? + for persistence + what for ? + e.g. crash server selection + hm ok + network configuration + i personally wouldn't make that persistent + it can be configured in files and installed at boot time + me neither, but that's how it's currently done + are you planning to actually work on that soon ? + if we need no persistence, we can just use tmpfs + it wouldn't be a mere tmpfs + it could + it's a tmpfs that performs automatic discovery and registration of + system services + with some special wrapper that preserves e.g. /servers/exec + oh + so rather, devtmpfs + it is o_O :p + ? + what is what ? + well, it could be a tmpfs and some utility creating the nodes + whether the node management is merged in or separate doesn't + matter that much i guess + i'd personally imagine it merged, and tmpfs available as a + library, so that stuff like sysfs or netstatfs can easily be written + + +## IRC, freenode, #hurd, 2014-02-12 + + braunr: i fixed all fsys-related receiver lookups in libdiskfs + and surely enough the bootstrap hangs with no indication whats wrong + teythoon: use mach_print :/ + braunr: the hurd bootstrap is both fragile and hard to tweak in + interesting ways :/ + teythoon: i agree with that + teythoon: maybe this will help : + http://wiki.hurdfr.org/upload/graphviz/dot9b65733655309d059dca236f940ef37a.png + although i guess you probably already know that + heh, unicode for the win >,< + :/ + + +# Source Code Documentation + +Provide a cross-linked sources documentation, including generated files, like +RPC stubs. + + * + + +# [[Hurd_101]] + + +# [[hurd/IO_path]] + +Need more stuff like that. + + +# IRC, freenode, #hurd, 2011-10-18 + + what happens @ boot. and which translators are started in what + order? + short version: grub loads mach, ext2, and ld.so/exec; mach starts + ext2; ext2 starts exec; ext2 execs a few other servers; ext2 execs + init. from there on, it's just standard UNIX stuff + + +# IRC, OFTC, #debian-hurd, 2011-11-02 + + is __dir_lookup a RPC ?? + where can i find the source of __dir_lookup ?? + grepping most gives out rvalue assignments + -assignments + but in hurs/fs.h it is used as a function ?? + it should be the mig-generated function for that rpc + how do i know how its implemented ?? + is there any way to delve deeprer into mig-generated functions + sekon_: The MIG-generated stuff will either be found in the + package's build directory (if it's building it for themselves), or in the + glibc build directory (libhurduser, libmachuser; which are all the + available user RPC stubs). + sekon_: The implementation can be found in the various Hurd + servers/libraries. + sekon_: For example, [hurd]/libdiskfs/dir-lookup.c. + sekon_: What MIG does is provide a function call interface for + these ``functions'', and the Mach microkernel then dispatches the + invocation to the corresponding server, for example a /hurd/ext2fs file + system (via libdiskfs). + sekon_: This may help a bit: + http://www.gnu.org/software/hurd/hurd/hurd_hacking_guide.html + + +# IRC, freenode, #hurd, 2012-01-08 + + can you tell me how is done in hurd: "ls | grep x" ? + in bash + ls's standard output is a port to the pflocal server, and grep x's + standard input is a port to the pflocal server + the connexion between both ports inside the pflocal server being + done by bash when it calls pipe() + youpi, so STDOUT_FILENO, STDIN_FILENO, STDERR_FILENO still exists + ? + sure, hurd is compatible with posix + so bash 1) creates T1 (ls) and T2 (grep), then create a pipe at + the pflocal server, then connects both ends to T1 and T2, then start(T1), + start(T2) ? + not exactly + it's like on usual unix, bash creates the pipe before creating the + tasks + then forks to create both of them, handling them each side of the + pipe + ok I see + s/handling/handing/ + but when you do pipe() on linux, it creates a kernel object, this + time it's 2 port on the pflocal ? + yes + how are spawned tasks ? + with fork() ? + yes + which is just task_create() and duplicating the ports into the new + task + ok + so it's easy to rewrite fork() with a good control of duplicated + fd + about threading, mutexes, conditions, etc.. are kernel objects or + just userland objects ? + just ports + (only threads are kernel objects) + so, about efficiency, are pipes and mutexes efficient ? + depends what you call "efficient" + it's less efficient than on linux, for sure + but enough for a workable system + maybe hurd is the right place for a userland thread library like + pth or any fiber library + ? + hurd already uses a userland thread library + libcthreads + is it M:N ? + libthreads, actually + yes + +Actually, the Hurd has never used an M:N model. Both libthreads (cthreads) and libpthread use an 1:1 model. + + nice + is the task scheduler in the kernel ? + the kernel thread scheduler, yes, of course + there has to be one + are the posix open()/readdir()/etc... the direct vfs or wraps an + hurd layer libvfs ? + they wrap RPCs to the filesystem servers + the Bushnell paper is probably the closest we have to a high-level + documentation of these concepts... + the Hurd does not have a central VFS component at all. name + lookups are performed directly on the individual FS servers + that's probably the most fundamental design feature of the Hurd + (all filesystem operations actually, not only lookups) + + +## IRC, freenode, #hurd, 2012-01-09 + + youpi: are you sure cthreads are M:N ? i'm almost sure they're 1:1 + and no modern OS is a right place for any thread userspace + library, as they wouldn't have support to run threads on different + processors (unless processors can be handled by userspace servers, but + still, it requires intimate cooperation between the threading library and + the kernel/userspace server in any case + braunr: in libthreads, they are M:N + you can run threads on different processors by using several kernel + threads, there's no problem in there, a lot of projects do this + a pure userspace library can't use kernel threads + at least pth was explacitely used on systems like bsd at a time + when they didn't have kernel threads exactly for that reason + explicitely* + and i'm actually quite surprised to learn that we have an M:N + threading model :/ + why do you say "can't" ? + but i wanted to reply to abique and he's not around + of course you need kernel threads + but all you need is to bind them + well, what i call a userspace threading library is a library that + completely implement threads without the support of the kernel + or only limited support, like signals + errr, you can't implement anything with absolutely no support of + the kernel + pth used only SIGALRM iirc + asking for more kernel threads to use more processors doesn't seem + much + it's not + but i'm refering to what abique said + 01:32 < abique> maybe hurd is the right place for a userland + thread library like pth or any fiber library + well, it's indeed more, because the glibc lets external libraries + provide their mutex + while on linux, glibc doesn't + i believe he meant removing thread support from the kernel :p + ah + and replying "nice" to an M:N threading model is also suspicious, + since experience seems to show 1:1 models are better + "better" ???? + yes + well + I don't have any time to argue about that + because that'd be extremely long + simpler, so far less bugs, and also less headache concerning posix + conformance + but there's no absolute "better" here + but less performant + less flexible + that's why i mention experience :) + I mean experience too + why less performant ? + because you pay kernel transition + because you don't know anything about the application threads + etc. + really ? + yes + i fail to see where the overhead is + I'm not saying m:n is generally better than 1:1 either + thread switch, thread creation, etc. + creation is slower, i agree, but i'm not sure it's used frequently + enough to really matter + it is sometimes used frequently enough + and in those cases it would be a headache to avoid it + ok + i thought thread pools were used in those cases + synchronized with kernel mutexes ? + that's still slow + it reduces to the thread switch overhead + which, i agree is slightly slower + ok, i's a bit less performant :) + well don't futexes exist just for that too ? + yes and no + in that case they don't help + because they do sleep + they help only when the threads are living + ok + now as I said I don't have to talk much more, I have to leave :) + + +# IRC, freenode, #hurd, 2012-12-06 + + spiderweb: have you read + http://www.gnu.org/software/hurd/hurd-paper.html ? + I'll have a look. + and also the beginning of + http://ftp.sceen.net/mach/mach_a_new_kernel_foundation_for_unix_development.pdf + these two should provide a good look at the big picture the hurd + attemtps to achieve + I can't help but wonder though, what advantages were really + achieved with early mach? + weren't they just running a monolithic unix server like osx does? + most mach-based systems were + but thanks to that, they could provide advanced features over + other well established unix systems + while also being compatible + so basically it was just an ease of development thing + well that's what mach aimed at being + same for the hurd + making things easy + but as a side effect hurd actually delivers on the advantages of + microkernels aside from that, but the older systems wouldn't, correct? + that's how there could be network file systems in very short time + and very scarce resources (i.e. developers working on it), while on other + systems it required a lot more to accomplish that + no, it's not a side effect of the microkernel + the hurd retains and extends the concept of flexibility introduced + by mach + the improved stability, etc. isn't a side effect of being able to + restart generally thought of as system-critical processes? + no + you can't restart system critical processes on the hurd either + that's one feature of minix, and they worked hard on it + ah, okay. so that's currently just the domain of minix + okay + spiderweb: well, there's 1 advantage of minix for you :P + the main idea of mach is to make it easy to extend unix + without having hundreds of system calls + +[[/system_call]]. + + the hurd keeps that and extends it by making many operations + unprivileged + you don't need special code for kernel modules any more + it's easy + you don't need special code to handle suid bits and other ugly + similar hacks, + it's easy + you don't need fuse + easy + etc.. + + +# Service Directory + +## IRC, freenode, #hurd, 2012-12-06 + + what is the #1 feature that distinguished hurd from other + operating systems. the concept of translators. (will read more when I get + more time). + yes, translators + using the VFS as a service directory + and the VFS permissions to control access to those services + + +## IRC, freenode, #hurd, 2013-05-23 + + Hi, is there any efficient way to control which backed + translators are called via RPC with a user space program? + Take for example io_stat: S_io_stat is defined in boot/boot.c, + pfinet/io-ops.c and pflocal/io.c + And the we have libdiskfs/io-stat.c:diskfs_S_io_stat, + libnetfs/io-stat.c:netfs_S_io_stat, libtreefs/s-io.c:treefs_S_io_stat, + libtrivfs/io-stat.c:trivfs_S_io_stat + How are they related? + gnu_srs: it depends on the server (translator) managing the files + (nodes) you're accessing + so use fsysopts to know the server, and see what this server uses + fsysopts /hurd/pfinet and fsysopts /hurd/pflocal gives the same + answer: ext2fs --writable --no-inherit-dir-group --store-type=typed + device:hd0s1 + of course + the binaries are regular files + see /servers/socket/1 and /servers/socket/2 instead + which are the nodes representing the *service* + again, the hurd uses the file system as a service directory + this usage of the file system is at the core of the hurd design + files are not mere files, they're service names + it happens that, for most files, the service behind them is the + same as for regular files + gnu_srs: this *must* be obvious for you to do any tricky work on + the hurd + + Anyway, if I create a test program calling io_stat I assume + S_io_stat in pflocal is called. + How to make the program call S_io_stat in pfinet instead? + create a socket managed by pfinet + i.e. an inet or inet6 socket + you can't assume io_stat is serviced by pflocal + only stats on unix sockets of pipes will be + or* + thanks, what about the *_S_io_stat functions? + what about them ? + How they fit into the picture, e.g. diskfs_io_stat? + *diskfs_S_io_stat + gnu_srs: if you open a file managed by a server using libdiskfs, + e.g. ext2fs, that one will be called + Using the same user space call: io_stat, right? + it's all userspace + say rather, client-side + the client calls the posix stat() function, which is implemented + by glibc, which converts it into a call to io_stat, and sends it to the + server managing the open file + the io_stat can change depending on the server + the remote io_stat implementation, i mean + identify the server, and you will identify the actual + implementation + + +## IRC, freenode, #hurd, 2013-06-30 + + hi, what is the replacer of netname_check_in? + + I want to ask another question. in my opinion, the rpc is the + mach's way, and the translator is the hurd's way. so somebody want to + lookup a service, it should not need to ask the mach kernel know about + this query. the hurd will take the control. + am I right? + no + that's nonsense + service lookups has never been in mach + first mach based systems used a service directory, whereas the + hurd uses the file system for that + you still need mach to communicate with either of those + how to understand the term of service directory here? + a server everyone knows + which gives references to other servers + usually, depending on the name + e.g. name_lookup("net") -> port right to network server + is that people use netname_check_in to register service in the + past? now used libtrivfs? + i don't know about netname_check_in + old mach (not gnumach) documentation might mention this service + directory + libtrivfs doesn't have much to do with that + on the hurd, the equivalent is the file system + maybe that is outdate, I just found that exist old doc, and old + code which can't be build. + every process knows / + the file system is the service directory + nodes refer to services + so the file system is the nameserver, any new service should + register in it before other can use + and the file system is distributed, so looking up a service may + require several queries + setting a translator is exactly that, registering a program to + service requests on a node + the file system isn't one server though + programs all know about /, but then, lookups are recursive + e.g. if you have / and /home, and are looking for + /home/hacklu/.profile, you ask / which tells you about /home, and /home + will give you a right to /home/hacklu/.profile + even in the past, the mach don't provide name register service, + there must be an other server to provide this service? + yes + what's nonsense in your sentence is comparing RPCs and translators + translators are merely servers attached to the file system, using + RPCs to communicate with the rest of the system + I know yet, the two just one thing. + no + two things :p + completely different and unrelated except for one using the other + ah, just one used aonther one. + is exist anyway to anounce service except settrans with file node? + more or less + tasks can have special ports + that's how one task knows about / for example + at task creation, a right to / is inserted in the new task + I think this is also a file node way. + no + if i'm right, auth is referenced the same way + and there is no node for auth + how the user get the port of auth with node? + it's given when a task is created + pre-set in the creation of one task? + i'm unconfortable with "pre-set" + inserted at creation time + auth is started very early + then tasks are given a reference to it + + +# IRC, freenode, #hurd, 2012-12-10 + + I want to work on hurd, but I think I'm going to start with + minix, I own the minix book 3rd ed. it seems like a good intro to + operating systems in general. like I don't even know what a semaphore is + yet. + well, enjoy learning :) + once I finish that book, what reading do you guys recommend? + other than the wiki + i wouldn't recommend starting with a book that focuses on one + operating system anyway + you tend to think in terms of what is done in that specific + implementation and compare everything else to that + tannenbaum is not only the main author or minix, but also the one + of the book http://en.wikipedia.org/wiki/Modern_Operating_Systems + + http://en.wikipedia.org/wiki/List_of_important_publications_in_computer_science#Operating_systems + should be a pretty good list :) + + +# IRC, freenode, #hurd, 2013-03-12 + + i have a question regarding ipc in hurd. if a task is created, does + it contain any default port rights in its space? i am trying to deduce + how one calls dir_lookup() on the root translator in glibc's open(). + mjjc: yes, there are some default port rights, but I don't + remember the details :/ + kilobug: do you know where i should search for details? + mjjc: hum either in the Hurd's hacking guide + https://www.gnu.org/software/hurd/hacking-guide/ or directly in the + source code of exec server/libc I would say, or just ask again the + question here later on to see if someone else has more information + ok, thanks + there's also rpctrace to, as the name says, trace all the rpc's + executed + some ports are introduced in new tasks, yes + see + http://www.gnu.org/software/hurd/hacking-guide/hhg.html#The-main-function + and + + http://www.gnu.org/software/hurd/gnumach-doc/Task-Special-Ports.html#Task-Special-Ports + yes, the second link was just what i was looking for, thanks + the second is very general + also, the first applies to translators only + if you're looking for how to do it for a non-translator + application, the answer is probably somewhere in glibc + _hurd_startup i'd guess + + +# IRC, freenode, #hurd, 2013-06-15 + + ive been reading a little about exokernels or unikernels, and i + was wondering if it might be relevant to the GNU/hurd design. I'm not + too familiar with hurd terminology so forgive me. what if every + privileged service was compiled as its own mini "kernel" that handled (a) + any hardware related to that service (b) any device nodes exposed by that + service etc... + yes but not really that way + under the current hurd model of the operating system, how would + you talk to hardware that required specific timings like sound hardware? + through mapped memory + is there such a thing as an interrupt request in hurd? + obviously + ok + is there any documentation i can read that involves a driver that + uses irqs for hurd? + you can read the netdde code + dde being another project, there may be documentation about it + somewhere else + i don't know where + thanks + i read a little about dde, apparently it reuses existing code from + linux or bsd by reimplementing parts of the old kernel like an api or + something + yes + it must translate these system calls into ipc or something + then mach handles it? + exactly + +[[/system_call]]. + + that's why i say it's not the exokernel way of doing things + ok + so does every low level hardware access go through mach?' + yes + well no + interrupts do + ports (on x86) + everything else should be doable through mapped memory + seems surprising that the code for it is so small + 1/ why surprising ? and 2/ "so small" ? + its like the core of the OS, and yet its tiny compared to say the + linux kernel + it's a microkenrel + well, rather an hybrid + the size of the equivalent code in linux is about the same + ok + with the model that privileged instructions get moved to + userspace, how does one draw the line between what is OS and what is user + code + privileged instructions remain in the kernel + that's one of the few responsibilities of the kernel + i see, so it is an illusion that the user has privilege in a sense + hum no + or, define "illusion" + well the user can suddenly do things never imaginable in linux + that would have required sudo + yes + well, they're not unimaginable on linux + it's just not how it's meant to work + :) + and why things like fuse are so slow + i still don't get "i see, so it is an illusion that the user has + privilege in a sense" + because the user doesnt actually have the elevated privilege its + the server thing (translator)? + it does + not at the hardware level, but at the system level + not being able to do it directly doesn't mean you can't do it + right + it means you need indirections + that's what the kernel provides + so the user cant do stuff like outb 0x13, 0x1 + he can + he also can on linux + oh + that's an x86 specifity though + but the user would need hardware privilege to do that + no + or some kind of privilege + there is a permission bitmap in the TSS that allows userspace to + directly access some ports + but that's really x86 specific, again + i was using it as an example + i mean you wouldnt want userspace to directly access everything + yes + the only problem with that is dma reall + y + because dma usually access physical memory directly + are you saying its good to let userspace access everything minus + dma? + otherwise you can just centralize permissions in one place (the + kernel or an I/O server for example) + no + you don't let userspace access everything + ah + yes + userspace asks for permission to access one specific part (a + memory range through mapping) + and can't access the rest (except through dma) + except through dma?? doesnt that pose a large security threat? + no + you don't give away dma access to anyone + only drivers + ahh + and drivers are normally privileged applications anyway + so a driver runs in userspace? + so the only effect is that bugs can affect other address spaces + indirectly + netdde does + interesting + and they all should but that's not the case for historical reasons + i want to port ALSA to hurd userspace :D + that's not so simple unfortunately + one of the reasons it's hard is that pci access needs arbitration + and we don't have that yet + i imagine that would be difficult + yes + also we're not sure we want alsa + alsa drivers, maybe, but probably not the interface itself + its tangled spaghetti + but the guy who wrote JACK for audio hates OSS, and believes it is + rubbish due to the fact it tries to read and write to a pcm device node + like a filesystem with no care for timing + i don't know audio well enough to tell you anything about that + was that about oss3 or oss4 ? + also, the hurd isn't a real time system + so we don't really care about timings + but with "good enough" latencies, it shouldn't be a problem + but if the audio doesnt reach the sound card in time, you will get + a crackle or a pop or a pause in the signal + yep + it happens on linux too when the system gets some load + some users find this unnacceptable + some users want real time systems + using soft real time is usually plenty enough to "solve" this kind + of problems + will hurd ever be a real time system? + no idea + if somebody works on it why not + it's the same as linux + it should certainly be simpler than on linux though + hmm + microkernels are well suited for real time because of the well + defined interfaces they provide and the small amount of code running in + kernel + that sounds promising + you usually need to add priority inheritance and take care of just + a few corner cases and that's all + but as youpi said, it still requires work + and nobody's working on it + you may want to check l4 fiasco.oc though + + +# System Personality + +## IRC, freenode, #hurd, 2013-07-29 + + over the past few days I gained a new understanding of the Hurd + teythoon: really ? :) + teythoon: That it's a complex and distributed system? ;-) + And at the same time a really simple one? + ;-D + it's just a bunch of mach programs and some do communicate and + behave in a way a posix system would, but that is more a convention than + anything else + tschwinge: yes, kind of simple and complex :) + the right terminology is "system personality" + 11:03 < teythoon> over the past few days I gained a new + understanding of the Hurd + teythoon: still no answer on that :) + braunr: ah, I spent lot's of time with the core servers and + early bootstrapping and now I gained the feeling that I've seen the Hurd + for what it really is for the first time + + +# RPC Interfaces + +## IRC, freenode, #hurd, 2013-09-03 + + I'm a little confused by the hurd and incubator git repos. + DDE is only found in the dde branch in incubator, but not in the + hurd repo. + Does this mean that DDE is not ready for master yet? + yes + If DDE is not yet used in the hurd (except in the dde branch in + the incubator repo), does pfinet use some custom glue code to use the + Linux drivers? + this has nothing to do with pfinet + pfinet is the networking stack, netdde are the networking drivers + the interface between them doesn't change, whether drivers are in + kernel or not + I see + + +# IRC, freenode, #hurd, 2013-09-20 + + HI there, I have no previous knowledge about OS's. I'm trying to + undestand the structure of the Hurd and the comparison between, say, + Linux way of managing stuff ... + for instance, I read: "Unlike other popular kernel software, the + Hurd has an object-oriented structure that allows it to evolve without + compromising its design." + that means that while for adding feature to the Linux-kernel you + have to add some stuff `inside` a procedure, whilst in the Hurd kernel + you can just, in principle at least, add an object and making the kernel + using it?... + Am I making stuff too simple? + Thanks + not exactly + unix historically has a "file-oriented" structure + the hurd allows servers to implement whatever type they want, + through the ability to create custom interfaces + custom interfaces means custom calls, custom semantics, custom + methods on objects + you're not restricted to the set of file interfaces (open, seek, + read, write, select, close, etc..) that unix normally provides + braunr: uhm ...some example? + see processes for example + see + http://darnassus.sceen.net/gitweb/savannah_mirror/hurd.git/tree/HEAD:/hurd + this is the collection of interfaces the hurd provides + most of them map to unix calls, because gnu aims at posix + compatibility too + some are internal, like processes + or authentication + but most importantly, you're not restricted to that, you can add + your own interfaces + on a unix, you'd need new system calls + or worse, extending through the catch-all ioctl call + braunr: mhn ...sorry, not getting that. + what part ? + ioctl has become such a mess :s + braunr: when you say that Unix is `file-oriented` you're + referring to the fact that sending/receiving data to/from the kernel is + designed like sending/receiving data to/from a file ...? + not merely sending/receiving + note how formatted your way of thinking is + you directly think in terms of sending/receiving (i.e. read and + write) + braunr: (yes) + that's why unix is file oriented, access to objects is done that + way + on the hurd, the file interface is one interface + there is nothing preventing you from implementing services with a + different interface + as a real world example, people interested in low latency + profesionnal audio usually dislike send/recv + see + http://lac.linuxaudio.org/2003/zkm/slides/paul_davis-jack/unix.html for + example + braunr: how big and messy ioctl has become is a good proof that + the Unix way, while powerful, does have its limits + giuscri: keep in mind the main goal of the hurd is extensibility + without special privileges + braunr: privileges? + root + braunr: what's wrong with privileges? + they allow malicious/buggy stuff to happne + and have dramatic effects + braunr: you're obviously *not* referring to the fact that once + one have the root privileges could change some critical-data + ? + i'm referring to why privilege separation exists in the first + place + if you have unprivileged users, that's because you don't want them + to mess things up + on unix, extending the system requires privileges, giving those + who do it the ability to destroy everything + braunr: yes, I think the same + the hurd is designed to allow unprivileged users to extend their + part of the system, and to some extent share that with other users + although work still remains to completely achieve that + braunr: mhn ...that's the `server`-layer between the + single-application and kernel ? + the multi-server based approach not only allows that, but + mitigates damage even when privileged servers misbehave + one aspect of it yes + but as i was just saying, even root servers can't mess things too + much + for example, our old (sometimes buggy) networking stack can be + restarted when it behaves wrong + the only side effect being some applications (ssh and exim come to + mind) which need to be restarted too because they don't expect the + network stack to be restarted + braunr: ...instead? + ? + giuscri: on Linux, if the network stack crash/freezes, you don't + have any other option than rebooting the system - usually with a nice + "kernel pani" + giuscri: and you may even get filesystem corruption "for free" in + the bundle + and hoping it didn't corrupt something important like file system + caches before being flushed + kilobug, braunr : mhn, ook + + +# IRC, freenode, #hurd, 2013-10-13 + + ahh, ^c isn't working to cancel a ping - is there alternative? + ahungry: ctrl-c does work, you just missed something somewhere and + are running a shell directly on a console, without a terminal to handle + signals + + +# IRC, freenode, #hurd, 2013-11-04 + + nalaginrut: you can't use the hurd for real embedded stuff without + a lot of work on it + but the hurd design applies very well to embedded environments + the fact that we're able to dynamically link practically all hurd + servers against the c library can visibly reduce the system code size + it also reduces the TCB + what about the memory occupation? + code size is about memory occupation + also, the system is composable like lego, don't need tcp - don't + include pfinet then + the memory overheald of a capability based system like the hurd + are, well, capabilities + teythoon: that's not an argument compared to modular kernels like + linux + yes it is + why ? + if you don't need tcp in linux, you just don't load it + same thing + ok, right + on the other hand, a traditional unix kernel can never be linked + against the c library + much less dynamically + right + I think the point is that it's easy to cut, since it has + better modularity than monolithic, and could be done in userland relative + easier + modularity isn't better + that's a big misconception + also, restarting components is easier on a distributed system + on the hurd, this is a side effect + and it doesn't apply well + braunr: oops, misconception + many core servers such as proc, auth, exec, the root fs server + can't be restarted at all + not yet + and servers like pfinet can be restarted, but at the cost of posix + servers not expecting that + looping on errors such as EBADF because the target socket doesn't + exist any more + I've been working on a restartable exec server during some of my + gsoc weekends + ah right + linux has kexec + and can be patched at run time + sounds like Hurd needs something similar to generalizable + continuation + so again, it's not a real advantage + no + sorry serilizable + that would persistence + personally, i don't want it at all + yes it is a real advantage, b/c the means of communication + (ports) is common to every IPC method on Hurd, and ports are first class + objects + so preserving the state is much easier on Hurd + if a monolithic kernel can do it too, it's not a real advantage + yes, but it is more work + that is one true advantage of the hurd + but don't reuse it each time + oh, that's nice for the ports + why not? + what we're talking about here is resilience + the fact that it's easier to implement doesn't mean the hurd is + better because it has resilience + it simply means the hurd is better because it's easier to + implement things on it + same for development in general + debugging + virtualization + etc.. + yes, but why we stick to compare it to monolithic + but it's still *one* property + well, minix advertises this feature a lot, even if minix can + only restart very simple things like printer servers + minix sucks + let them advertise what they can + ^^ + it has cool features, that's enough, no need to find a feature + that monolithic can never done + no it's not enough + minix isn't a general purpose system + let's just not compare it to general purpose systems + + +# IRC, freenode, #hurd, 2013-11-08 + + and, provided you have suitable language bindings, you can + replace almost any hurd server with your own implementation in any + language + teythoon: language bindings? + Do you mean language bindings against C libraries? + either that or for the low level mach primitives + For your information, IPC is independent of languages. + sure, that's the beauty + Why is hurd best for replacing parts written in C with other + languages? + because Hurd consists of many servers, each server managing one + kind of resource + so you have /hurd/proc managing posix processes + you could reimplement /hurd/proc in say python or go, and + replace just that component of the Hurd system + you cannot do this with any other (general purpose) operating + system that I know of + you could incrementally replace the Hurd with your own + Hurd-compatible set of servers written in X + use a language that you can verify, i.e. prove that a certain + specification is fulfilled, and you end up with an awesome stable and + secure operating system + Any microkernel OS fits the description. + teythoon, Does hurd have formal protocols for IPC communications? + sure, name some other general purpose and somewhat + posix-compatible microkernel based operating system please + what do you mean by formal protocols ? + IPC communications need to be defined in documents. + the "wire" format is specified of course, the semantic not so + much + network protocols exist. + HTTP is a transport protocol. + Without formal protocols, IPC communications suffer from + debugging difficulties. + Formal protocols make it possible to develop and test each module + independently. + as I said, the wire format is specified, the semantics only in + written form in the source + this is an example of the ipc specification for the proc server + http://git.savannah.gnu.org/cgit/hurd/hurd.git/tree/hurd/process.defs + teythoon, how file server interacts with file clients should be + defined as a formal protocol, too. + do you consider the ipc description a kind of formal protocol ? + + http://git.savannah.gnu.org/cgit/hurd/hurd.git/tree/hurd/process.defs can + be considered as a formal protocol. + However, the file server protocol should be defined on top of IPC + protocol. + the file server protocol is in fs.defs + every protocol spoken is defined in that ipc description + language + it is used to derive code from + crocket: not any system can be used to implement system services + in any language + in theory, they do, but in theory only + the main reason they don't is because most aren't posix compliant + from the ground up + posix compliance is achieved through virtualization + which isolates services too much for them to get useful, + notwithstanding the impacts on performance, memory, etc.. + braunr, Do you mean it's difficult to achieve POSIX compliance + with haskell? + crocket: i mean most l4 based systems aren't posix + genode isn't posix + helenos is by design not posix + the hurd is the only multi server system providing such a good + level of posix conformance + and with tls on the way, we'll support even more non-posix + applications that are nonetheless very common on unices because of + historical interfaces still present, such as mcontext + and modern ones + e.g. ruby is now working, go should be there after tls + * teythoon drools over the perspective of having go on the Hurd... + braunr, Is posix relevant now? + it's hugely relevant + conforming to posix and some native unix interfaces is the only + way to reuse a lot of existing production applications + and for the matter at hand (system services not written in c), it + means almost readily getting runtimes for other languages than c + something other microkernel based system will not have + imagine this + one day, one of us could create a company for a hurd-like system, + presenting this idea as the killer feature + by supporting posix, customers could port their software with very + little effort + *very little effort* is what makes software attractive + + http://stackoverflow.com/questions/1806585/why-is-linux-called-a-monolithic-kernel/1806597#1806597 + says "The disadvantage to a microkernel is that asynchronous IPC + messaging can become very difficult to debug, especially if fibrils are + implemented." + " GNU Hurd suffers from these debugging problems (reference)." + stackoverflow is usually a nice place + but concerning microkernel stuff, you'll read a lot of crap + anywhere + whether it's sync or async, tracking references is a hard task + it's a bit more difficult in distributed systems, but not that + much if the proper debugging features are provided + we actually don't suffer from that too much + many of us have been able to debug reference leaks in the past, + without too much trouble + we lack some tools that would give us a better view of the system + state + braunr, But is it more difficult with microkernel? + crocket: it's more difficult with distributed systems + How much more difficult? + i don't know + distributed systems + not much + braunr, How do you define distributed systems? + crocket: not monolithic + braunr, Hurd is distributed, then. + multiserver if you prefer + yes it is + braunr, So it is more difficult with hurd. + How much more difficult? How do you debug? + just keep in mind that a monolithic system can run on a + microkenrel + we use tools that show us references + braunr, like? + like portinfo + braunr, Does hurd use unix-socket to implement IPC? + no + unix-socket use mach ipc + I'm confused + ipc is provided by the microkernel, gnumach (a variant of mach) + unix sockets are provided by one of the hurd servers (pflocal) + servers and clients communicate through mach ipc + braunr, Do you think it's feasible to build servers in haskell? + why not ? + ok + I've been thinking about that + in go, with cgo, you can call go functions from c code + so it should be possible to create bindings for say libtrivfs + I'd like to write an OS in clojure or haskell. + crocket: what for ? + braunr, I want to see a better system programming language than + C. + i don't see how clojure or haskell would be "better system + programming languages" than c + and even assuming that, what for ? + braunr, It's better for programmers. + haskell + haskell is expressive. + personally i disagree + it's better for some things + not for system programming + For system programming, Google Go is trying to replace C. But I + doubt it will. + we may not be referring to the same thing here when we say "system + programming" + braunr, What do you think is a better one? + crocket: i don't think there is a better one currently + braunr, Even Rust and D? + i don't know them well enough + certainly not D if it's what i think it is + C is too slow. + C is too slow to develop. + depends + again, i disagree + rust looks good but i don't know it well to comment + C is a tank, and clojure is an airplane. + A tank is reliable but slow. + Clojure is fast but lacks some accuracy. + c is as reliable as the developer is skilled with it + it's clearly not a tank + there are many traps + crocket: are you suggesting to rewrite Hurd in Clojure? + no + Why rewrite hud? + hurd + I'd rather start from scratch. + which is what a rewrite is + I am not expert on Clojure, but I don't think it is made for + system programming. + If you want alternate language, I thing Go is only serious + candidate other than C + Or Rust + However, some people wrote OSes in haskell. + again, why ? + if it's only for the sake of using another language, i think it's + bad reason + Because haskell provides a high level of abstraction that helps + programmers. + It is more secure with monads. + If you want your OS to become successful Free Software project, + you have to use popular language. Haskell is not. + Most Haskell programmers are not into kernels + They do high level stuff. + So little contributors. + crocket: so you aim at security ? + I mean, candidats for contribution + braunr, security and higher abstraction. + i don't understand higher abstraction + braunr, FP can be useful to systems. + FP ? + functional programming + right + but you can abstract a lot with c too, with more efforts + braunr, like that's easy. + it's not that hard + i'm just questioning the goals and the solution of using a + particular language + the reason c is still the preferred language for system + programming is because it provides control over how the hardware does + stuff + which is very important for performance + the hurd never took off because of bad performance + performance doesn't mean doing things faster, it means being able + to do things or not, or doing things a new way + so ok, great, you have your amazing file system written in + haskell, and you find out it doesn't scale at all beyond some threshold + of processors or memory + braunr, L4 is fast. + l4 is merely an architecture abstraction + and it's not written in haskell :p + don't assume anything running on top of something fast will be + fast + Hurd is slow and written in C. + yes + not because of c though + Becuase it's microkernel? + because c wasn't used well enough to make the most of the hardware + in many places + far too many places + A microkernel can be as fast as a monolithic kernel according to + L4. + no + it can't + it can for very specific cases + almost none of which are real world + but that's not the problem + again, i'm questioning your choice of another language in relation + to your goals, that's all + c can do things you really can't do easily in other languages + be aware of that + braunr, "Monolithic kernel are faster than microkernel . while + The first microkernel Mach is 50% slower than Monolithic kernel while + later version like L4 only 2% or 4% slower than the Monolithic kernel ." + 14:05 < braunr> but concerning microkernel stuff, you'll read a + lot of crap anywhere + simple counterexample : + the measurements you're giving consider a bare l4 kernel with + nothing on top of it + doing thread-to-thread ipc + this model of communication is hardly used in any real world + application + one of the huge features people look for with microkernels are + capabilities + and that alone will bump your 4% up + since capabilities will be used for practically every ipc + ok + + +# Hurd From Scratch + +## IRC, freenode, #hurd, 2013-11-30 + + because I think there is no way to understand the whole pile, + you need to go step by step + for example, I'm starting with mach only, then adding one + server, then another and on each step I have working system + that's how I want to understand it + you are interested in the early bootstrapping of the hurd system + ? + now I'm starting debian gnu/mach, it hungs, show me black + screen and I have no idea how to fix it + if you are unable to fix this, why do you think you can build a + hurd system from scratch ? + not gnu/mach, gnu/hurd I mean + or, you could describe your problem in more detail and one of + the nice people around here might help you ;) + as I said, it will be easier to understand and fix bugs, if I + will go step by step, and I will be able to see where bugs appears + so you should help me with that + and I tend to disagree + but you could always read my blog. you'll learn lots of things + about bootstrapping a hurd system + but it's complicated + http://www.linuxfromscratch.org/ + also, you'll need at least four hurd servers before you'll + actually see much + five + yeah, i know lfs + if somebody is interested in creating such a project, let me + know + you seem to be interested + yes, but I need the a real hurd master to help me + become one. fix your system and get to know it + I need knowledge, somebody built the system but didn't write + documentation about it, I have to extract it from your heads + hurdmaster: extract something from here + http://teythoon.cryptobitch.de + I need my head ;) + thanks + okay, what's the smallest thing I can run? + life of a Hurd system starts with the root filesystem, and the + exec server is loaded but not started + you could get rid of the exec server and replace the root + filesystem with your own program + statically linked, uses no unix stuff, only mach stuff + can I get 'hello world' on pure mach? + you could + hurdmaster: actually, here it is: + http://darnassus.sceen.net/gitweb/rbraun/mach_print.git/ + compile it statically, put it somewhere in /boot + make sure you're running a debug kernel + load it from grub instead of /hurd/ext2fs.static + look at the grub config for how this is done + let me know if it worked ;) + + +# IRC, freenode, #hurd, 2014-03-04 + + Can I run a single instance of hurd on multiple computers + With them acting as different servers? + no + Like the fs server on one pc etc. + Which os could I do this with? + I assumed Mach RPC would support that. + it can + but we don't use it that way + plan9 is probably better suited to what you want + inferno too + maybe dragonflybsd + Yep. + irAwesome. + Plan9 is exactly it. + enjoy + + +# IRC, freenode, #hurd, 2014-03-11 + + Does anyone have a distributed OS over GNU/hurd project running? + (GNU/hurd has many of the utilities to make this easy, but it still + requires some more utilities for transparent computation) + not at the moment, no + and i consider our ipc inappropriate if you want system able to + run over heterogeneous hardware + or rather, our RPCs + I haven't spent the time, this is speculative (in the worse "let's do + everything magically!" way.) + Just wondering if this exists outside of plan9 (which is limited in + some ways.) + dragonflybsd had plans for a SSI + there are ancient research systems that actually did the job + such as amoeba + here at the hurd, we just don't have the manpower, and the people + spending time on the project have other interests + Yeah, that seems like a large problem. + GNU/hurd is self hosting (in the "I like working on it" way), yes? + I've done some work on it, but don't really know how nice it is. + yes it is + Working from a microkernel to add pseudo-SSI features to a bunch of + servers seems like a much more trivial task than, say, modifying TLK. + posix conformance and stability are good enough that more than 70% + of debian packages build and most of them work fine + tlk the linux kernel ? + Yes. + first time i see this acronym :) + and yes i agree, a microkernel is much much more suited for that + but then, i consider a microkernel better suited for practically + everything ... :) + :) + I'm wondering how to mix SSI with network-awareness. + mach used to have a network server + which would merely act as a proxy for capabilities + network drivers were in kernel though + That's the simple way of sharing the sources. + I'm wondering how we can make a software stack that's network aware; + completely transparent SSI can lead to inefficiencies in userspace, as it + may do things the kernels won't expect. Having to deal with the network + through a network server is a headache. + what kind of problems do you have in mind ? + Still working on defining the problem. I think that's half the + problem. + (For any problem.) + Beyond that, it's just some coding ;) + ok + sounds interesting :) + i'd love to see a modern SSI in action + but that's really a secondary goal for me so glad to see someone + making this his primary goal + doctoral thesis ? + ... Undergrad who's been hacking away since grade school. + heh :) + 18 y/o sophomore at a respected technical college, dealing with + boredom :) + well throroughly thinking about "defining the problem" is an + excellent reflex + :) stick around, the hurd is fun + it does help fight boredom a lot indeed ...... ) + :) + maybe it'd be possible to port the relevant features from plan9 now + that there is a gpl'ed version + either way, we'd need network-transparent mach messaging + which mach messaging could do, but gnumach does not implement + this currently + teythoon: afaiui if there was a proper 9fs2000 hurd server the rest + could be hidden behind the curtains + ah, well, that sounds like a 9p network filesystem translator + teythoon: also iirc plan9 uses libmach for some things so i suppose + a port wouldn't be completely impossible + given that in plan9 everything is a file, that might be enough + to use plan9 services + teythoon: yes, it'd be the easiest route (at least initially) i + believe + careful, lots of stuff is named mach-something + bloody ernest mach and his damned famous-ness-ish + =) + :D -- cgit v1.2.3