From 49a086299e047b18280457b654790ef4a2e5abfa Mon Sep 17 00:00:00 2001 From: Samuel Thibault Date: Wed, 18 Feb 2015 00:58:35 +0100 Subject: Revert "rename open_issues.mdwn to service_solahart_jakarta_selatan__082122541663.mdwn" This reverts commit 95878586ec7611791f4001a4ee17abf943fae3c1. --- .../anatomy_of_a_hurd_system.mdwn | 1430 -------------------- 1 file changed, 1430 deletions(-) delete mode 100644 service_solahart_jakarta_selatan__082122541663/anatomy_of_a_hurd_system.mdwn (limited to 'service_solahart_jakarta_selatan__082122541663/anatomy_of_a_hurd_system.mdwn') diff --git a/service_solahart_jakarta_selatan__082122541663/anatomy_of_a_hurd_system.mdwn b/service_solahart_jakarta_selatan__082122541663/anatomy_of_a_hurd_system.mdwn deleted file mode 100644 index 496d2a06..00000000 --- a/service_solahart_jakarta_selatan__082122541663/anatomy_of_a_hurd_system.mdwn +++ /dev/null @@ -1,1430 +0,0 @@ -[[!meta copyright="Copyright © 2011, 2012, 2013, 2014 Free Software Foundation, -Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -[[!taglink open_issue_documentation]] - -A bunch of this should also be covered in other (introductionary) material, -like Bushnell's Hurd paper. All this should be unfied and streamlined. - -[[!toc]] - - -# IRC, freenode, #hurd, 2011-03-08 - - I've a question on what are the "units" in the hurd project, if - you were to divide them into units if they aren't, and what are the - dependency relations between those units(roughly, nothing too pedantic - for now) - there is GNU Mach (the microkernel); there are the server - libraries in the Hurd package; there are the actual servers in the same; - and there is the POSIX implementation layer in glibc - relations are a bit tricky - Mach is the base layer which implements IPC and memory management - hmm I'll probably allocate time for dependency graph generation, - in the worst case - on top of this, the Hurd servers, using the server libraries, - implement various aspects of the system functionality - client programs use libc calls to use the servers - (servers also use libc to communicate with other servers and/or - Mach though) - so every server depends solely on mach, and no other server? - s/mach/mach and/or libc/ - I think these things should be pretty clear one you are somewhat - familiar with the Hurd architecture... nothing really tricky there - no - servers often depend on other servers for certain functionality - - -# Bootstrap - -## [[hurd_init]] - -## IRC, freenode, #hurd, 2011-03-12 - - when mach first starts up, does it have some basic i/o or fs - functionality built into it to start up the initial hurd translators? - I/O is presently completely in Mach - filesystems are in userspace - the root filesystem and exec server are loaded by grub - o I see - so in order to start hurd, you would have to start mach and - simultaneously start the root filesystem and exec server? - not exactly - GRUB loads all three, and then starts Mach. Mach in turn starts - the servers according to the multiboot information passed from GRUB - ok, so does GRUB load them into ram? - I'm trying to figure out in my mind how hurd is initially started - up from a low-level pov - yes, as I said, GRUB loads them - ok, thanks antrik...I'm new to the idea of microkernels, but a - veteran of monolithic kernels - although I just learned that windows nt is a hybrid kernel which I - never knew! - note there's a /hurd/ext2fs.static - I belive that's what is used initially... right? - yes - loading the shared libraries in addition to the actual server - would be unweildy - so the root FS server is linked statically instead - what does the root FS server do? - well, it serves the root FS ;-) - it also does some bootstrapping work during startup, to bring the - rest of the system up - - -## IRC, freenode, #hurd, 2014-01-03 - - hmpf, the hurd bootstrapping process is complicated and fragile, - maybe to the point that it is to be considered broken - aiui the hurd uses the filesystem for service lookup - older mach documentation suggests that there once existed a name - server instead for this purpose - the hurd approach is elegant and plan9ish - the problem is in the early bootstrapping - what if the root filesystem is r/o and there is no /servers or - /servers/exec ? - e. g. rm /servers/exec && reboot -> the rootfs dies early in the - hurd server bootstrap :/ - well yes - it's normal to have such constraints - uh no - at the same time, the boot protocol must be improved, if only to - support userspace disk drivers - totally unacceptable - why not ? - b/c my box just died and lost it's exec node - so ? - loosing the exec node is unacceptable - well, linux dies too if you don't have /dev populated at least a - bit - not being able to boot without the "exec" service is pretty normal - the hurd turns the vfs into a service directory - the exec service is there, only the lookup mechanism is broken - replacing the name server you mentioned earlier - yes - if you don't have services, you don't have them - i don't see the problem - the problem is the lookup mechanism getting broken - ... that easily - imagine a boot protocol based on a ramfs filled from a cpio - i do actually ;) - there would be no reason at all the lookup mechanism would break - yes - but the current situation is not acceptable - i agree - ^^ - ext2fs is too unreliable for that - but using the VFS as a directory is more than acceptable - it's probably the main hurd feature - yes - i see it rather as a circular dependency problem - and if you have good ideas, i'm all ear for propel ... :> - antrik already talked about some of them for the bootstrap - protocol - we should sum them up somewhere if not done already - i've been pondering how to install a tmpfs translator as root - translator - braunr: we could create a special translator for /servers - maybe - very much like fakeroot, it just proxies messages to a real - translator - but if operations like settrans fail, we handle them - transparently, like an overlay - i consider /servers to be very close to /dev - yes - so something like devfs seems obvious yes - i don't even think there needs to be an overlay - y not ? - why does /servers need real nodes ? - for persistence - what for ? - e.g. crash server selection - hm ok - network configuration - i personally wouldn't make that persistent - it can be configured in files and installed at boot time - me neither, but that's how it's currently done - are you planning to actually work on that soon ? - if we need no persistence, we can just use tmpfs - it wouldn't be a mere tmpfs - it could - it's a tmpfs that performs automatic discovery and registration of - system services - with some special wrapper that preserves e.g. /servers/exec - oh - so rather, devtmpfs - it is o_O :p - ? - what is what ? - well, it could be a tmpfs and some utility creating the nodes - whether the node management is merged in or separate doesn't - matter that much i guess - i'd personally imagine it merged, and tmpfs available as a - library, so that stuff like sysfs or netstatfs can easily be written - - -## IRC, freenode, #hurd, 2014-02-12 - - braunr: i fixed all fsys-related receiver lookups in libdiskfs - and surely enough the bootstrap hangs with no indication whats wrong - teythoon: use mach_print :/ - braunr: the hurd bootstrap is both fragile and hard to tweak in - interesting ways :/ - teythoon: i agree with that - teythoon: maybe this will help : - http://wiki.hurdfr.org/upload/graphviz/dot9b65733655309d059dca236f940ef37a.png - although i guess you probably already know that - heh, unicode for the win >,< - :/ - - -# Source Code Documentation - -Provide a cross-linked sources documentation, including generated files, like -RPC stubs. - - * - - -# [[Hurd_101]] - - -# [[hurd/IO_path]] - -Need more stuff like that. - - -# IRC, freenode, #hurd, 2011-10-18 - - what happens @ boot. and which translators are started in what - order? - short version: grub loads mach, ext2, and ld.so/exec; mach starts - ext2; ext2 starts exec; ext2 execs a few other servers; ext2 execs - init. from there on, it's just standard UNIX stuff - - -# IRC, OFTC, #debian-hurd, 2011-11-02 - - is __dir_lookup a RPC ?? - where can i find the source of __dir_lookup ?? - grepping most gives out rvalue assignments - -assignments - but in hurs/fs.h it is used as a function ?? - it should be the mig-generated function for that rpc - how do i know how its implemented ?? - is there any way to delve deeprer into mig-generated functions - sekon_: The MIG-generated stuff will either be found in the - package's build directory (if it's building it for themselves), or in the - glibc build directory (libhurduser, libmachuser; which are all the - available user RPC stubs). - sekon_: The implementation can be found in the various Hurd - servers/libraries. - sekon_: For example, [hurd]/libdiskfs/dir-lookup.c. - sekon_: What MIG does is provide a function call interface for - these ``functions'', and the Mach microkernel then dispatches the - invocation to the corresponding server, for example a /hurd/ext2fs file - system (via libdiskfs). - sekon_: This may help a bit: - http://www.gnu.org/software/hurd/hurd/hurd_hacking_guide.html - - -# IRC, freenode, #hurd, 2012-01-08 - - can you tell me how is done in hurd: "ls | grep x" ? - in bash - ls's standard output is a port to the pflocal server, and grep x's - standard input is a port to the pflocal server - the connexion between both ports inside the pflocal server being - done by bash when it calls pipe() - youpi, so STDOUT_FILENO, STDIN_FILENO, STDERR_FILENO still exists - ? - sure, hurd is compatible with posix - so bash 1) creates T1 (ls) and T2 (grep), then create a pipe at - the pflocal server, then connects both ends to T1 and T2, then start(T1), - start(T2) ? - not exactly - it's like on usual unix, bash creates the pipe before creating the - tasks - then forks to create both of them, handling them each side of the - pipe - ok I see - s/handling/handing/ - but when you do pipe() on linux, it creates a kernel object, this - time it's 2 port on the pflocal ? - yes - how are spawned tasks ? - with fork() ? - yes - which is just task_create() and duplicating the ports into the new - task - ok - so it's easy to rewrite fork() with a good control of duplicated - fd - about threading, mutexes, conditions, etc.. are kernel objects or - just userland objects ? - just ports - (only threads are kernel objects) - so, about efficiency, are pipes and mutexes efficient ? - depends what you call "efficient" - it's less efficient than on linux, for sure - but enough for a workable system - maybe hurd is the right place for a userland thread library like - pth or any fiber library - ? - hurd already uses a userland thread library - libcthreads - is it M:N ? - libthreads, actually - yes - -Actually, the Hurd has never used an M:N model. Both libthreads (cthreads) and libpthread use an 1:1 model. - - nice - is the task scheduler in the kernel ? - the kernel thread scheduler, yes, of course - there has to be one - are the posix open()/readdir()/etc... the direct vfs or wraps an - hurd layer libvfs ? - they wrap RPCs to the filesystem servers - the Bushnell paper is probably the closest we have to a high-level - documentation of these concepts... - the Hurd does not have a central VFS component at all. name - lookups are performed directly on the individual FS servers - that's probably the most fundamental design feature of the Hurd - (all filesystem operations actually, not only lookups) - - -## IRC, freenode, #hurd, 2012-01-09 - - youpi: are you sure cthreads are M:N ? i'm almost sure they're 1:1 - and no modern OS is a right place for any thread userspace - library, as they wouldn't have support to run threads on different - processors (unless processors can be handled by userspace servers, but - still, it requires intimate cooperation between the threading library and - the kernel/userspace server in any case - braunr: in libthreads, they are M:N - you can run threads on different processors by using several kernel - threads, there's no problem in there, a lot of projects do this - a pure userspace library can't use kernel threads - at least pth was explacitely used on systems like bsd at a time - when they didn't have kernel threads exactly for that reason - explicitely* - and i'm actually quite surprised to learn that we have an M:N - threading model :/ - why do you say "can't" ? - but i wanted to reply to abique and he's not around - of course you need kernel threads - but all you need is to bind them - well, what i call a userspace threading library is a library that - completely implement threads without the support of the kernel - or only limited support, like signals - errr, you can't implement anything with absolutely no support of - the kernel - pth used only SIGALRM iirc - asking for more kernel threads to use more processors doesn't seem - much - it's not - but i'm refering to what abique said - 01:32 < abique> maybe hurd is the right place for a userland - thread library like pth or any fiber library - well, it's indeed more, because the glibc lets external libraries - provide their mutex - while on linux, glibc doesn't - i believe he meant removing thread support from the kernel :p - ah - and replying "nice" to an M:N threading model is also suspicious, - since experience seems to show 1:1 models are better - "better" ???? - yes - well - I don't have any time to argue about that - because that'd be extremely long - simpler, so far less bugs, and also less headache concerning posix - conformance - but there's no absolute "better" here - but less performant - less flexible - that's why i mention experience :) - I mean experience too - why less performant ? - because you pay kernel transition - because you don't know anything about the application threads - etc. - really ? - yes - i fail to see where the overhead is - I'm not saying m:n is generally better than 1:1 either - thread switch, thread creation, etc. - creation is slower, i agree, but i'm not sure it's used frequently - enough to really matter - it is sometimes used frequently enough - and in those cases it would be a headache to avoid it - ok - i thought thread pools were used in those cases - synchronized with kernel mutexes ? - that's still slow - it reduces to the thread switch overhead - which, i agree is slightly slower - ok, i's a bit less performant :) - well don't futexes exist just for that too ? - yes and no - in that case they don't help - because they do sleep - they help only when the threads are living - ok - now as I said I don't have to talk much more, I have to leave :) - - -# IRC, freenode, #hurd, 2012-12-06 - - spiderweb: have you read - http://www.gnu.org/software/hurd/hurd-paper.html ? - I'll have a look. - and also the beginning of - http://ftp.sceen.net/mach/mach_a_new_kernel_foundation_for_unix_development.pdf - these two should provide a good look at the big picture the hurd - attemtps to achieve - I can't help but wonder though, what advantages were really - achieved with early mach? - weren't they just running a monolithic unix server like osx does? - most mach-based systems were - but thanks to that, they could provide advanced features over - other well established unix systems - while also being compatible - so basically it was just an ease of development thing - well that's what mach aimed at being - same for the hurd - making things easy - but as a side effect hurd actually delivers on the advantages of - microkernels aside from that, but the older systems wouldn't, correct? - that's how there could be network file systems in very short time - and very scarce resources (i.e. developers working on it), while on other - systems it required a lot more to accomplish that - no, it's not a side effect of the microkernel - the hurd retains and extends the concept of flexibility introduced - by mach - the improved stability, etc. isn't a side effect of being able to - restart generally thought of as system-critical processes? - no - you can't restart system critical processes on the hurd either - that's one feature of minix, and they worked hard on it - ah, okay. so that's currently just the domain of minix - okay - spiderweb: well, there's 1 advantage of minix for you :P - the main idea of mach is to make it easy to extend unix - without having hundreds of system calls - -[[/system_call]]. - - the hurd keeps that and extends it by making many operations - unprivileged - you don't need special code for kernel modules any more - it's easy - you don't need special code to handle suid bits and other ugly - similar hacks, - it's easy - you don't need fuse - easy - etc.. - - -# Service Directory - -## IRC, freenode, #hurd, 2012-12-06 - - what is the #1 feature that distinguished hurd from other - operating systems. the concept of translators. (will read more when I get - more time). - yes, translators - using the VFS as a service directory - and the VFS permissions to control access to those services - - -## IRC, freenode, #hurd, 2013-05-23 - - Hi, is there any efficient way to control which backed - translators are called via RPC with a user space program? - Take for example io_stat: S_io_stat is defined in boot/boot.c, - pfinet/io-ops.c and pflocal/io.c - And the we have libdiskfs/io-stat.c:diskfs_S_io_stat, - libnetfs/io-stat.c:netfs_S_io_stat, libtreefs/s-io.c:treefs_S_io_stat, - libtrivfs/io-stat.c:trivfs_S_io_stat - How are they related? - gnu_srs: it depends on the server (translator) managing the files - (nodes) you're accessing - so use fsysopts to know the server, and see what this server uses - fsysopts /hurd/pfinet and fsysopts /hurd/pflocal gives the same - answer: ext2fs --writable --no-inherit-dir-group --store-type=typed - device:hd0s1 - of course - the binaries are regular files - see /servers/socket/1 and /servers/socket/2 instead - which are the nodes representing the *service* - again, the hurd uses the file system as a service directory - this usage of the file system is at the core of the hurd design - files are not mere files, they're service names - it happens that, for most files, the service behind them is the - same as for regular files - gnu_srs: this *must* be obvious for you to do any tricky work on - the hurd - - Anyway, if I create a test program calling io_stat I assume - S_io_stat in pflocal is called. - How to make the program call S_io_stat in pfinet instead? - create a socket managed by pfinet - i.e. an inet or inet6 socket - you can't assume io_stat is serviced by pflocal - only stats on unix sockets of pipes will be - or* - thanks, what about the *_S_io_stat functions? - what about them ? - How they fit into the picture, e.g. diskfs_io_stat? - *diskfs_S_io_stat - gnu_srs: if you open a file managed by a server using libdiskfs, - e.g. ext2fs, that one will be called - Using the same user space call: io_stat, right? - it's all userspace - say rather, client-side - the client calls the posix stat() function, which is implemented - by glibc, which converts it into a call to io_stat, and sends it to the - server managing the open file - the io_stat can change depending on the server - the remote io_stat implementation, i mean - identify the server, and you will identify the actual - implementation - - -## IRC, freenode, #hurd, 2013-06-30 - - hi, what is the replacer of netname_check_in? - - I want to ask another question. in my opinion, the rpc is the - mach's way, and the translator is the hurd's way. so somebody want to - lookup a service, it should not need to ask the mach kernel know about - this query. the hurd will take the control. - am I right? - no - that's nonsense - service lookups has never been in mach - first mach based systems used a service directory, whereas the - hurd uses the file system for that - you still need mach to communicate with either of those - how to understand the term of service directory here? - a server everyone knows - which gives references to other servers - usually, depending on the name - e.g. name_lookup("net") -> port right to network server - is that people use netname_check_in to register service in the - past? now used libtrivfs? - i don't know about netname_check_in - old mach (not gnumach) documentation might mention this service - directory - libtrivfs doesn't have much to do with that - on the hurd, the equivalent is the file system - maybe that is outdate, I just found that exist old doc, and old - code which can't be build. - every process knows / - the file system is the service directory - nodes refer to services - so the file system is the nameserver, any new service should - register in it before other can use - and the file system is distributed, so looking up a service may - require several queries - setting a translator is exactly that, registering a program to - service requests on a node - the file system isn't one server though - programs all know about /, but then, lookups are recursive - e.g. if you have / and /home, and are looking for - /home/hacklu/.profile, you ask / which tells you about /home, and /home - will give you a right to /home/hacklu/.profile - even in the past, the mach don't provide name register service, - there must be an other server to provide this service? - yes - what's nonsense in your sentence is comparing RPCs and translators - translators are merely servers attached to the file system, using - RPCs to communicate with the rest of the system - I know yet, the two just one thing. - no - two things :p - completely different and unrelated except for one using the other - ah, just one used aonther one. - is exist anyway to anounce service except settrans with file node? - more or less - tasks can have special ports - that's how one task knows about / for example - at task creation, a right to / is inserted in the new task - I think this is also a file node way. - no - if i'm right, auth is referenced the same way - and there is no node for auth - how the user get the port of auth with node? - it's given when a task is created - pre-set in the creation of one task? - i'm unconfortable with "pre-set" - inserted at creation time - auth is started very early - then tasks are given a reference to it - - -# IRC, freenode, #hurd, 2012-12-10 - - I want to work on hurd, but I think I'm going to start with - minix, I own the minix book 3rd ed. it seems like a good intro to - operating systems in general. like I don't even know what a semaphore is - yet. - well, enjoy learning :) - once I finish that book, what reading do you guys recommend? - other than the wiki - i wouldn't recommend starting with a book that focuses on one - operating system anyway - you tend to think in terms of what is done in that specific - implementation and compare everything else to that - tannenbaum is not only the main author or minix, but also the one - of the book http://en.wikipedia.org/wiki/Modern_Operating_Systems - - http://en.wikipedia.org/wiki/List_of_important_publications_in_computer_science#Operating_systems - should be a pretty good list :) - - -# IRC, freenode, #hurd, 2013-03-12 - - i have a question regarding ipc in hurd. if a task is created, does - it contain any default port rights in its space? i am trying to deduce - how one calls dir_lookup() on the root translator in glibc's open(). - mjjc: yes, there are some default port rights, but I don't - remember the details :/ - kilobug: do you know where i should search for details? - mjjc: hum either in the Hurd's hacking guide - https://www.gnu.org/software/hurd/hacking-guide/ or directly in the - source code of exec server/libc I would say, or just ask again the - question here later on to see if someone else has more information - ok, thanks - there's also rpctrace to, as the name says, trace all the rpc's - executed - some ports are introduced in new tasks, yes - see - http://www.gnu.org/software/hurd/hacking-guide/hhg.html#The-main-function - and - - http://www.gnu.org/software/hurd/gnumach-doc/Task-Special-Ports.html#Task-Special-Ports - yes, the second link was just what i was looking for, thanks - the second is very general - also, the first applies to translators only - if you're looking for how to do it for a non-translator - application, the answer is probably somewhere in glibc - _hurd_startup i'd guess - - -# IRC, freenode, #hurd, 2013-06-15 - - ive been reading a little about exokernels or unikernels, and i - was wondering if it might be relevant to the GNU/hurd design. I'm not - too familiar with hurd terminology so forgive me. what if every - privileged service was compiled as its own mini "kernel" that handled (a) - any hardware related to that service (b) any device nodes exposed by that - service etc... - yes but not really that way - under the current hurd model of the operating system, how would - you talk to hardware that required specific timings like sound hardware? - through mapped memory - is there such a thing as an interrupt request in hurd? - obviously - ok - is there any documentation i can read that involves a driver that - uses irqs for hurd? - you can read the netdde code - dde being another project, there may be documentation about it - somewhere else - i don't know where - thanks - i read a little about dde, apparently it reuses existing code from - linux or bsd by reimplementing parts of the old kernel like an api or - something - yes - it must translate these system calls into ipc or something - then mach handles it? - exactly - -[[/system_call]]. - - that's why i say it's not the exokernel way of doing things - ok - so does every low level hardware access go through mach?' - yes - well no - interrupts do - ports (on x86) - everything else should be doable through mapped memory - seems surprising that the code for it is so small - 1/ why surprising ? and 2/ "so small" ? - its like the core of the OS, and yet its tiny compared to say the - linux kernel - it's a microkenrel - well, rather an hybrid - the size of the equivalent code in linux is about the same - ok - with the model that privileged instructions get moved to - userspace, how does one draw the line between what is OS and what is user - code - privileged instructions remain in the kernel - that's one of the few responsibilities of the kernel - i see, so it is an illusion that the user has privilege in a sense - hum no - or, define "illusion" - well the user can suddenly do things never imaginable in linux - that would have required sudo - yes - well, they're not unimaginable on linux - it's just not how it's meant to work - :) - and why things like fuse are so slow - i still don't get "i see, so it is an illusion that the user has - privilege in a sense" - because the user doesnt actually have the elevated privilege its - the server thing (translator)? - it does - not at the hardware level, but at the system level - not being able to do it directly doesn't mean you can't do it - right - it means you need indirections - that's what the kernel provides - so the user cant do stuff like outb 0x13, 0x1 - he can - he also can on linux - oh - that's an x86 specifity though - but the user would need hardware privilege to do that - no - or some kind of privilege - there is a permission bitmap in the TSS that allows userspace to - directly access some ports - but that's really x86 specific, again - i was using it as an example - i mean you wouldnt want userspace to directly access everything - yes - the only problem with that is dma reall - y - because dma usually access physical memory directly - are you saying its good to let userspace access everything minus - dma? - otherwise you can just centralize permissions in one place (the - kernel or an I/O server for example) - no - you don't let userspace access everything - ah - yes - userspace asks for permission to access one specific part (a - memory range through mapping) - and can't access the rest (except through dma) - except through dma?? doesnt that pose a large security threat? - no - you don't give away dma access to anyone - only drivers - ahh - and drivers are normally privileged applications anyway - so a driver runs in userspace? - so the only effect is that bugs can affect other address spaces - indirectly - netdde does - interesting - and they all should but that's not the case for historical reasons - i want to port ALSA to hurd userspace :D - that's not so simple unfortunately - one of the reasons it's hard is that pci access needs arbitration - and we don't have that yet - i imagine that would be difficult - yes - also we're not sure we want alsa - alsa drivers, maybe, but probably not the interface itself - its tangled spaghetti - but the guy who wrote JACK for audio hates OSS, and believes it is - rubbish due to the fact it tries to read and write to a pcm device node - like a filesystem with no care for timing - i don't know audio well enough to tell you anything about that - was that about oss3 or oss4 ? - also, the hurd isn't a real time system - so we don't really care about timings - but with "good enough" latencies, it shouldn't be a problem - but if the audio doesnt reach the sound card in time, you will get - a crackle or a pop or a pause in the signal - yep - it happens on linux too when the system gets some load - some users find this unnacceptable - some users want real time systems - using soft real time is usually plenty enough to "solve" this kind - of problems - will hurd ever be a real time system? - no idea - if somebody works on it why not - it's the same as linux - it should certainly be simpler than on linux though - hmm - microkernels are well suited for real time because of the well - defined interfaces they provide and the small amount of code running in - kernel - that sounds promising - you usually need to add priority inheritance and take care of just - a few corner cases and that's all - but as youpi said, it still requires work - and nobody's working on it - you may want to check l4 fiasco.oc though - - -# System Personality - -## IRC, freenode, #hurd, 2013-07-29 - - over the past few days I gained a new understanding of the Hurd - teythoon: really ? :) - teythoon: That it's a complex and distributed system? ;-) - And at the same time a really simple one? - ;-D - it's just a bunch of mach programs and some do communicate and - behave in a way a posix system would, but that is more a convention than - anything else - tschwinge: yes, kind of simple and complex :) - the right terminology is "system personality" - 11:03 < teythoon> over the past few days I gained a new - understanding of the Hurd - teythoon: still no answer on that :) - braunr: ah, I spent lot's of time with the core servers and - early bootstrapping and now I gained the feeling that I've seen the Hurd - for what it really is for the first time - - -# RPC Interfaces - -## IRC, freenode, #hurd, 2013-09-03 - - I'm a little confused by the hurd and incubator git repos. - DDE is only found in the dde branch in incubator, but not in the - hurd repo. - Does this mean that DDE is not ready for master yet? - yes - If DDE is not yet used in the hurd (except in the dde branch in - the incubator repo), does pfinet use some custom glue code to use the - Linux drivers? - this has nothing to do with pfinet - pfinet is the networking stack, netdde are the networking drivers - the interface between them doesn't change, whether drivers are in - kernel or not - I see - - -# IRC, freenode, #hurd, 2013-09-20 - - HI there, I have no previous knowledge about OS's. I'm trying to - undestand the structure of the Hurd and the comparison between, say, - Linux way of managing stuff ... - for instance, I read: "Unlike other popular kernel software, the - Hurd has an object-oriented structure that allows it to evolve without - compromising its design." - that means that while for adding feature to the Linux-kernel you - have to add some stuff `inside` a procedure, whilst in the Hurd kernel - you can just, in principle at least, add an object and making the kernel - using it?... - Am I making stuff too simple? - Thanks - not exactly - unix historically has a "file-oriented" structure - the hurd allows servers to implement whatever type they want, - through the ability to create custom interfaces - custom interfaces means custom calls, custom semantics, custom - methods on objects - you're not restricted to the set of file interfaces (open, seek, - read, write, select, close, etc..) that unix normally provides - braunr: uhm ...some example? - see processes for example - see - http://darnassus.sceen.net/gitweb/savannah_mirror/hurd.git/tree/HEAD:/hurd - this is the collection of interfaces the hurd provides - most of them map to unix calls, because gnu aims at posix - compatibility too - some are internal, like processes - or authentication - but most importantly, you're not restricted to that, you can add - your own interfaces - on a unix, you'd need new system calls - or worse, extending through the catch-all ioctl call - braunr: mhn ...sorry, not getting that. - what part ? - ioctl has become such a mess :s - braunr: when you say that Unix is `file-oriented` you're - referring to the fact that sending/receiving data to/from the kernel is - designed like sending/receiving data to/from a file ...? - not merely sending/receiving - note how formatted your way of thinking is - you directly think in terms of sending/receiving (i.e. read and - write) - braunr: (yes) - that's why unix is file oriented, access to objects is done that - way - on the hurd, the file interface is one interface - there is nothing preventing you from implementing services with a - different interface - as a real world example, people interested in low latency - profesionnal audio usually dislike send/recv - see - http://lac.linuxaudio.org/2003/zkm/slides/paul_davis-jack/unix.html for - example - braunr: how big and messy ioctl has become is a good proof that - the Unix way, while powerful, does have its limits - giuscri: keep in mind the main goal of the hurd is extensibility - without special privileges - braunr: privileges? - root - braunr: what's wrong with privileges? - they allow malicious/buggy stuff to happne - and have dramatic effects - braunr: you're obviously *not* referring to the fact that once - one have the root privileges could change some critical-data - ? - i'm referring to why privilege separation exists in the first - place - if you have unprivileged users, that's because you don't want them - to mess things up - on unix, extending the system requires privileges, giving those - who do it the ability to destroy everything - braunr: yes, I think the same - the hurd is designed to allow unprivileged users to extend their - part of the system, and to some extent share that with other users - although work still remains to completely achieve that - braunr: mhn ...that's the `server`-layer between the - single-application and kernel ? - the multi-server based approach not only allows that, but - mitigates damage even when privileged servers misbehave - one aspect of it yes - but as i was just saying, even root servers can't mess things too - much - for example, our old (sometimes buggy) networking stack can be - restarted when it behaves wrong - the only side effect being some applications (ssh and exim come to - mind) which need to be restarted too because they don't expect the - network stack to be restarted - braunr: ...instead? - ? - giuscri: on Linux, if the network stack crash/freezes, you don't - have any other option than rebooting the system - usually with a nice - "kernel pani" - giuscri: and you may even get filesystem corruption "for free" in - the bundle - and hoping it didn't corrupt something important like file system - caches before being flushed - kilobug, braunr : mhn, ook - - -# IRC, freenode, #hurd, 2013-10-13 - - ahh, ^c isn't working to cancel a ping - is there alternative? - ahungry: ctrl-c does work, you just missed something somewhere and - are running a shell directly on a console, without a terminal to handle - signals - - -# IRC, freenode, #hurd, 2013-11-04 - - nalaginrut: you can't use the hurd for real embedded stuff without - a lot of work on it - but the hurd design applies very well to embedded environments - the fact that we're able to dynamically link practically all hurd - servers against the c library can visibly reduce the system code size - it also reduces the TCB - what about the memory occupation? - code size is about memory occupation - also, the system is composable like lego, don't need tcp - don't - include pfinet then - the memory overheald of a capability based system like the hurd - are, well, capabilities - teythoon: that's not an argument compared to modular kernels like - linux - yes it is - why ? - if you don't need tcp in linux, you just don't load it - same thing - ok, right - on the other hand, a traditional unix kernel can never be linked - against the c library - much less dynamically - right - I think the point is that it's easy to cut, since it has - better modularity than monolithic, and could be done in userland relative - easier - modularity isn't better - that's a big misconception - also, restarting components is easier on a distributed system - on the hurd, this is a side effect - and it doesn't apply well - braunr: oops, misconception - many core servers such as proc, auth, exec, the root fs server - can't be restarted at all - not yet - and servers like pfinet can be restarted, but at the cost of posix - servers not expecting that - looping on errors such as EBADF because the target socket doesn't - exist any more - I've been working on a restartable exec server during some of my - gsoc weekends - ah right - linux has kexec - and can be patched at run time - sounds like Hurd needs something similar to generalizable - continuation - so again, it's not a real advantage - no - sorry serilizable - that would persistence - personally, i don't want it at all - yes it is a real advantage, b/c the means of communication - (ports) is common to every IPC method on Hurd, and ports are first class - objects - so preserving the state is much easier on Hurd - if a monolithic kernel can do it too, it's not a real advantage - yes, but it is more work - that is one true advantage of the hurd - but don't reuse it each time - oh, that's nice for the ports - why not? - what we're talking about here is resilience - the fact that it's easier to implement doesn't mean the hurd is - better because it has resilience - it simply means the hurd is better because it's easier to - implement things on it - same for development in general - debugging - virtualization - etc.. - yes, but why we stick to compare it to monolithic - but it's still *one* property - well, minix advertises this feature a lot, even if minix can - only restart very simple things like printer servers - minix sucks - let them advertise what they can - ^^ - it has cool features, that's enough, no need to find a feature - that monolithic can never done - no it's not enough - minix isn't a general purpose system - let's just not compare it to general purpose systems - - -# IRC, freenode, #hurd, 2013-11-08 - - and, provided you have suitable language bindings, you can - replace almost any hurd server with your own implementation in any - language - teythoon: language bindings? - Do you mean language bindings against C libraries? - either that or for the low level mach primitives - For your information, IPC is independent of languages. - sure, that's the beauty - Why is hurd best for replacing parts written in C with other - languages? - because Hurd consists of many servers, each server managing one - kind of resource - so you have /hurd/proc managing posix processes - you could reimplement /hurd/proc in say python or go, and - replace just that component of the Hurd system - you cannot do this with any other (general purpose) operating - system that I know of - you could incrementally replace the Hurd with your own - Hurd-compatible set of servers written in X - use a language that you can verify, i.e. prove that a certain - specification is fulfilled, and you end up with an awesome stable and - secure operating system - Any microkernel OS fits the description. - teythoon, Does hurd have formal protocols for IPC communications? - sure, name some other general purpose and somewhat - posix-compatible microkernel based operating system please - what do you mean by formal protocols ? - IPC communications need to be defined in documents. - the "wire" format is specified of course, the semantic not so - much - network protocols exist. - HTTP is a transport protocol. - Without formal protocols, IPC communications suffer from - debugging difficulties. - Formal protocols make it possible to develop and test each module - independently. - as I said, the wire format is specified, the semantics only in - written form in the source - this is an example of the ipc specification for the proc server - http://git.savannah.gnu.org/cgit/hurd/hurd.git/tree/hurd/process.defs - teythoon, how file server interacts with file clients should be - defined as a formal protocol, too. - do you consider the ipc description a kind of formal protocol ? - - http://git.savannah.gnu.org/cgit/hurd/hurd.git/tree/hurd/process.defs can - be considered as a formal protocol. - However, the file server protocol should be defined on top of IPC - protocol. - the file server protocol is in fs.defs - every protocol spoken is defined in that ipc description - language - it is used to derive code from - crocket: not any system can be used to implement system services - in any language - in theory, they do, but in theory only - the main reason they don't is because most aren't posix compliant - from the ground up - posix compliance is achieved through virtualization - which isolates services too much for them to get useful, - notwithstanding the impacts on performance, memory, etc.. - braunr, Do you mean it's difficult to achieve POSIX compliance - with haskell? - crocket: i mean most l4 based systems aren't posix - genode isn't posix - helenos is by design not posix - the hurd is the only multi server system providing such a good - level of posix conformance - and with tls on the way, we'll support even more non-posix - applications that are nonetheless very common on unices because of - historical interfaces still present, such as mcontext - and modern ones - e.g. ruby is now working, go should be there after tls - * teythoon drools over the perspective of having go on the Hurd... - braunr, Is posix relevant now? - it's hugely relevant - conforming to posix and some native unix interfaces is the only - way to reuse a lot of existing production applications - and for the matter at hand (system services not written in c), it - means almost readily getting runtimes for other languages than c - something other microkernel based system will not have - imagine this - one day, one of us could create a company for a hurd-like system, - presenting this idea as the killer feature - by supporting posix, customers could port their software with very - little effort - *very little effort* is what makes software attractive - - http://stackoverflow.com/questions/1806585/why-is-linux-called-a-monolithic-kernel/1806597#1806597 - says "The disadvantage to a microkernel is that asynchronous IPC - messaging can become very difficult to debug, especially if fibrils are - implemented." - " GNU Hurd suffers from these debugging problems (reference)." - stackoverflow is usually a nice place - but concerning microkernel stuff, you'll read a lot of crap - anywhere - whether it's sync or async, tracking references is a hard task - it's a bit more difficult in distributed systems, but not that - much if the proper debugging features are provided - we actually don't suffer from that too much - many of us have been able to debug reference leaks in the past, - without too much trouble - we lack some tools that would give us a better view of the system - state - braunr, But is it more difficult with microkernel? - crocket: it's more difficult with distributed systems - How much more difficult? - i don't know - distributed systems - not much - braunr, How do you define distributed systems? - crocket: not monolithic - braunr, Hurd is distributed, then. - multiserver if you prefer - yes it is - braunr, So it is more difficult with hurd. - How much more difficult? How do you debug? - just keep in mind that a monolithic system can run on a - microkenrel - we use tools that show us references - braunr, like? - like portinfo - braunr, Does hurd use unix-socket to implement IPC? - no - unix-socket use mach ipc - I'm confused - ipc is provided by the microkernel, gnumach (a variant of mach) - unix sockets are provided by one of the hurd servers (pflocal) - servers and clients communicate through mach ipc - braunr, Do you think it's feasible to build servers in haskell? - why not ? - ok - I've been thinking about that - in go, with cgo, you can call go functions from c code - so it should be possible to create bindings for say libtrivfs - I'd like to write an OS in clojure or haskell. - crocket: what for ? - braunr, I want to see a better system programming language than - C. - i don't see how clojure or haskell would be "better system - programming languages" than c - and even assuming that, what for ? - braunr, It's better for programmers. - haskell - haskell is expressive. - personally i disagree - it's better for some things - not for system programming - For system programming, Google Go is trying to replace C. But I - doubt it will. - we may not be referring to the same thing here when we say "system - programming" - braunr, What do you think is a better one? - crocket: i don't think there is a better one currently - braunr, Even Rust and D? - i don't know them well enough - certainly not D if it's what i think it is - C is too slow. - C is too slow to develop. - depends - again, i disagree - rust looks good but i don't know it well to comment - C is a tank, and clojure is an airplane. - A tank is reliable but slow. - Clojure is fast but lacks some accuracy. - c is as reliable as the developer is skilled with it - it's clearly not a tank - there are many traps - crocket: are you suggesting to rewrite Hurd in Clojure? - no - Why rewrite hud? - hurd - I'd rather start from scratch. - which is what a rewrite is - I am not expert on Clojure, but I don't think it is made for - system programming. - If you want alternate language, I thing Go is only serious - candidate other than C - Or Rust - However, some people wrote OSes in haskell. - again, why ? - if it's only for the sake of using another language, i think it's - bad reason - Because haskell provides a high level of abstraction that helps - programmers. - It is more secure with monads. - If you want your OS to become successful Free Software project, - you have to use popular language. Haskell is not. - Most Haskell programmers are not into kernels - They do high level stuff. - So little contributors. - crocket: so you aim at security ? - I mean, candidats for contribution - braunr, security and higher abstraction. - i don't understand higher abstraction - braunr, FP can be useful to systems. - FP ? - functional programming - right - but you can abstract a lot with c too, with more efforts - braunr, like that's easy. - it's not that hard - i'm just questioning the goals and the solution of using a - particular language - the reason c is still the preferred language for system - programming is because it provides control over how the hardware does - stuff - which is very important for performance - the hurd never took off because of bad performance - performance doesn't mean doing things faster, it means being able - to do things or not, or doing things a new way - so ok, great, you have your amazing file system written in - haskell, and you find out it doesn't scale at all beyond some threshold - of processors or memory - braunr, L4 is fast. - l4 is merely an architecture abstraction - and it's not written in haskell :p - don't assume anything running on top of something fast will be - fast - Hurd is slow and written in C. - yes - not because of c though - Becuase it's microkernel? - because c wasn't used well enough to make the most of the hardware - in many places - far too many places - A microkernel can be as fast as a monolithic kernel according to - L4. - no - it can't - it can for very specific cases - almost none of which are real world - but that's not the problem - again, i'm questioning your choice of another language in relation - to your goals, that's all - c can do things you really can't do easily in other languages - be aware of that - braunr, "Monolithic kernel are faster than microkernel . while - The first microkernel Mach is 50% slower than Monolithic kernel while - later version like L4 only 2% or 4% slower than the Monolithic kernel ." - 14:05 < braunr> but concerning microkernel stuff, you'll read a - lot of crap anywhere - simple counterexample : - the measurements you're giving consider a bare l4 kernel with - nothing on top of it - doing thread-to-thread ipc - this model of communication is hardly used in any real world - application - one of the huge features people look for with microkernels are - capabilities - and that alone will bump your 4% up - since capabilities will be used for practically every ipc - ok - - -# Hurd From Scratch - -## IRC, freenode, #hurd, 2013-11-30 - - because I think there is no way to understand the whole pile, - you need to go step by step - for example, I'm starting with mach only, then adding one - server, then another and on each step I have working system - that's how I want to understand it - you are interested in the early bootstrapping of the hurd system - ? - now I'm starting debian gnu/mach, it hungs, show me black - screen and I have no idea how to fix it - if you are unable to fix this, why do you think you can build a - hurd system from scratch ? - not gnu/mach, gnu/hurd I mean - or, you could describe your problem in more detail and one of - the nice people around here might help you ;) - as I said, it will be easier to understand and fix bugs, if I - will go step by step, and I will be able to see where bugs appears - so you should help me with that - and I tend to disagree - but you could always read my blog. you'll learn lots of things - about bootstrapping a hurd system - but it's complicated - http://www.linuxfromscratch.org/ - also, you'll need at least four hurd servers before you'll - actually see much - five - yeah, i know lfs - if somebody is interested in creating such a project, let me - know - you seem to be interested - yes, but I need the a real hurd master to help me - become one. fix your system and get to know it - I need knowledge, somebody built the system but didn't write - documentation about it, I have to extract it from your heads - hurdmaster: extract something from here - http://teythoon.cryptobitch.de - I need my head ;) - thanks - okay, what's the smallest thing I can run? - life of a Hurd system starts with the root filesystem, and the - exec server is loaded but not started - you could get rid of the exec server and replace the root - filesystem with your own program - statically linked, uses no unix stuff, only mach stuff - can I get 'hello world' on pure mach? - you could - hurdmaster: actually, here it is: - http://darnassus.sceen.net/gitweb/rbraun/mach_print.git/ - compile it statically, put it somewhere in /boot - make sure you're running a debug kernel - load it from grub instead of /hurd/ext2fs.static - look at the grub config for how this is done - let me know if it worked ;) - - -# IRC, freenode, #hurd, 2014-03-04 - - Can I run a single instance of hurd on multiple computers - With them acting as different servers? - no - Like the fs server on one pc etc. - Which os could I do this with? - I assumed Mach RPC would support that. - it can - but we don't use it that way - plan9 is probably better suited to what you want - inferno too - maybe dragonflybsd - Yep. - irAwesome. - Plan9 is exactly it. - enjoy - - -# IRC, freenode, #hurd, 2014-03-11 - - Does anyone have a distributed OS over GNU/hurd project running? - (GNU/hurd has many of the utilities to make this easy, but it still - requires some more utilities for transparent computation) - not at the moment, no - and i consider our ipc inappropriate if you want system able to - run over heterogeneous hardware - or rather, our RPCs - I haven't spent the time, this is speculative (in the worse "let's do - everything magically!" way.) - Just wondering if this exists outside of plan9 (which is limited in - some ways.) - dragonflybsd had plans for a SSI - there are ancient research systems that actually did the job - such as amoeba - here at the hurd, we just don't have the manpower, and the people - spending time on the project have other interests - Yeah, that seems like a large problem. - GNU/hurd is self hosting (in the "I like working on it" way), yes? - I've done some work on it, but don't really know how nice it is. - yes it is - Working from a microkernel to add pseudo-SSI features to a bunch of - servers seems like a much more trivial task than, say, modifying TLK. - posix conformance and stability are good enough that more than 70% - of debian packages build and most of them work fine - tlk the linux kernel ? - Yes. - first time i see this acronym :) - and yes i agree, a microkernel is much much more suited for that - but then, i consider a microkernel better suited for practically - everything ... :) - :) - I'm wondering how to mix SSI with network-awareness. - mach used to have a network server - which would merely act as a proxy for capabilities - network drivers were in kernel though - That's the simple way of sharing the sources. - I'm wondering how we can make a software stack that's network aware; - completely transparent SSI can lead to inefficiencies in userspace, as it - may do things the kernels won't expect. Having to deal with the network - through a network server is a headache. - what kind of problems do you have in mind ? - Still working on defining the problem. I think that's half the - problem. - (For any problem.) - Beyond that, it's just some coding ;) - ok - sounds interesting :) - i'd love to see a modern SSI in action - but that's really a secondary goal for me so glad to see someone - making this his primary goal - doctoral thesis ? - ... Undergrad who's been hacking away since grade school. - heh :) - 18 y/o sophomore at a respected technical college, dealing with - boredom :) - well throroughly thinking about "defining the problem" is an - excellent reflex - :) stick around, the hurd is fun - it does help fight boredom a lot indeed ...... ) - :) - maybe it'd be possible to port the relevant features from plan9 now - that there is a gpl'ed version - either way, we'd need network-transparent mach messaging - which mach messaging could do, but gnumach does not implement - this currently - teythoon: afaiui if there was a proper 9fs2000 hurd server the rest - could be hidden behind the curtains - ah, well, that sounds like a 9p network filesystem translator - teythoon: also iirc plan9 uses libmach for some things so i suppose - a port wouldn't be completely impossible - given that in plan9 everything is a file, that might be enough - to use plan9 services - teythoon: yes, it'd be the easiest route (at least initially) i - believe - careful, lots of stuff is named mach-something - bloody ernest mach and his damned famous-ness-ish - =) - :D -- cgit v1.2.3