[[!meta copyright="Copyright © 2009, 2011, 2012, 2013, 2014 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] [[!tag open_issue_gnumach open_issue_hurd]] This is a collection of resources concerning *user-space device drivers*. Also see [[device drivers and IO systems]]. [[community/gsoc/project ideas/driver glue code]]. [[!toc levels=2]] # Open Issues ## IRQs * Can be modeled using [[RPC]]s. * Security considerations: IRQ sharing. * *Omega0* paper defines an interface. * As is can be read in the *Mach 3 Kernel Principles*, there is an *event object* facility in Mach that can be used for having user-space tasks react to IRQs. However, at least in GNU Mach, that code (`kern/eventcount.c`) doesn't seem functional at all and isn't integrated properly in the kernel. * IRC, freenode, #hurd, 2011-07-29 < antrik> regarding performance of userspace drivers, there is one thing that really adds considerable overhead: interrupt handling. whether this is relevant very much depends on the hardware in question. when sending many small packets over gigabit ethernet, it might be noticable; in most other cases it's irrelevant < youpi> some cards support interrupt coalescin < youpi> could be supported by DDE too ## DMA * Security considerations. * I/O MMU. ### IRC, freenode, #hurd, 2012-08-15 hi. does hurd support mesa? carli2: software only, but yes :( so you did not solve the problem with the CS checkers and GPU DMA for microkernels yet, right? cs = ? control stream the data sent to the gpu no and to be honest we're not currently trying to well, a microkernel containing cs checkers for each hardware is not a microkernel any more the problem is having the ability to check or rather, giving only what's necessary to delegate checking to mmus but maybe the kernel could have a smaller interface like a function to check if a memory block is owned by a process i'm not sure what you refer to about DMA-capable devices you can send messages to carli2: dma must be delegated to a trusted server linux checks the data sent to these devices, parses them and checks all pointers if they are in a memory range that the client is allowed to read/write from the client ? in linux, 3d drivers are in user space, so the kernel side checks the pointer sent to the GPU carli2: mach could do that as well well, there is a rather large part in kernel space too so in hurd I trust some drivers to not do evil things? those in the kernel yes what does "in the kernel" mean? afaik a microkernel only has memory manager and some basic memory sharing and messaging functionality did you read about the hurd ? mach is considered an hybrid kernel, not a true microkernel even with all drivers outside, it's still an hybrid although we're to move some parts into userlands :) braunr: ah, why? youpi: the vm part is too large ok the microkernel dogma is no policy inside the kernel "except scheduling because it's very complicated" but all modern systems have moved memory management outisde the kernel, leaving just the kernel abstraction inside the adress space kernel abstraction and the two components required to make it work are what l4re calls region mappers (the rough equivalent of our vm_map), which decides how to allocate regions in an address space and the pager, like ours, which are already external i'm not a OS developer, i mostly develop games, web services and sometimes I fix gpu drivers that was just FYI but yes, dma must be considered something privileged and the hurd doesn't have the infrastructure you seem to be looking for ## I/O Ports * Security considerations. ## PCI and other buses * Security considerations: sharing. ## Latency of doing RPCs * [[GNU Mach|microkernel/mach/gnumach]] is said to have a high overhead when doing RPC calls. ## System Boot A similar problem is described in [[community/gsoc/project_ideas/unionfs_boot]], and needs to be implemented. ### IRC, freenode, #hurd, 2011-07-27 < braunr> btw, was there any formulation of the modifications required to have disk drivers in userspace ? < braunr> (which would obviously need something like initrd/initramfs/whatever and may also need the root file system not to be the first task started) < braunr> hm actually, we may not need initrd < braunr> the boot loader could just load more modules < antrik> braunr: I have described all that in my thesis report... in German :-( < braunr> and the boot scripts could be adjusted to pass around the right ports < Tekk_> braunr: yeah, we could probably load a module that kciks us into userspace and starts the disk driver < braunr> modules are actualy userspace executables < Tekk_> ah < Tekk_> so what's the issue? < Tekk_> oh! I'm thinking the ext2fs server, which is already in userspce < braunr> change the file systems to tell them which underlying disk driver to use < Tekk_> mhm < braunr> s/disk/storage/ #### IRC, freenode, #hurd, 2012-04-25 btw, remember the initrd thing? I just came across task.c in libstore/ :) #### IRC, freenode, #hurd, 2013-06-24 we added a new initrd command to gnumach, to expose a new mach device, which ext2fs can open and unzip we consider replacing that with simply putting the data in a dead process s/process/task and let ext2fs read data from the task, and kill it when done ok alternatively, tmps would work with an initial .tar.gz payload that would be best for memory usage tmpfs* can't we replace the initrd concept with sub/neighbourhood? setting up tmpfs with an initial payload could be done with a bootstrap subhurd yes but it seems to me that having tmpfs being able to have an initial payload is interesting is there any advantage of the tmpfs translator prefilled with a tarball over ext2fs with copy & bunzip? memory usage ext2fs with copy&bunzip takes memory for zeroes and we have to forecast how much data might be stored (if writable) ah sure but why would it have to be in the tmpfs translator? I why not start the translator and have tar extract stuff there? with the livecd I had trouble replacing the root translator, but when using subhurds that shouldn't be a prwoblem at all I don't have a real opinion on this except that people don't usually like initrd :) 12:43 < teythoon> but why would it have to be in the tmpfs translator? I why not start the translator and have tar extract stuff there? that sounds an awful lot like an initramfs yes, exactly, without actually having an initramfs of course yep i actually prefer that way too a system on a r/o isofs cannot do much, but it can do this on the other hand, i wouldn't spend much time on a virtio disk driver for now the hurd as it is can't boot on a device that isn't managed by the kernel we'd need to change the boot protocol [[virtio]]. #### IRC, freenode, #hurd, 2013-06-28 I'm tempted to redo a livecd, simpler and without the initrd hack that youpi used for d-i initrd hack ? you mean more a la initramfs then ? no, I thought about using a r/o isofs translator, but instead of fixing that one up with a r/w overlay and lot's of firmlinks like I used to, it would just start an ext2fs translator with copy on an image stored on the iso and start a subhurd why a subhurd ? neighbourhurd even b/c back in the days I had trouble replacing / yes, that's hard subhurd would take of that for free are you sure ? somewhat i'm not, but this requires thorough thinking and i'm not there yet y would it not? just start a subhurd and let that one take over the console and let the user and d-i play nicely in that environment no hacks involved because it would require sharing things between the two system instances, and that's not easy no but the bootstrap system does nothing after launching the subhurd I mean yes, technically true, but why would it be hard to share with someone who does nothing? the context isn't well defined enough to clearly state anything if you don't use the resources of the first hurd, that's ok otherwise, it may be easy or not, i don't know yet you think it's worth a shot and see what issues crop up? sure definitely it doesn't sound complicated at all it's easy enough to the point we see something goes wrong or works completely so worth testin cool :) #### IRC, freenode, #hurd, 2014-02-10 braunr: i have a question wrt memory allocation in gnumach i made a live cd with a rather large ramdisk it works fine in qemu, when i tried it on a real machine it failed to allocate the buffer for the ramdisk i was wondering why i believe the function that failed was kmem_alloc trying to allocate 64 megabytes teythoon: how much memory on the real machine ? 4 gigs so 1.8G yes does it fail systematically ? but surely enough uh, i must admit i only tried it once it's likely a 64M kernel allocation would fail the kmem_map is 128M wide iirc and likely fragmented it doesn't take much to prevent a 64M contiguous virtual area i see i suggest you try my last gnumach patch hm surely there is a way to make this more robust, like using a different map for the allocation ? the more you give to the kernel, the less you have for userspace merging maps together was actually a goal the kernel should never try to allocate such a large region can you trace the origin of the allocation request ? i'm pretty sure it is for the ram disk makes sense but still, it's huge well... the ram disk should behave as any other mapping, i.e. pages should be mapped in on demand right, so the implementation could be improved ? we need to understand why the kernel makes such big requests first oh ? i thought i asked it to do so ? for the ram disk normally, i would expect this to translate to the creation of a 64M anonymous memory vm object the kernel would then fill that object with zeroed pages on demand (on page fault) at no time would there be a single 64M congituous kernel memory allocation such big allocations are a sign of a serious bug for reference, linux (which is even more demanding because physical memory is directly mapped in kernel space) allows at most 4M contiguous blocks on most architectures on my systems, the largest kernel allocation is actually 128k and there are only two such allocations teythoon: i need you to reproduce it so we understand what happens better braunr: currently the ramdisk implementation kmem_allocs the buffer in the kernel_map hum did you add this code ? no where is it ? debian/patches ugh heh ok, don't expect that to scale it's a quick and dirty hack teythoon: why not use tmpfs ? i use it as root filesystem :/ ok so update on what i said before kmem_map is exclusively used for kernel object (slab) allocations kmem_map is a submap of kernel_map which is 192M on i386 so a 64M allocation can't work at all it would work on xen, where the kernel map is 224M large teythoon: do you use xen ? ok, thanks for the pointers :) i don't use xen then i can't explain how it worked in your virtual machine unless the size was smaller i'll look into improving the ramdisk patch if time permits no it wasnt :/ and it works reliably in qemu that's very strange unless the kernel allocates nothing at all inside kernel_map on qemu ##### IRC, freenode, #hurd, 2014-02-11 braunr: http://paste.debian.net/81339/ teythoon: oO ? teythoon: you can't allocate memory from a non kernel map what you're doing here is that you create a separate, non-kernel address space, that overlaps kernel memory, and allocate from that area it's like having two overlapping heaps and allocating from them braunr: i do? o_O so i need to map it instead ? teythoon: what do you want to do ? i'm currently reading up on the vm system, any pointers ? teythoon: but what do you want to achieve here ? 12:24 < teythoon> so i need to map it instead ? i'm trying to do what you said the other day, create a different map to back the ramdisk no no ? i said an object, not a map but it means a complete rework ok i'll head back into hurd-land then, though i'd love to see this done properly teythoon: what you want basically is tmpfs as a rootfs right ? sure i'd need a way to populate it though how is it done currently ? grub loads an ext2 image, then it's copied into the ramdisk device, and used by the root translator how is it copied ? what makes use of the kernel ramdisk ? in ramdisk_create, currently via memcpy the ext2fs translator that provides / ah so it's a kernel device like hd0 ? yes hm ok then you could create an anonymous memory object in the kernel, and map read/write requests to object operations the object must not be mapped in the kernel though, only temporary on reads/writes right so i'd not use memcpy, but one of the mach functions that copy stuff to memory objects ? i'm not sure you could simply map the object, memcpy to/from it, and unmap it what documentation should i read ? vm/vm_map.h for one i can only find stuff describing the kernel interface to userspace vm/vm_kern.h may help copyinmap and copyoutmap maybe hm no vm_map.h isn't overly verbose :( vm_map_enter/vm_map_remove ah, i actually tried vm_map_enter look at the .c files, functions are described there that leads to funny results vm_map_enter == mmap basically and vm_object.h panic: kernel thread accessed user space! heh :) right, i hoped vm_map_enter to be the in-kernel equivalent of vm_map braunr: uh, it worked teythoon: ? weird :) teythoon: what's happening ? i refined the ramdisk patch, and it seems to work not sure if i got it right though, i'll paste the patch yes please http://paste.debian.net/81376/ no it can't work either :/ you can't map the complete object (amusingly it does) you have to temporarily map the pages you want to access it does for the same obscure reason the previous code worked on qemu ok, i think i see increase the size a lot more like 512M and see you could also use the kernel debugger to print the kernel map before and after mapping how ? hm see show task maybe you can call the in kernel function directly with the kernel map as argument which one ? the one for "show task" hm no it shows threads, show map and show map crashes on darnassus .. here as well ugh personally i'd use something like vm_map_info in x15 but you may not want to waste time with that try with a bigger size and see what it does, should be quick and simple enough right braunr: ok, you were right, mapping the entire object fails if it is too big teythoon: fyi, kmem_alloc and vm_map have some common code, namely the allocation of an virtual area inside a vm_map kmem_alloc requires a kernel map (kernel_map or a submap) whereas vm_map can operate on any map what differs is the backing store braunr: i believe i want to use vm_object_copy_slowly to create and populate the vm object for that, i'd need a source vm_object the data is provided as a multiboot_module kmem_alloc backs the virtual range with wired down physical memory whereas vm_map maps part of an object that is usually pageable i see and you probably want your object to be pageable here yes :) yes object copy functions could work let me check what would i specify as source object ? let's assume a device write the source object would be where the source data is e.g. the data provided by the user yes trouble is, i'm not sure what the source is it looks a bit complicated yes i mean the boot loader put it into memory, not sure what mach makes of that i guess there already are device functions that look up the object from the given address it's anonymous memory but that's not the problem here so i need to create a memory object for that ? you probably don't want to populate your ramdisk from the kernel wire it down to the physical memory ? don't bother with the wire property oh ? if it can't be paged out, it won't be ah, that's not what i meant you probably want ext2fs to populate it, or another task loaded by the boot loader interesting idea and then, this task will have a memory object somewhere imagine a task which sole purpose is to embedd an archive to extract into the ramdisk sweet, my thoughts exactly :) the data section of a program will be backed by an anonymous memory object the problem is the interface the device interface passes addresses and sizes you need to look up the object from that but i guess there is already code doing that in the device code somewhere teythoon: vm_object_copy_slowly seems to create a new object that's not exactly what we want either why not ? again, let's assume a device_write scenario ah you want to populate the ramdisk, which is merely one object not a new object yes teythoon: i suggest using vm_page_alloc and vm_page_copy and vm_page_lookup teythoon: perhaps vm_fault_page too although you might want wired pages initially teythoon: but i guess you see what i mean when i say it needs to be reworked i do braunr: aww, screw that, using a tmpfs is much nicer anyway the ramdisk strikes again ... teythoon: :) teythoon: an extremely simple solution would be to enlarge the kernel map this would reduce the userspace max size to ~1.7G but allow ~64M ramdisks nah or we could reduce the kmem_map i think i'll do that anyway the slab allocator rarely uses more than 50-60M and the 64M remaining area in kernel_map can quickly get fragmented braunr: using a tmpfs as the root translator won't be straight forward either ... damn the early boostrapping stuff ... yes .. that's one of the downsides of the vfs-as-namespace approach i'm not sure it could be simplified hm it could even use a temporary name server to avoid dependencies indeed there's even still the slot for that somewhere braunr: hm... I have a vague recollection that the fixed-sized kmem-map was supposed to be gone with the introduction of the new allocator?... antrik: the kalloc_map and kmem_map were merged we could directly use kernel_map but we may still want to isolate it to avoid fragmentation See also the discussion on [[gnumach_memory_management]], *IRC, freenode, \#hurd, 2013-01-06*, *IRC, freenode, #hurd, 2014-02-11* (`KENTRY_DATA_SIZE`). ### IRC, freenode, #hurd, 2012-07-17 OK, here is a stupid question I have always had. If you move PCI and disk drivers in to userspace, how do do initial bootstrap to get the system booting? that's hard basically you make the boot loader load all the components you need in ram then you make it give each component something (ports) so they can communicate ### IRC, freenode, #hurd, 2012-08-12 braunr: so, about booting with userspace disk drivers after rereading the chapter in my thesis, I see that there aren't really all than many interesting options... I pondered some variants involving a temporary boot filesystem with handoff to the real root FS; but ultimately concluded with another option that is slightly less elegant but probably gets a much better usefulness/complexity ratio: just start the root filesystem as the first process as we used to; only hack it so that initially it doesn't try to access the disk, but instead gets the files from GRUB once the disk driver is operational, we flip a switch, and the root filesystem starts reading stuff from disk normally transparently for all other processes How does grub access the disk without drivers? bddebian: GRUB obviously has its own drivers... that's how it loads the kernel and modules bddebian: basically, it would have to load additional modules for all the components necessary to get the Hurd disk driver going Right, why wouldn't that be possible? (I have some more crazy ideas too -- but these are mostly orthogonal :-) ) ? I'm describing this because I'm pretty sure it *is* possible :-) That grub loads the kernel and whatever server/module gets access to the disk not sure what you mean Well as usual I probably don't know the proper terminology but why could grub load gnumach and the hurd "disk server" that contains the userspace drivers? disk server? Oh FFS whatever contains the disk drivers :) diskdde, whatever :) actually, I never liked the idea of having a big driver blob very much... ideally each driver should have it's own file but that's admittedly beside the point :-) its so to restate: in addition to gnumach, ext2fs.static, and ld.so, in the new scenario GRUB will also load exec, the disk driver, any libraries these two depend upon, and any additional infrastructure involved in getting the disk driver running (for automatic probing or whatever) probably some other Hurd core servers too, so we can have a more complete POSIX environment for the disk driver to run in There ya go :) the interesting part is modifying ext2fs so it will access only the GRUB-provided files, until it is told that it's OK now to access the real disk (and the mechanism how ext2 actually gets at the GRUB-provided files) Or write some new really small ext2fs? :) ? I'm just talking out my butt. Something temporary that gets disposed of when the real disk is available :) well, I mentioned above that I considered some handoff schemes... but they would probably be more complex to implement than doing the switchover internally in ext2 Ah boot up in a ramdisk? :) (and the temporary FS would *not* be an ext2 obviously, but rather some special ramdisk-like filesystem operating from GRUB-loaded files...) again, that would require a complicated handoff-scheme Bah, what do I know? :) (well, you could of course go with a trivial chroot()... but that would be ugly and inefficient, as the initial processes would still run from the ramdisk) Aren't most things running in memory initially anyway? At what point must it have access to the real disk? antrik: but doesn't that require that disk drivers be statically linked ? and having all disk drivers in separate tasks (which is what we prefer to blobs as you put it) seems to pretty much forbid using static linking hm actually, i don't see how any solution could work without static linking, as it would create a recursion and the only one required is the one used by the root file system others can be run from the dynamically linked version antrik: i agree, it's a good approach, requiring only a slightly more complicated boot script/sequence bddebian: at some point we have to access the real disk so we don't have to work exclusively with stuff loaded by grub... but there is no specific point where it *has* to happen. generally speaking, the sooner the better braunr: why wouldn't that work with a dynamically linked disk driver? we only need to make sure all required libraries are loaded by grub too antrik: i have a problem with that approach :p antrik: it would probably require a reboot when those libraries are upgraded, wouldn't it ? I'd actually wish we could run with a dynamically linked ext2fs as well... but that would require a separated boot filesystem and some kind of handoff approach, which would be much more complicated I fear... and if a driver is restarted, would it use those libraries too ? and if so, how to find them ? but how can you run a dynamically linked root file system ? unless the libraries it uses are provided by something else, as you said braunr: well, if you upgrade the libraries, *and* want the disk driver to use the upgraded libraries, you are obviously in a tricky situation ;-) yes perhaps you could tell ext2 to preload the new libraries before restarting the disk driver... but that's a minor quibble anyways IMHO but that case isn't that important actually, since upgrading these libraries usually means we're upgrading the system, which can imply a reoobt