From 9933cec0a18ae2a3d752f269d1bb12c19f51199d Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Sun, 21 Jul 2013 15:35:02 -0400 Subject: IRC. --- microkernel/mach/deficiencies.mdwn | 128 ++++++++++++++++++++ microkernel/mach/gnumach/debugging.mdwn | 5 +- .../mach/gnumach/interface/syscall/mach_print.mdwn | 29 +++++ microkernel/mach/history.mdwn | 134 +++++++++++++++++++++ microkernel/mach/mig.mdwn | 6 +- microkernel/mach/mig/structured_data.mdwn | 119 ++++++++++++++++++ 6 files changed, 417 insertions(+), 4 deletions(-) create mode 100644 microkernel/mach/mig/structured_data.mdwn (limited to 'microkernel') diff --git a/microkernel/mach/deficiencies.mdwn b/microkernel/mach/deficiencies.mdwn index d1cdeb54..03e4a8b0 100644 --- a/microkernel/mach/deficiencies.mdwn +++ b/microkernel/mach/deficiencies.mdwn @@ -2190,3 +2190,131 @@ In context of [[open_issues/multithreading]] and later [[open_issues/select]]. i really need to dig into this code at some point :) well you may but you may not see that property from the code itself + + +## IRC, freenode, #hurd, 2013-06-28 + + so tell me about x15 if you are in the mood to talk about that + what do you want to know ? + well, the high level stuff first + like what's the big picture + the big picture is that x15 is intended to be a "better mach for + the hurd + " + mach is too general purpose + its ipc mechanism too powerful + too complicated, error prone, and slow + so i intend to build something a lot simpler and faster :p + so your big picture includes actually porting hurd? i thought i + read somewhere that you have a rewrite in mind + it's a clone, yes + x15 will feature mostly sync ipc, and no high level types inside + messages + the ipc system call will look like what qnx does + send-recv from the client, recv/reply/reply-recv from the server + but doesn't sync mean that your context switch will have to be + quite fast? + how does that differ from the async approach ? + (keep in mind that almost all hurd RPCs are synchronous) + yes, I know, and it also affects async mode, but a slow switch + is worse for the sync case, isn't it? + ok so your ipc will be more agnostic wrt to what it transports? + unlike mig I presume? + no it's the same + yes + input will be an array, each entry denoting either memory or port + rights + (or directly one entry for fast ipcs) + memory as in pointers? + (well fast ipc when there is only one entry to avoid hitting a + table) + pointer/size yes + hm, surely you want a way to avoid copying that, right? + the only operation will be copy (i.e. unlike mach which allows + sharing) + why ? + copy doesn't exclude zero copy + (zero copy being adjusting page tables with copy on write + techniques) + right + but isn't that too coarse, like in cow a whole page? + depends on the message size + or options provided by the caller, i don't know yet + oh, you are going to pack the memory anyway? + depends on the caller + i'm not yet sure about these details + ideally, i'd like to avoid serialization altogether + wouldn't that be like cheating b/c it's the first copy? + directly pass pointers/sizes from the sender address space, and + either really copy or use zero copy + right, but then you're back at the page size issue + yes + it's not a real issue + the kernel must support both ways + the minor issue is determining which way to choose + it's not a critical issue + my current plan is to always copy, unless the caller has + explicitely set a flag and is passing properly aligned buffers + u sure? I mean the caller is free to arange the stuff he intends + to send anyway he likes, how are you going to cow that then? + ok + right + properly aligned buffers :) + otherwise the kernel rejects the request + that's reasonable, yes + in addition to being synchronous, ipc will also take a special + path in the scheduler to directly use the client scheduling context + avoiding the sleep/wakeup overhead, and providing priority + inheritence by side effect + uh, but wouldn't dropping serialization create security and + reliability issues? if the receiver isn't doing a proper job sanitizing + its stuff + why would the client not sanitize ? + err + server + it has to anyway + sure, but a proper parser written once might be more robust, + even if it adds overhead + the serialization i mean + it's just a layer + even with high level types, you still need to sanitize + the real downside is loosing cross architecture portability + making the potential implementation of a single system image a lot + more restricted or difficult + but i don't care about that much + mach was built with this in mind though + it's a nice idea, but i don't believe anyone does ssi anymore + i don't know + and certainly not across architectures + there are few projects + anyway it's irrelevant currently + and my interface just restricts it, it doesn't prevent it + so i consider it an acceptable compromise + so, does it run? what does it do? + it certainly is, yes + for now, it manages memory (physical, virtual, kernel, and soon, + anonymous) + support multiple processors with the required posix scheduling + policies + (it uses a cute proportionally fair time sharing algorithm) + there are locks (spin locks, mutexes, condition variables) and + lockless stuff (à la rcu) + both x86 and x86_64 are supported + (even pae) + work queues + sounds impressive :) + :) + i also added basic debugging + stack trace (including getting the symbol table) handling + so yes, it's much much better than what i previously did + and on the right track + it already scales a lot better than mach for what it does + there are generic data structures (linked list, red-black tree, + radix tree) + the radix tree supports lockless lookups, so looking up both the + page cache and the ipc spaces is lockless) + that's nice :) + there are a few things using global locks, but there are TODOs + about them + even with that, it should be scalable enough for a start + and improving those parts shouldn't be too difficult diff --git a/microkernel/mach/gnumach/debugging.mdwn b/microkernel/mach/gnumach/debugging.mdwn index 71e92459..7e7cfb4e 100644 --- a/microkernel/mach/gnumach/debugging.mdwn +++ b/microkernel/mach/gnumach/debugging.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2007, 2008, 2009, 2011, 2012 Free Software +[[!meta copyright="Copyright © 2007, 2008, 2009, 2011, 2012, 2013 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -75,6 +75,9 @@ When you're [[running_a_system_in_QEMU|hurd/running/qemu]] you can directly kernel](http://www.nongnu.org/qemu/qemu-doc.html#SEC48). +## [[open_issues/debugging_gnumach_startup_qemu_gdb]] + + # Code Inside the Kernel Alternatively you can use an approach like this one: add the following code diff --git a/microkernel/mach/gnumach/interface/syscall/mach_print.mdwn b/microkernel/mach/gnumach/interface/syscall/mach_print.mdwn index ca52dca5..a169e92e 100644 --- a/microkernel/mach/gnumach/interface/syscall/mach_print.mdwn +++ b/microkernel/mach/gnumach/interface/syscall/mach_print.mdwn @@ -59,3 +59,32 @@ License|/fdl]]."]]"""]] it [[Makefile]], [[mach_print.S]], [[main.c]]. + + +## IRC, freenode, #hurd, 2013-07-01 + + braunr: btw, we are missing the symbol in mach/Versions + youpi: what symbol ? + so the libc-provided RPC stub is not available + mach_printf + -f + it's a system calll + not exported + s/RPC/system call/ + that's expected + libc does provide stubs for system calls too + yes but not for this one + I don't see why we wouldn't want to include it + ?! it does + it's temporary + oh + there must be automatic parsing during build + sure + nice + + youpi: if we're going to make this system call exported by glibc, + i should change its interface first + it was meant as a very quick-and-dirty hack and directly accesses + the caller's address space without going through a special copy-from-user + function + not very portable diff --git a/microkernel/mach/history.mdwn b/microkernel/mach/history.mdwn index 776bb1d7..c22ea739 100644 --- a/microkernel/mach/history.mdwn +++ b/microkernel/mach/history.mdwn @@ -78,3 +78,137 @@ IRC, freenode, #hurd, 2012-08-29: https://developer.apple.com/library/mac/#documentation/Darwin/Conceptual/KernelProgramming/About/About.html can't be anymore http://developer.apple.com/techpubs/macosx/Darwin/General/KernelProgramming/About/index.html + +IRC, freenode, #hurd, 2013-07-03: + + *** natsukao (~natsukao@dynamic-adsl-94-37-184-109.clienti.tiscali.it) has + joined channel #hurd + hi + on 2012-08-29: i wrote a part of messages that then were posted + on http://www.gnu.org/software/hurd/microkernel/mach/history.html + i am sorry to inform you that apple computer cuèertino.inc, has + moved the URL: https://ssl.apple.com/science/profiles/cornell + and i have not found nothing on the source code of that page, + i used lftp without any success + and then wget, nothing to do + i have not found a copy cache of + https://ssl.apple.com/science/profiles/cornell + next time we save the documents and we provide to do our + archive/s + so that will be always available the infos + *** natsukao (~natsukao@dynamic-adsl-94-37-184-109.clienti.tiscali.it) is + now known as pavlx + happy hacking !!!! + "paolo del bene" + p.s: i'll turn back as soon as possible + + i found the page of Darwin History, removed from apple compter + cupertino.inc + "Cached_ http___developer.apple.com_darwin_history.html" + the page http://developer.apple.com/darwin/history.html was moved + and now is available on: + + http://www.google.it/url?q=http://www.macmark.de/files/darwin_evolution.pdf%3FPHPSESSID%3D8b8683a81445f69d510734baa13aabfc&sa=U&ei=wMzTUb-NBIeFOOL4gNgE&ved=0CCQQFjAD&usg=AFQjCNFlLwC24nB5t14VUmedK4EmeE7Ohw + or simply: http://www.macmark.de/files/darwin_evolution.pdf + slides on: "Travel - Computer Science and Software Engineering" + www.csse.uwa.edu.au/~peterj/personal/PDFs/WWDC2004-6UP.pdf + about apple computer cupertino.inc, but there are many interesting + news + pavlx: uh, lot's of marketing noise from apple >,< + i found better material just now: + http://www.pcs.cnu.edu/~mzhang/CPSC450_550/2003CaseStudy/Mach_Presentation_DavidRamsey.ppt + teythoon, sorry, i turn back to sleep, see you later, paolo + W3C_Freedom@riseup.net + i'll charge of that page only things dedicated to GNU/HURD, but + slides are not mine, i found on internet + pavlx: sure, I didn't ment to offend you in any way + +IRC, freenode, #hurd, 2013-07-04: + + there are few problems: + http://www.gnu.org/software/hurd/microkernel/mach/history.html + on the page GrantBow wrote: Apple's Macintosh OSX (OS 10.x) is + based on Darwin. "Darwin uses a monolithic kernel based on ?FreeBSD 4.4 + and the OSF/mk Mach 3." Darwin also has a Kernel Programming Book. + the link to Darwin was moved, is not anymore + http://www.apple.com/macosx/technologies/darwin.html + then it's not FreeBSD 4.4 but BSD + and the link to Kernel Programming was moved is not + http://developer.apple.com/techpubs/macosx/Darwin/General/KernelProgramming/About/index.html + but + https://developer.apple.com/library/mac/#documentation/Darwin/Conceptual/KernelProgramming/About/About.html + apple has moved the URL: + https://ssl.apple.com/science/profiles/cornell + apple has moved the URL: + http://www.apple.com/macosx/technologies/darwin.html + so on the website you can left few things about my old post: + from IRC, freenode, #hurd, 2012-08-29: needs to remove + http://dpaste.com/1286266/ + the new one will be: http://pastebin.com/yMm8tcTN + IRC, freenode, #hurd, 2013-07-04: + + was moved the page from apple.com about darwin kernel programming + as described on the https://www.gnu.org/software/hurd/microkernel/mach/history.html + + the link to Kernel Programming: + https://developer.apple.com/library/mac/#documentation/Darwin/Conceptual/KernelProgramming/About/About.html + (anyway i searching with any key the things moved from apple) + about Darwin type http://apple.com/darwin + on the right side, towards the end of the website it says: Darwin + Technologies + click on it, or copy the URL in an other tab of your own browser, + and read: + https://developer.apple.com/library/mac/#documentation/MacOSX/Conceptual/OSX_Technology_Overview/SystemTechnology/SystemTechnology.html + and something is related to Darwin + and again : http://pastebin.com/DHkJDxy8 + # Mac OS X Server + + ... This kernel, known as Darwin, provides a stable, high-performance platform + for developing groundbreaking applications and system technologies. ... + http://www.apple.com/server/docs/MacOSX_Server_TO_300195.pdf + + # Mac OS X Server Command-Line Administration + + Page 1. Mac OS X Server Command-Line Administration For Version 10.3 + http://www.apple.com/server/docs/Command_Line.pdf + + # Press Info - Apple “Open Sources” Rendezvous + + ... Rendezvous is part of a broader open source release today from Apple at + http://developer.apple.com/darwin which includes the Darwin 6.0.1 ... + http://www.apple.com/pr/library/2002/09/25Apple-Open-Sources-Rendezvous.html + + # Press Info - Apple Releases Darwin 1.0 Open Source + + ... Apple Releases Darwin 1.0 Open Source. New ... modules. Darwin 1.0 gives + developers access to essential Mac OS X source code. ... + http://www.apple.com/pr/library/2000/04/05Apple-Releases-Darwin-1-0-Open-Source.html + + # Press Info - Apple's Mac OS X to Ship on March 24 + + ... Mac OS X is built upon an incredibly stable, open source, UNIX based + foundation called Darwin and features true memory protection, preemptive ... + http://www.apple.com/pr/library/2001/01/09Apples-Mac-OS-X-to-Ship-on-March-24.html + + # Press Info - Mac OS X “Gold Master” Released To ... + + ... Mac OS X is built upon an incredibly stable, open source, UNIX + basedfoundation called Darwin and features true memory protection ... + http://www.apple.com/pr/library/2001/03/07Mac-OS-X-Gold-Master-Released-To-Manufacturing.html + + * Press Info - Apple Announces Mac OS X “Jaguar” ... + + ... As an active member of the Open Source community, Apple has distributed + Open Directory technology through the Darwin Open Source Project. ... + http://www.apple.com/pr/library/2002/07/17Apple-Announces-Mac-OS-X-Jaguar-Server-Worlds-Easiest-to-Manage-UNIX-Based-Server-Software.html + and: + http://lists.apple.com/archives/darwinos-users/2005/Apr/msg00021.html + pavlx: it's hard to follow the changes you are talking + about. Perhaps you could simply edit these wiki pages? + anyway i am saying to you that i found a mailing list where are + availables the sources codes of darwin ppc-801 and x86 + and as last thing mac os x 10.4 + pavlx: what's all this about ? + i am sorry, i did changes on the wiki + but after the preview and after to have saved, it show again the + old chat of 2012 diff --git a/microkernel/mach/mig.mdwn b/microkernel/mach/mig.mdwn index 331b3bf4..d6340574 100644 --- a/microkernel/mach/mig.mdwn +++ b/microkernel/mach/mig.mdwn @@ -1,5 +1,5 @@ -[[!meta copyright="Copyright © 2001, 2002, 2003, 2006, 2007, 2008, 2010 Free -Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2001, 2002, 2003, 2006, 2007, 2008, 2010, 2013 +Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -22,9 +22,9 @@ wait for a result on a newly created [[reply port|port]], decode return arguments from the reply message (*demarshalling*, or *unmarshalling*) and pass them to the client program. Similar actions are provided in the skeletons that are linked to server programs. - MIG allows very precise semantics to be specified about what the arguments are and how to be passed. +It has its problems with [[structured_data]], however. * [[Documentation]] diff --git a/microkernel/mach/mig/structured_data.mdwn b/microkernel/mach/mig/structured_data.mdwn new file mode 100644 index 00000000..1c8abe08 --- /dev/null +++ b/microkernel/mach/mig/structured_data.mdwn @@ -0,0 +1,119 @@ +[[!meta copyright="Copyright © 2013 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_mig]] + +# IRC, freenode, #hurd, 2013-06-25 + + is there a nice way to get structured data through mig that I + haven't found yet? + say an array of string triples + no + :/ + but you shouldn't need that + my use case is getting info about fs translators from init to + procfs + +[[community/gsoc/project_ideas/mtab]]. + + should I go for an iterator like interface instead? + depends + how many do you need ? + you could go for a variable sized array too + have a look at what already exists + records, maybe 10-15, depends on many fs translators are running + a variable sized array is ok if the size isn't too big (and when i + say too big, i mean hundreds of MiB) + an iterator is ok too if there aren't too many items + you may want to combine both (i think that's what proc does) + be aware that the maximum size of a message is limited to 512 MiB + yeah I saw the array[] of stuff stuff, but array[] of string_t + does not work, I guess b/c string_t is also an array + how would I send an array of variable length strings? + i'm not sure you can + or maybe out of line + somehow I expected mig to serialize arbitrary data structures, + maybe it's to old for that? + yeah, I read about uot of line, but that seems overkill + it is old yes + and not very user friendly in the end + let me check + we could stuff json into mig... + see proc_getallpids for example + we could get rid of low level serialization altogether :p + hah, exactly what I was looking at + (which is what i'll do in x15) + type pidarray_t = array[] of pid_t; + but that is trivial b/c its array[] of pid_t + and always have the server writing guide near you + yes + well, make one big string and an array of lengths :p + thought about that and said to myself, there must be a better + way that I haven't found yet + or one big string filled with real null-terminated c strings that + you keep parsing until you ate all input bytes + i'm almost certain there isn't + type string_t = c_string[1024]; /* XXX */ + yes + even that isn't really variable sized + you think anyone would object to me putting a json encoder in + /hurd/init? it is probably better than me at serializing stuff... + try with mig anyway + the less dependencies we have for core stuff, the simpler it is + but i agree, mig is painful + would it be too hacky if I abused the argz functions? they do + exactly what I'd need + + +## IRC, freenode, #hurd, 2013-06-26 + + there is https://code.google.com/p/protobuf-c/ and it has a rpc + mechanism and I believe one could plug arbitrary transports easily + please don't think about it + we really don't want to add another layer of serialization + it's better to completely redesign mach ipc anyway + and there is a project for that :p + ive seen x15 + just food for thought + i've studied google protocol buffers + and fyi, no, it wouldn't be easy to plug arbitrary transports on + top of mach + there is a lot of knowledge about mach ports in mig + +[[community/gsoc/project_ideas/mtab]]. + + but again I face the challenge of serializing a arbitrary sized + list of arbitrary sized strings + yes + list of ports is easier ;) but I think its worthwile + so what about abusing argz* for this? you think it's too bad a + hack? + no since it's in glibc + awesome :) + but i don't remember the details well and i'm not sure the way you + use it is safe + yeah, I might have got the details wrong, I hadn't had the + chance to test it ;) + + about this dynamic size problem + a "simple" varying size array should do + you can easily put all your strings in there + seperated by 0? + yes + that's exactly what the argz stuff does + you'll get the size of the array anyway, and consume it until + there is no byte left + good + but be careful with this too + since translators can be run by users, they somtimes can't be + trusted + and even a translator running as root may behave badly + so careful with parsing + noted -- cgit v1.2.3