summaryrefslogtreecommitdiff
path: root/community
diff options
context:
space:
mode:
Diffstat (limited to 'community')
-rw-r--r--community/gsoc/project_ideas/language_bindings.mdwn42
-rw-r--r--community/gsoc/project_ideas/object_lookups.mdwn409
2 files changed, 448 insertions, 3 deletions
diff --git a/community/gsoc/project_ideas/language_bindings.mdwn b/community/gsoc/project_ideas/language_bindings.mdwn
index 61a3fa01..344b741c 100644
--- a/community/gsoc/project_ideas/language_bindings.mdwn
+++ b/community/gsoc/project_ideas/language_bindings.mdwn
@@ -1,5 +1,5 @@
-[[!meta copyright="Copyright © 2008, 2009, 2010, 2011, 2012, 2013 Free Software
-Foundation, Inc."]]
+[[!meta copyright="Copyright © 2008, 2009, 2010, 2011, 2012, 2013, 2014 Free
+Software Foundation, Inc."]]
[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
id="license" text="Permission is granted to copy, distribute and/or modify this
@@ -61,3 +61,41 @@ bindings](http://www.nongnu.org/hurdextras/#pith), which might serve as a
reference if you want to work on Perl.
Possible mentors: Anatoly A. Kazantsev (anatoly) for Python
+
+
+# Discussion
+
+## [[user/jkoenig/Java]]
+
+## IRC, freenode, #hurd, 2013-12-19
+
+ <antrik_> teythoon_: I don't think wrapping libtrivfs etc. for guile
+ bindings is really desirable... for the lisp bindings, we agreed that
+ it's better to hook in at a lower level, and build more lispish
+ abstractions
+ <antrik> trivfs is a C framework; it probably doesn't map very well to
+ other languages -- especially non-imperative ones...
+ <antrik> (it is arguable whether trivfs is really a good abstraction even
+ for C... but that's another discussion :-) )
+ <antrik> ArneBab: same for Python bindings. when I suggested ignoring
+ libtrivfs etc., working around the pthread problem was just a side effect
+ -- the real goal has always been nicer abstraction
+ <anatoly> antrik: agree with you
+ <anatoly> antrik: about nicer abstractions
+ <teythoon_> antrik: I agree too, but wrapping libtrivfs is much easier
+ <teythoon_> otherwise, one needs to reimplement lots of stuff to get some
+ basic functionality
+ <teythoon_> like a mig that emits your language
+ <braunr> i agree with antrik too
+ <braunr> yes, the best would be mig handling multiple languages
+
+[[!taglink open_issue_mig]].
+
+ <antrik> teythoon_: not exactly. for dynamic languages, code generation is
+ silly. just handle the marshalling on the fly. that's what the Lisp
+ bindings are doing (AFAIK)
+ <teythoon> antrik: ok, but you'd still need to parse the rpc definitions,
+ no?
+ <antrik> teythoon: yeah, you still need to parse the .defs -- unless we add
+ reflection to RPC interfaces...
+ <antrik> err, I mean introspection
diff --git a/community/gsoc/project_ideas/object_lookups.mdwn b/community/gsoc/project_ideas/object_lookups.mdwn
index ca586dea..d3e17dc9 100644
--- a/community/gsoc/project_ideas/object_lookups.mdwn
+++ b/community/gsoc/project_ideas/object_lookups.mdwn
@@ -1,4 +1,4 @@
-[[!meta copyright="Copyright © 2013 Free Software Foundation, Inc."]]
+[[!meta copyright="Copyright © 2013, 2014 Free Software Foundation, Inc."]]
[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
id="license" text="Permission is granted to copy, distribute and/or modify this
@@ -132,3 +132,410 @@ In context of [[!message-id "20130918081345.GA13789@dalaran.sceen.net"]].
<braunr> but murmur is better
<braunr> we prefer distribution over hashing performances
<braunr> https://131002.net/siphash/
+
+
+## IRC, freenode, #hurd, 2013-11-21
+
+ <teythoon> btw, about protected payloads in mach
+ <teythoon> I'm thinking about adding a flag to indicate that mach_msg
+ should return the protected payload pointer instead of the local port
+ field in the message header
+ <braunr> when would you set it ?
+ <braunr> i mean, how is it set ?
+ <teythoon> we don't really need the port name, right? and when we do, we
+ look it up in the referenced data structure
+ <teythoon> using a new rpc perhaps
+ <braunr> ok
+ <teythoon> what do you think?
+ <braunr> a new rpc on ports themselves, like mach_port_mod_refs i assume ?
+ <braunr> i think it's a good solution
+ <teythoon> the field is a natural_t, as far as i can see, pointers should
+ fit into it, right?
+ <teythoon> yes
+ <braunr> the big problem is backward compatibility
+ <teythoon> why?
+ <braunr> and your solution solves that
+ <teythoon> yes
+ <braunr> hum
+ <braunr> natural_t was originally intended to be as large as the machine
+ word
+ <braunr> but it may no longer stay true
+ <braunr> i think youpi decided to keep it an int and not a long in his
+ x86_64 branch
+ <braunr> mach uses a trick for in-kernel port rights
+ <braunr> where the right is the port address
+ <teythoon> yes, I've seen that
+ <braunr> but i remember hearing about problems with this strategy in
+ 64-bits
+ <braunr> or maybe compat problems in mig interfaces
+ <braunr> i don't remember exactly
+ <braunr> so youpi looked at how macosx mach deals with the problem
+ <teythoon> well, but so far there is no 64 bit mach, so we do not need to
+ worry about compatibility there, no ?
+ <braunr> and if i'm right, they forced the ports on 32-bits
+ <braunr> no you're right
+ <braunr> we can simply force the field to 64-bits, whatever it contains
+ <teythoon> or change the message format from the beginning to include both
+ the name and the payload
+ <teythoon> then again, why bother
+ <braunr> ?
+ <braunr> have a 64-bits specific message format ?
+ <teythoon> well, it's worth discussing, no?
+ <braunr> maybe
+ <braunr> i personally don't like the idea
+ <teythoon> we could fix stuff
+ <braunr> forcing the field to 64-bits should be enough
+ <teythoon> right
+ <teythoon> do you think the idea is worth prototyping ?
+ <braunr> teythoon: yes
+ <teythoon> braunr: cool :)
+ <braunr> teythoon: definitely :p
+ <braunr> actually, doing that can remove a large part (if not all)
+ contention from libports
+ <teythoon> indeed
+ <braunr> i still think we should work on libihash first
+
+[[hurd/libihash]].
+
+ <braunr> converting libihash to murmur2/3 impacts more data structures
+ overall
+ <braunr> it's also much easier
+ <teythoon> what exactly do you mean by that
+ <teythoon> ?
+ <braunr> libports uses libihash
+ <teythoon> yes
+ <braunr> but it's not the only user
+ <braunr> libihash is known to have high collision rates
+ <braunr> that should be fixed
+ <teythoon> right, but what do you mean by using murmur2/3
+ <braunr> that's a hashing algorithm name
+ <teythoon> using the integer finalizer used by murmur?
+ <braunr> hm
+ <braunr> i didn't dig the details
+ <braunr> and simply assumed it could be used for integer hashing as well
+ <teythoon> the way i see it, murmur can hash arbitrary ammounts
+ <braunr> if there are better integer hashing algorithms, let's just use
+ that
+ <teythoon> but that is not what we need
+ <braunr> yes
+ <teythoon> we have a fixed size integer
+ <braunr> but from what i remember, it's also very efficient for integer
+ hashing
+
+
+## IRC, freenode, #hurd, 2013-11-22
+
+ <teythoon> braunr: /test-pp: msgh_protected_payload is 0x12345678
+ <teythoon> :)
+ <braunr> :)
+ <teythoon> but currently I do another ipc_port_translate which is clearly
+ not desireable
+ <teythoon> the msg handling in the kernel is... involved...
+ <teythoon> here is the thing... there are two (kernel) threads involved,
+ the sender and the receiver
+ <teythoon> for the sender, kmsg->ikm_header.msgh_remote_port is a pointer
+ (thanks to ipc_port_translate) to the destination's ipc_port_t
+ <teythoon> that's where the protected_payload is stored
+ <teythoon> but at the receiving thread, the pointer is gone, replaced by a
+ port name
+ <teythoon> so currently I'm doing the lookup there again
+ <braunr> hum
+ <braunr> are you sure kmsg is the general structure for all messages ?
+ <braunr> or is it only for kernel messages ?
+ <braunr> i don't remember exactly
+ <teythoon> no, for all messages
+ <braunr> ok
+ <teythoon> I just need to get this pointer across cleanly
+ <braunr> i thought you wanted to replace that port name in the receiving
+ thread with the payload
+ <teythoon> I do
+ <braunr> i don't see the problem then
+ <teythoon> well, only the sending thread has the pointer, the receiving
+ thread only has the name
+ <braunr> i don't see what makes it hard to change it
+ <braunr> since that's what you want to do
+ <braunr> the sending thread doesn't have the pointer
+ <teythoon> yes it has
+ <braunr> well
+ <braunr> only for in kernel objects
+ <braunr> and it's an optimization
+ <braunr> and you shouldn't have to care much about it
+ <braunr> your work only involves changing how messages are received
+ normally
+ <teythoon> let me push it somewhere, so I can show you the patches
+ <braunr> ok
+ <teythoon> braunr:
+ http://darnassus.sceen.net/gitweb/teythoon/gnumach.git/shortlog/refs/heads/feature-protected-payload-1
+ <braunr> teythoon: where should i look at ?
+ <teythoon> the last commit
+ <braunr> hm
+ <braunr> see what calls mach_msg_receive
+ <braunr> the payload flag must be handled before, when the message is
+ actually transferred
+ <braunr> ipc_kmsg_copyin perhaps
+ <teythoon> well
+ <teythoon> but this is the tricky part
+ <braunr> i'm not sure which of the sender or receiver code actually
+ performs these translations
+ <braunr> yes
+ <teythoon> b/c at this point it is not known whether the receiver has
+ specified the MACH_RCV_PROTECTED_PAYLOAD flag
+ <teythoon> or my understanding of the whole process is still somewhat off,
+ which might very well be...
+ <braunr> it's not something the receiver should set
+ <braunr> i.e. the flag shouldn't be set at mach_msg time
+ <braunr> because it's asynchronous
+ <braunr> it's a flag that should be set near port creation time
+ <teythoon> oh
+ <teythoon> right, I can see how that could work
+ <braunr> mach_reply_port(); mach_port_set_payload(); mach_msg();
+ <teythoon> braunr:
+ http://darnassus.sceen.net/gitweb/teythoon/gnumach.git/log/refs/heads/feature-protected-payload-2
+ <teythoon> I think I found the right spot
+ <braunr> teythoon: looks better indeed :)
+ <teythoon> braunr: yes, thanks for the hint :)
+ <braunr> teythoon: keep in mind gnumach supports moving receive rights
+ between tasks
+ <braunr> i don't think it's much of a burden but don't forget :)
+ <teythoon> right, if that happens, the protected payload field should
+ probably be just reset to 0
+ <teythoon> that preserves the old default behavior
+ <braunr> teythoon: you shouldn't name the payload "address" though
+ <braunr> but really "payload" or "label"
+ <braunr> vm_offset_t isn't the appropriate type either
+ <braunr> i suggest unsigned long payload
+ <teythoon> braunr: noted
+ <braunr> what i mean is
+ <braunr> the payload isn't the userspace structure you want to use
+ <braunr> it's the value stored in that unsigned long
+ <braunr> whether it's used as a pointer or an array index or whatever
+ should be at the application discretion
+ <teythoon> yes, I got that
+ <braunr> concerning vm_offset_t, it's misused a lot, mostly for historical
+ reasons
+ <braunr> vm_offset_t is actually the ancestor of off_t
+ <braunr> i.e. an offset inside a *memory object*
+ <braunr> the size of which may differ from the size of a pointer
+ <teythoon> ok
+ <braunr> historically, physical and virtual addresses, in addition to
+ memory object sizes, were the same, hence the confusion
+ <braunr> today we might have 32-bits virtual addresses, 36-bits physical
+ addresses, and 44- to 64-bits file offsets
+ <braunr> (e.g. PAE with large file support)
+
+
+## IRC, freenode, #hurd, 2013-11-25
+
+ <teythoon> braunr: the object lookup problem is worse than i assumed
+ <teythoon> the lookup is actually done twice...
+ <braunr> teythoon: isn't that usually the case :) ?
+ <braunr> inside gnumach ?
+ <teythoon> no
+ <teythoon> once in libports, once in the intrans function
+ <braunr> which intrans function ?
+ <braunr> can you point at an example ?
+ <teythoon> right
+ <teythoon> routine file_get_fs_options ( file: file_t;
+ <teythoon> file_t is special
+ <teythoon> mig magic
+ <teythoon> type file_t = mach_port_copy_send_t
+ <teythoon> #ifdef FILE_INTRAN
+ <teythoon> intran: FILE_INTRAN
+ <braunr> trivfs_begin_using_protid ?
+ <teythoon> for example, yes
+ <braunr> ugh
+ <teythoon> however, I believe that can be fixed cleanly
+ <teythoon> I revised my gnumach changes
+ <teythoon> it works surprisingly well
+ <braunr> gnumach is largely clean code
+ <teythoon> i patched libports to use the new falicilty
+ <teythoon> all the fs translators i tested work fine
+ <braunr> nice
+ <teythoon> tmpfs, ext2fs, nfs, hello*
+ <teythoon> so does exec
+ <braunr> very nice
+ <teythoon> howevcer, the bootstrap fails
+ <braunr> a lot more straightforward than i expected
+ <teythoon> i believe proc crashes
+ <teythoon> yes
+ <braunr> you can use mach_print to manually trace the bootstrap process
+
+[[microkernel/mach/gnumach/interface/syscall/mach_print]].
+
+ <teythoon> i did that
+ <braunr> ok
+ <teythoon> it's nice
+ <braunr> bare knives are :)
+
+ <teythoon> uh oh, this lookup fix requires some mig changes
+ <braunr> teythoon: have you built some packages on it ?
+ <teythoon> braunr: some clang test builds
+ <braunr> nice
+ <braunr> where is mig getting in the way ?
+ <teythoon> yes, and i compiled lots of stuff
+ <braunr> any debian package ?
+ <teythoon> let me just push my changes somewhere...
+ <teythoon> no, no deb
+ <braunr> ok
+ <teythoon> braunr:
+ http://darnassus.sceen.net/gitweb/teythoon/gnumach.git/log/refs/heads/feature-protected-payload-3
+ <teythoon>
+ http://darnassus.sceen.net/gitweb/teythoon/hurd.git/log/refs/heads/feature-protected-payload-1
+ <teythoon> braunr: in particular,
+ http://darnassus.sceen.net/gitweb/teythoon/hurd.git/blob/refs/heads/feature-protected-payload-1:/libports/manage-multithread.c#l161
+
+
+## IRC, freenode, #hurd, 2013-11-27
+
+ <teythoon> btw, my protected payload work is progressing nicely
+ <teythoon> the system actually boots now :)
+ <braunr> that's great
+ <braunr> looking forward to seeing it in action
+ <teythoon> I'd love to quickly discuss my mig changes if you've got a
+ minute
+ <braunr> sure
+ <teythoon> ok
+ <teythoon> first, please look at this
+ http://darnassus.sceen.net/gitweb/teythoon/hurd.git/blob/refs/heads/feature-protected-payload-1:/libports/manage-multithread.c#l161
+ <teythoon> in line 165, the msgh_local_port is restored
+ <teythoon> b/c later some intrans function might use this for the object
+ (re-)lookup
+ <braunr> yes ok
+ <teythoon>
+ http://darnassus.sceen.net/gitweb/teythoon/mig.git/commitdiff/64b7d34f90a41d017a9e1e8179c0533a97012f6f
+ <braunr> makes sense
+ <teythoon> this makes mig payload aware
+ <teythoon> we'd specify another intrans function that takes a label and
+ returns an object
+ <braunr> let me remind
+ <braunr> you said there were 3 lookups actually
+ <braunr> the mach one
+ <braunr> the libports one
+ <braunr> and is the intran one the last, right ?
+ <teythoon> yes
+ <teythoon> so now i optimized away the second one, the one in libports
+ <braunr> ok so you need intran aware functions to replace that lookup
+ <braunr> well
+ <braunr> payload aware intran functions
+ <teythoon> yes
+ <braunr> ok
+ <teythoon> mostly cast the label, ports_port_ref the object
+ <braunr> i assume they'd be pretty straightforward
+ <teythoon> yes
+ <braunr> and easy to add for all existing intran functions
+ <teythoon> most likely
+ <braunr> the proposed change looks very appropriate
+ <teythoon> :)
+ <braunr> i'd never thought about intran functions because i didn't want
+ that in my clone ;p
+ <braunr> they do add a bit of complexity
+ <braunr> but this upgrade path looks right
+ <teythoon> yes
+ <teythoon> I think so too
+ <braunr> nothing more to say :)
+ <braunr> it's so simple i actually don't understand how i could miss it
+ last time i looked
+ <braunr> i guess i was exhausted heh
+ <teythoon> thanks for the review :)
+ <braunr> thanks for your work
+ <braunr> it's been a long time since we had someone spend that much time on
+ the hurd
+
+
+## IRC, freenode, #hurd, 2013-11-29
+
+ <teythoon> I came to believe that there is actually a lot of room for
+ improvement in our rpc path
+
+
+## IRC, freenode, #hurd, 2013-12-19
+
+ <braunr> teythoon_: how is protected payload branch now ?
+ <braunr> ready for review ?
+ <teythoon_> the kernel and mig patch are
+ <teythoon_> patches
+ <braunr> so pending for approval rathr
+ <braunr> rather
+ <teythoon_> documentation is still missing for those ofc
+ <braunr> the last parts are the mig mutations iirc
+ <teythoon_> err, you lost me
+ <teythoon_> i haven't continued to work on the hurd patch series
+ <teythoon_> the patch series for gnu mach and mig are feature complete from
+ my point of view
+ <braunr> i mean the changes needed to remove the third lookup in the
+ mutation functions
+ <teythoon_> to do that in hurd, we need a patched mig
+ <braunr> i was just trying to remember correctly
+ <teythoon_> those patches need to be reviewed
+ <teythoon_> the hurd patch series is not yet working, but you can see the
+ approach i've taken
+ <braunr> yes
+ <braunr> ok
+ <teythoon_> the next thing i'd do in this regard is to fix all object
+ lookups
+ <braunr> so it didn't change from last time i looked
+ <teythoon_> no
+ <teythoon_> some code, notoriously the control port handling in the *fs
+ libs, uses mach_port_t for the receiver and do the lookup themself. i'd
+ fix that next.
+
+
+## IRC, freenode, #hurd, 2014-01-20
+
+ <teythoon> i've tied up enough loose ends, that i can start working on the
+ protected payload stuff again
+ <teythoon> the next step is fixing the receiver lookups everywhere
+ <braunr> good :)
+ <teythoon> if everyone uses mig magic for that, the switch will be easy
+ <teythoon> undoing the hack in mach-defpager too
+
+
+## IRC, freenode, #hurd, 2014-01-24
+
+ <braunr> teythoon: what are you currently working on ? protected payload ?
+ <teythoon> braunr: yes, i'm working with coccinelle to fix all object
+ lookups in the hurd
+ <teythoon> i figured it is easier and cleaner to just fix them instead of
+ converting pointers back to port names for those functions that want port
+ names
+
+
+## IRC, freenode, #hurd, 2014-02-17
+
+ <teythoon> braunr: do you think it's okay to make the 0 payload special ?
+ <braunr> teythoon: for our needs, sure
+ <braunr> it's similar to NULL or MACH_PORT_NULL
+ <teythoon> yes
+ <teythoon> maybe i should add a symbolic name for that
+ <teythoon> for consistency
+ <braunr> but is it wise to let mach_port_set_protected_payload reset the
+ behaviour on a zero payload ?
+ <braunr> i don't think a symbolic name is needed
+ <braunr> or maybe
+ <braunr> as you want
+ <teythoon> what else should it do then ?
+ <braunr> 00:25 < braunr> but is it wise to let
+ mach_port_set_protected_payload reset the behaviour on a zero payload ?
+ <braunr> it could return invalid argument instead
+ <braunr> and the documentation would clearly state 0 is invalid
+ <braunr> but that would also prevent reverting the mode
+ <teythoon> yes, i consider that not really useful, but i'd be okay with the
+ current behavior
+ <teythoon> but yes, the documentation should make that clear
+
+
+## IRC, freenode, #hurd, 2014-02-22
+
+ <teythoon> braunr: once the pp patch set is in gnumach, i'll make
+ mach-defpager use it
+ <teythoon> it's a good target, as it does not use libports
+ <teythoon> and it's currently abusing the port rename procedure for the
+ lookup, making the rights spill into the splay tree
+ <teythoon> braunr: the wiki mentioned that you once considered to remove
+ the ability to rename ports altogether
+ <braunr> teythoon: ah right
+ <braunr> i actually intend to keep it for x15, but only because i want port
+ names to blend as file descriptors
+ <teythoon> right, for dup and friends
+ <braunr> and the radix tree is a data structure that can cope decently with
+ not too sparsed distributions