From 95878586ec7611791f4001a4ee17abf943fae3c1 Mon Sep 17 00:00:00 2001 From: "https://me.yahoo.com/a/g3Ccalpj0NhN566pHbUl6i9QF0QEkrhlfPM-#b1c14" Date: Mon, 16 Feb 2015 20:08:03 +0100 Subject: rename open_issues.mdwn to service_solahart_jakarta_selatan__082122541663.mdwn --- open_issues/dde.mdwn | 661 --------------------------------------------------- 1 file changed, 661 deletions(-) delete mode 100644 open_issues/dde.mdwn (limited to 'open_issues/dde.mdwn') diff --git a/open_issues/dde.mdwn b/open_issues/dde.mdwn deleted file mode 100644 index e7083557..00000000 --- a/open_issues/dde.mdwn +++ /dev/null @@ -1,661 +0,0 @@ -[[!meta copyright="Copyright © 2010, 2011, 2012, 2013, 2014 Free Software -Foundation, Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -[[!tag open_issue_hurd open_issue_gnumach]] - -[[General Information|/dde]]. - -Still waiting for interface finalization and proper integration. - -[[!toc]] - -See [[user-space_device_drivers]] for generic discussion related to user-space -device drivers. - - -# Disk Drivers - -Not yet supported. - -The plan is to use [[libstore_parted]] for accessing partitions. - - -# Upstream Status - - -## IRC, freenode, #hurd, 2012-02-08 - -After the microkernel devroom at [[community/meetings/FOSDEM_2012]]: - - there was quite some talk about DDE. I learnt that there are newer - versions in Genode and in Minix (as opposed to the DROPS one we are - using) - but apparently none of the guys involved is interested in creating - a proper upstream project with central repository and communication - channels :-( - the original DDE creator was also there, but said he isn't working - on it anymore - OK, and the other two projects basically have their own forks. - Or are they actively cooperating? - (If you know about it.) - well, Genode is also part of the Dresden L4 group; but apart from - that, I'd rather call it a fork... - hm... actually, I'm not sure anymore whether the guy I talked to - was from Genode or Nova... - (both from the Dresdem L4 group) - - -### IRC, freenode, #hurd, 2012-08-12 - - - http://genode.org/documentation/release-notes/12.05#Re-approaching_the_Linux_device-driver_environment - I wonder whether the very detailed explanation was prompted by our - DDE discussions at FOSDEM... - antrik: one could think about approaching them to develop the - common dde libs + dde_linux together - pinotree: that's what I did at FOSDEM -- they weren't interested - antrik: this year's one? why weren't they? - maybe at that time dde was not integrated properly yet (netdde - is just few months "old") - do you really consider it integrated properly ? - no, but a bit better than last year - I don't see what our integration has to do with anything... - they just prefer hacking thing ad-hoc than having some central - usptream - the helenos people? - err... how did helenos come into the picture?... - we are talking about genode - sorry, confused wrong microkernel OS - actually, I don't remember exactly who said what; there were - people from genode there and from one or more other DDE projects... but - none of them seemed interested in a common DDE - err... one or two other L4 projects - - -## IRC, freenode, #hurd, 2012-02-19 - - antrik: do we know exactly which DDE version Zheng Da took as a - base ? - (so as to be able to merge new changes easily) - youpi: not sure... but from what I gathered at FOSDEM, the version - he based on (from DROPS) is not really actively developed right now; if - we want to go for newer versions, we probably have to look at other - projects (like Genode or Nova or Minix) - there's no central project for dde ? - that sucks - no... and nobody seemed interested in having one :-( - pff - which makes me seriously question the original decision to build - on DDE... - .. - if we have to basically maintain it ourselfs anyways, we could - just as well have gone with custom glue - well, the advantage of DDE is that it already exists now - on the positive side, one of the projcets (not sure which) - apparently have both USB and SATA working with some variant of DDE - - -### IRC, freenode, #hurd, 2012-11-03 - - DrChaos: there is DDEUSB framework for L4. You could port it, if - you want. It uses Linux 2.6.26 usb subsystem. - - -## IRC, freenode, #hurd, 2013-02-15 - -After the microkernel devroom at [[community/meetings/FOSDEM_2013]]. - - youpi: speaking of dde, was there any will among other - microkernel os developers to eventually develop one single dde (with - every team handling the custom glue of the own kernel)? - well, there is still upstream dde actually - in dresden - nothing was really decided or anything (it was a round table, not a - workgroup) - but conversation converged into sharing the DDE maintenance, yes - and dresden would be the logical central place - pb is that they don't have the habit of being very open - http://svn.tudos.org/repos/oc/tudos/trunk/l4/pkg/dde has a recent - enough version - which macsim confirmed having all the latest commits from the - internal repository - i see - so it seems a viable solution on the medium term - the long term might need a real visible open source project - but we should probably still keep basing on dresden work - (better take work being done anywhere) - well, if the upstream is not really open, microkernel teams - could just fork it and all work on it - that's what I mean - should still be a win than everybody maintaining their own dde - sure - ah yes, i was writing and i'm slow at it :) - but at least we can try to work with dresden - see how open they could become by just asking :) - right - - -# IRC, OFTC, #debian-hurd, 2012-02-15 - - i have no idea how the dde system works - gnumach patch to provide access to physical memory and interrupts - then userland accesses i/o ports by hand to drive things - but that assumes that no kernel driver is interfering - so one has to disable kernel drivers - how are dde drivers used? can they be loaded on their own - automatically, or you have to settrans yourself to setup a device? - there's no autoloader for now - we'd need a bus arbitrer that'd do autoprobing - -[[PCI_arbiter]]. - - i see - (you see i'm not really that low level, so pardon the flood of - posssibly-noobish questions ;) ) - I haven't set it up yet, but IIRC you need to specify which driver - to be used - well, I mostly have the same questions actually :) - I just have some guesswork here :) - i wonder whether the following could be feasible: - I'm wondering how we'll manage to make it work in d-i - a) you create a package which would b-d on linux-source, build a - selection of (network only for now) drivers and install them in - /hurd/dde/ - probably through a choice at the boot menu - I wouldn't dare depending on linux-source - dde is usually not up-to-date - b) add a small utility over the actual fsys_settrans() which - would pick the driver from /hurd/dde/ - ... so you could do `set-dde-driver b43 ` (or something - like that) - we can provide something like b) yes - although documenting the settrans should be fine enough ;) - the a) would help/ease with the fact that you need to compile on - your own the drivers - otherwise we would need to create a new linux-dde-sources-X.Y.Z - only with the sources of the drivers we want from linux X.Y.Z - (or hurd-dde-linux-X.Y.Z) - samuel.thibault * raccdec3 gnumach/debian/ (changelog - patches/70_dde.patch patches/series): - Add DDE experimental support - * debian/patches/70_dde.patch: Add experimental support for irq - passing and - physical memory allocation for DDE. Also adds nonetdev boot - parameter to - disable network device drivers. - ok, boots fine with the additional nonetdev option - now I need to try that dde hurd branch :) - samuel.thibault * rf8b9426 gnumach/debian/patches/70_dde.patch: Add - experimental.defs to gnuamch-dev - - -# IRC, freenode, #hurd, 2012-02-19 - - * youpi got dde almost working - it's able to send packets, but apparently not receive them - (e1000) - ok, rtl8139 works - antrik: the wiki instructions are correct - with e1000 I haven't investigated - (Archhurd guys also reported problems with e1000 IIRC... the one I - built a while back works fine though on my T40p with real e1000 NIC) - maybe I should try with current versions... something might got - broken by later changes :-( - at least testing could tell the changeset which breaks it - Mmm, it's very odd - with the debian package, pfinet's call to device_set_filter returns - D_INVALID_OPERATION - and indeed devnode.c returns that - ah but it's libmachdev which is supposed to answer here - youpi: so, regarding the failing device_set_filter... I guess you - are using some wrong combination of gnumach and pfinet - no it's actually that my pfinet was not using bpf - I've now fixed it - the DDE drivers rely on zhengda's modified pfinet, which uses - devnode, but also switched to using proper BPF filters. so you also need - his BPF additions/fixes in gnumach - OK - that's the latter - I had already fixed the devnode part - but hadn't seen that the filter was different - err... did I say gnumach? that of course doesn't come into play - here - so yes, you just need a pfinet using BPF - libmachdev does ;) - I'm just using pfinet from zhengda's DDE branch... I think devnode - and BPF are the only modifications - there's also a libpcap modification in the incubator - probably for tcpdump etc. - libpcap is used by the modified pfinet to compile the filter rule - why does pfinet need to compile the rule ? - it's libbpf which is used in the dde driver - it doesn't strictly need to... but I guess zhengda considered it - more elegant to put the source rule in pfinet on compile it live, rather - than the compiled blob - I probably discussed this with him myself a few years back... but - my memory on this is rather hazy ;-) - err... and compile it live - ah, right, it's only used when asking pfinet to change its filter - but it does not need it for the default filter - which is hardcoded - I see - when would pfinet change its filter? - * youpi now completely converting his hurd box to debian packages with dde - support - on SIOCSIFADDR apparently - to set "arp or (ip host %s)", - well, that sounds like the default filter... - the default filter does not choose an IP - oh, right... pfinet has to readjust the filter when setting the IP - arg we lack support for kernel options for gnumach in update-grub - again, I have a vague recollection of discussing this - * youpi crosses fingers - yay, works - so we *do* need libpcap in pfinet to set proper rules... though I - guess it can also work with a static catchall rule (like it did before - zhengda's changes), only less efficient - well in the past we were already catching everything anyway, so at - least it's not a regression :) - right - - -# [[PCI_Arbiter]] - -## IRC, freenode, #hurd, 2012-02-21 - - since all drivers need is interrupts, io ports and iomem - the latter was already available through /dev/mem - io ports through the i386 rpcs - the changes provide both interrupts, and physical-contiguous - allocation - it should be way enough - youpi: ok - youpi: thanks for the details :) - braunr: this was mentioned in the context of the interrupt - forwarding interface... the original one implemented by zhengda isn't - suitable for a PCI server; but the ones proposed by youpi and tschwinge - would work - same for the physical memory interface: the current implementation - doesn't allow delegation; but I already said that it's wrong - - -# IRC, freenode, #hurd, 2012-02-20 - - I was a bit wary of including the ton of dde headers in hurd-dev - maybe adding another package for that - but that would have delayed introducing the dde binaries - probably we can do that for next upload - i can try to work on it, if is feasible (ie if the dde drivers - can currently be built from outside the hurd source tree) - it should be, it's a matter of pointing its makefile to a place - where the make scripts and include headers are - (and the libraries) - ok - youpi: you mean DDEKit headers? - pinotree: actually it doesn't matter where the dde-ified Linux - drivers are built -- libdde_linux26 and the actual drivers use a - completetly different build system anyways - in fact we concluded at some point that they should live in a - separate repository -- but that change never happened - only the base stuff (ddekit, libmachdev etc.) belong in the Hurd - source tree - antrik: yes - antrik: err, not really completely different - the actual drivers' Makefile include the libdde_linux26 mk files - the build itself is separate, though - youpi: yes, I mean both libdde_linux26 and the drivers use a build - system that is completely distinct from the Hurd one - ah, yes - libdde_linux26 should however be shipped in the system - ideally libdde_linux26 and all the drivers should be built in one - go I'd say... - it should be easily feasible to also have a separate driver too - e.g. to quickly try a 2.6 driver - youpi: I'm not sure about that. it's not even dynamically linked - IIRC?... - with scripts to build it - it's not - but that doesn't mean it can't be separate - .a files are usually shipped in -dev packages - youpi: ideally we should try to come with a build system that - reuses the original Linux makefile snippets to build all the drivers - automatically without any manual per-driver work - there's usually no modification of the drivers themselves? - but yeah - "ideally", when somebody takes the time to do it - unfortunately, it's necessary to include one particular - Hurd/DDE-specific header file in each driver :-( - can't it be done through gcc's -include option? - zhengda didn't find a way to avoid this... though I still hope - that it must be possible somehow - I think the problem is that it has to be included *after* the - other Linux headers. don't remember the details though - uh - well, a good script can add a line after the last occurrence of - #include - yeah... rather hacky, but might work - even with a bit of grep, tail, cut, and sed it should work :) - note that this is Hurd-specific; the L4 guys didn't need that - what is it? - don't remember off-hand - - -# IRC, freenode, #hurd, 2012-02-22 - - antrik: AIUI, it should be possible to include all network drivers - in just one binary? - that'd permit to use it in d-i - and completely replace the mach drivers - we just need to make sure to include at least what the mach drivers - cover - (all DDE network drivers, I mean) - of course that doesn't hinder from people to carefully separate - drivers in several binaries if they wish - antrik: it does link at least, I'll give a try later - yes it works! - that looks like a plan - throw all network drivers in a /hurd/dde_net - settrans it on /dev/dde_net, and settrans devnode on /dev/eth[0-9] - I'm also uploading a version of hurd which includes headers & - libraries, so you just need a make in dde_{e100,e1000,etc,net} - (uploading it with the dde driver itself :) ) - btw, a nice thing is that we don't really care that all drivers are - stuffed into a single binary, since it's a normal process only the useful - pages are mapped and actually take memory :) - is that really a nice thing though? compared to other systems I - mean - I know on linux it only loads the modules I need, for example. It's - definitely a step up for hurd though :D - that's actually precisely what I mean - you don't need to load only the modules you need - you just load them all - and paging eliminates automatically what's not useful - even parts of the driver that your device will not need - ooh - awesome - (actually, it's not even loaded, but the pci tables of the drivers - are loaded, then paged out) - - -# IRC, freenode, #hurd, 2012-02-24 - - antrik_: about the #include , I see the issue, it's - about jiffies - it wouldn't be a very good thing to have a jiffies variable which - we'd have to update 100times per second - so ZhengDa preferred to make jiffies a macro which calls a function - which reads the mapped time - -[[Mapped-time_interface|microkernel/mach/gnumach/interface/device/time]]. - - however, that break any use of the work "jiffies", e.g. structure - members & such - actually it's not only after headers that the #include has to be - done, but after any code what uses the word "jiffies" for something else - than the variable - pb is: it has to be done *before* any code that uses the word - "jiffies" for the variable, e.g. inline functions in headers - in l4dde, there's already the jiffies variable so it's not a - problem - - -# IRC, OFTC, #debian-hurd, 2012-02-27 - - I plan to do some light performance testing w.r.t. DDE - Ethernet. That is DDE vs. Mach, etc. - that'd be good, indeed - I'm getting 4MiB/s with dde - I don't remember with mach - Yes. It just shouldn't regress too much. - Aha, OK. - - -## IRC, freenode, #hurd, 2012-02-27 - - tschwinge: nttcp tells me ~80Mbps for mach-rtl8139, ~72Mbps for - dde-rtl8139, ~72Mbps for dde-e1000 - civodul: ↑ btw - youpi: so the dde network device is not much slower than the - kernel-one? - youpi: yes, looks good - rather almost the same speed - apparently - that’s quite a deal. - what speed should it have as maximum? - (means: does the mach version get out all that’s possible?) - differently put: What speed would GNU/Linux get? - I'm checking that right now - cool! - we need those numbers for the moth after the next - Mmm, I'm not sure you really want the linux number :) - 1.6Gbps :) - oh… - let me check with udp rather than tcp - so the Hurd is a “tiny bit” worse at using the network… - it might simply be an issue with tcp tuning in pfinet - hm, yes - tcp is not that cheap - and has some pretty advanced stuff for getting to high speeds - well, I'm not thinking about being cheap - but using more recent tuning - that does not believe only 1Mbps network cards exist :) - like adaptive windows and such? - :) - yes - * ArneBab remembers that TCP was invented when the connections went over - phone lines - by audio :) - yep - what’s the system load while doing the test? - yes, udp seems not so bad - ah, cool! - it's very variable (300-3000Mbps), but like on linux - that pushing it into user space has so low cost is pretty nice. - * ArneBab thinks that that’s a point where Hurd pays off - that's actually what AST said to fosdem - he doesn't care about putting an RPC for each and every port i/o - because hardware is slow anyway - jupp - but it is important to see that in real life - - -# IRC, freenode, #hurd, 2012-04-01 - - antrik: I wonder whether you could actually not route the IRQs to a - non-zero ring, AIUI you can in the x86 IDT table - youpi: you mean having a userspace server for each (non-timer) - interrupt? - youpi: how would a userspace IRQ handler interact with the - scheduler? - antrik: it doesn't necessarily have to - provided that it's trusted - youpi: how would you do CPU time accounting if there is no - interaction with the scheduler?... - antrik: you don't necessarily want to care about it - youpi: well, that would mean that all drivers handling interrupts - would have to be trusted to not use more than a very small part of CPU - time... - yes - which is usually needed for interrupt handlers anyway - youpi: nah, the bottom handler only has to do very basic stuff; - afterwards, we can pass off to "normal" driver processes, scheduled just - like other processes... but that requires some interaction between the - IRQ handler and the scheduler I think - the IRQ handler can wake up a thread, yes - no need for anything special there - so the userspace IRQ server would just decide what process to wake - up, and then call the scheduler to do a normal task switch? I guess - that's possible; but I'm not sure it would buy much... - it would permit userland to quickly react to the IRQ - such as acknowledge it to the hardware etc. - yeah, but my point is that I don't see much benefit in having this - part of the code isolated in a userspace process... it has to be trusted - anyways, and it's pretty trivial too - I never said it was a good idea - - -# IRC, freenode, #hurd, 2012-04-06 - - oh i forgot about my work on pcap - is devnode (or devopen or whatever) in the upstream repository now - ? - can't say for sure, but I'd be surprised... don't remember seeing - any movement in that regard :-( - wasn't it needed for dde ? - hm... good point - - -## IRC, freenode, #hurd, 2013-09-20 - - i should take some time to integrate my pcap changes into the - libpcap debian package at least - braunr: if upstream is active, i'd say to go there directly - the problem with that approach is that netdde is still not part of - our upstream code - don't understand the relation - i don't want to send the pcap guys code for an interface that is - still not considered upstream ... - - -# IRC, freenode, #hurd, 2012-08-14 - - it's amazing how much code just gets reimplemented needlessly ... - libddekit has its own mutex, condition, semaphore etc.. objects - with the *exact* same comment about the dequeueing-on-timeout - problem found in libpthread - *sigh* - - -## IRC, freenode, #hurd, 2012-08-18 - - hum, leaks and potential deadlocks in libddekit/thread.c :/ - - -## IRC, freenode, #hurd, 2012-08-18 - - nice, dde relies on a race to start .. - - -## IRC, freenode, #hurd, 2012-08-21 - -In context of [[libpthread]]. - - hm, i thought my pthreads patches introduced a deadlock, but - actually this one is present in the current upstream/debian code :/ - (the deadlock occurs when receiving data fast with sftp) - either in netdde or pfinet - - -## IRC, freenode, #hurd, 2013-02-28 - - (which needs the same kinds of fixes that libpthread got) - actually i'm not sure why he didn't simply reuse the pthread - functions :/ - which kind of fixes? - cancellation? - timeouts - cancellation too but that's less an issue - I'm not sure it really needs timeout work - on what RPC? - pfinet is just using the mach interface - i don't know but it clearly copies some of the previous pthread - code from pthread_cond_timedwait - see libddekit/thread.c:_sem_timedwait_internal - I recognize the comment indeed :) - I guess he thought he might need some particular semantic that - libpthread may not provide - also, now that i think about it, he couldn't have used libpthread, - could he ? - and there was no condition_timedwait in cthreads - there is a deadlock in netdde - it occurs sometimes, at high network speeds - (well high, 4 MiB/s or more) - - -## IRC, freenode, #hurd, 2013-11-20 - - for example, netdde needs more reviewing and polishing - it is known to deadlock sometimes - what deadlocks ? - i'm not sure - ah, netdde - right - yes - I'm seeing that to on one of my vms - nasty one - i know something is wrong with the condition_wait_timeout function - for example - breaks sysvinit shutdown - because it was taken without modification from libpthread - it might be that, or something else - well, dhclient hangs releasing the lease - that's still on my todo list - so I'm pretty sure it's related - hm - maybe - :/ - - -## IRC, freenode, #hurd, 2014-02-11 - - teythoon: looks like a netdde/pfinet freeze/deadlock - yes a netdde deadlock - i really have to fix that too one day :( - hehe :) - the netdde locking privimites are copies of the "old" pthread - ones, instead of reusing pthread - primitives* - - -## IRC, freenode, #hurd, 2014-03-08 - - what to do if network freezes? - gg0: depends on what caused the freeze - gg0: you could try to kill the netdde process - it's just apt-get'ing, download phase - yess kill netdde - there are known deadlocks in netdde - - -# IRC, freenode, #hurd, 2012-08-18 - - hm looks like if netdde crashes, the kernel doesn't handle it - cleanly, and we can't attach another netdde instance - -[[!message-id "877gu8klq3.fsf@kepler.schwinge.homeip.net"]] - - -# DDE for Filesystems - -## IRC, freenode, #hurd, 2012-10-07 - - * pinotree wonders whether the dde layer could aldo theorically support - also file systems - pinotree: yeah, I also brought up the idea of creating a DDE - extension or DDE-like wrapper for Linux filesystems a while back... don't - know enough about it though to decide whether it's doable - OTOH, I'm not sure it would be worthwhile. we still should - probably have a native (not GPLv2-only) implementation for the main FS at - least; so the wrapper would only be for accessing external - partitions/media... - - -## IRC, freenode, #hurd, 2013-12-03 - - how about porting linux block device layer via dde as mcsim wanted to - do? then all linux filesystems could be brought in, right? - gg0: that should be done, but we need to correctly deal with - multiple pci devices in userspace and arbitration - wouldn't adding support to passive translator into Linux - filesystems be quite some work ? IIRC ext2fs needs a special "owner = - hurd" mode to handle them - - -# [[virtio]] -- cgit v1.2.3