From 95878586ec7611791f4001a4ee17abf943fae3c1 Mon Sep 17 00:00:00 2001 From: "https://me.yahoo.com/a/g3Ccalpj0NhN566pHbUl6i9QF0QEkrhlfPM-#b1c14" Date: Mon, 16 Feb 2015 20:08:03 +0100 Subject: rename open_issues.mdwn to service_solahart_jakarta_selatan__082122541663.mdwn --- .../dde.mdwn | 661 +++++++++++++++++++++ 1 file changed, 661 insertions(+) create mode 100644 service_solahart_jakarta_selatan__082122541663/dde.mdwn (limited to 'service_solahart_jakarta_selatan__082122541663/dde.mdwn') diff --git a/service_solahart_jakarta_selatan__082122541663/dde.mdwn b/service_solahart_jakarta_selatan__082122541663/dde.mdwn new file mode 100644 index 00000000..e7083557 --- /dev/null +++ b/service_solahart_jakarta_selatan__082122541663/dde.mdwn @@ -0,0 +1,661 @@ +[[!meta copyright="Copyright © 2010, 2011, 2012, 2013, 2014 Free Software +Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd open_issue_gnumach]] + +[[General Information|/dde]]. + +Still waiting for interface finalization and proper integration. + +[[!toc]] + +See [[user-space_device_drivers]] for generic discussion related to user-space +device drivers. + + +# Disk Drivers + +Not yet supported. + +The plan is to use [[libstore_parted]] for accessing partitions. + + +# Upstream Status + + +## IRC, freenode, #hurd, 2012-02-08 + +After the microkernel devroom at [[community/meetings/FOSDEM_2012]]: + + there was quite some talk about DDE. I learnt that there are newer + versions in Genode and in Minix (as opposed to the DROPS one we are + using) + but apparently none of the guys involved is interested in creating + a proper upstream project with central repository and communication + channels :-( + the original DDE creator was also there, but said he isn't working + on it anymore + OK, and the other two projects basically have their own forks. + Or are they actively cooperating? + (If you know about it.) + well, Genode is also part of the Dresden L4 group; but apart from + that, I'd rather call it a fork... + hm... actually, I'm not sure anymore whether the guy I talked to + was from Genode or Nova... + (both from the Dresdem L4 group) + + +### IRC, freenode, #hurd, 2012-08-12 + + + http://genode.org/documentation/release-notes/12.05#Re-approaching_the_Linux_device-driver_environment + I wonder whether the very detailed explanation was prompted by our + DDE discussions at FOSDEM... + antrik: one could think about approaching them to develop the + common dde libs + dde_linux together + pinotree: that's what I did at FOSDEM -- they weren't interested + antrik: this year's one? why weren't they? + maybe at that time dde was not integrated properly yet (netdde + is just few months "old") + do you really consider it integrated properly ? + no, but a bit better than last year + I don't see what our integration has to do with anything... + they just prefer hacking thing ad-hoc than having some central + usptream + the helenos people? + err... how did helenos come into the picture?... + we are talking about genode + sorry, confused wrong microkernel OS + actually, I don't remember exactly who said what; there were + people from genode there and from one or more other DDE projects... but + none of them seemed interested in a common DDE + err... one or two other L4 projects + + +## IRC, freenode, #hurd, 2012-02-19 + + antrik: do we know exactly which DDE version Zheng Da took as a + base ? + (so as to be able to merge new changes easily) + youpi: not sure... but from what I gathered at FOSDEM, the version + he based on (from DROPS) is not really actively developed right now; if + we want to go for newer versions, we probably have to look at other + projects (like Genode or Nova or Minix) + there's no central project for dde ? + that sucks + no... and nobody seemed interested in having one :-( + pff + which makes me seriously question the original decision to build + on DDE... + .. + if we have to basically maintain it ourselfs anyways, we could + just as well have gone with custom glue + well, the advantage of DDE is that it already exists now + on the positive side, one of the projcets (not sure which) + apparently have both USB and SATA working with some variant of DDE + + +### IRC, freenode, #hurd, 2012-11-03 + + DrChaos: there is DDEUSB framework for L4. You could port it, if + you want. It uses Linux 2.6.26 usb subsystem. + + +## IRC, freenode, #hurd, 2013-02-15 + +After the microkernel devroom at [[community/meetings/FOSDEM_2013]]. + + youpi: speaking of dde, was there any will among other + microkernel os developers to eventually develop one single dde (with + every team handling the custom glue of the own kernel)? + well, there is still upstream dde actually + in dresden + nothing was really decided or anything (it was a round table, not a + workgroup) + but conversation converged into sharing the DDE maintenance, yes + and dresden would be the logical central place + pb is that they don't have the habit of being very open + http://svn.tudos.org/repos/oc/tudos/trunk/l4/pkg/dde has a recent + enough version + which macsim confirmed having all the latest commits from the + internal repository + i see + so it seems a viable solution on the medium term + the long term might need a real visible open source project + but we should probably still keep basing on dresden work + (better take work being done anywhere) + well, if the upstream is not really open, microkernel teams + could just fork it and all work on it + that's what I mean + should still be a win than everybody maintaining their own dde + sure + ah yes, i was writing and i'm slow at it :) + but at least we can try to work with dresden + see how open they could become by just asking :) + right + + +# IRC, OFTC, #debian-hurd, 2012-02-15 + + i have no idea how the dde system works + gnumach patch to provide access to physical memory and interrupts + then userland accesses i/o ports by hand to drive things + but that assumes that no kernel driver is interfering + so one has to disable kernel drivers + how are dde drivers used? can they be loaded on their own + automatically, or you have to settrans yourself to setup a device? + there's no autoloader for now + we'd need a bus arbitrer that'd do autoprobing + +[[PCI_arbiter]]. + + i see + (you see i'm not really that low level, so pardon the flood of + posssibly-noobish questions ;) ) + I haven't set it up yet, but IIRC you need to specify which driver + to be used + well, I mostly have the same questions actually :) + I just have some guesswork here :) + i wonder whether the following could be feasible: + I'm wondering how we'll manage to make it work in d-i + a) you create a package which would b-d on linux-source, build a + selection of (network only for now) drivers and install them in + /hurd/dde/ + probably through a choice at the boot menu + I wouldn't dare depending on linux-source + dde is usually not up-to-date + b) add a small utility over the actual fsys_settrans() which + would pick the driver from /hurd/dde/ + ... so you could do `set-dde-driver b43 ` (or something + like that) + we can provide something like b) yes + although documenting the settrans should be fine enough ;) + the a) would help/ease with the fact that you need to compile on + your own the drivers + otherwise we would need to create a new linux-dde-sources-X.Y.Z + only with the sources of the drivers we want from linux X.Y.Z + (or hurd-dde-linux-X.Y.Z) + samuel.thibault * raccdec3 gnumach/debian/ (changelog + patches/70_dde.patch patches/series): + Add DDE experimental support + * debian/patches/70_dde.patch: Add experimental support for irq + passing and + physical memory allocation for DDE. Also adds nonetdev boot + parameter to + disable network device drivers. + ok, boots fine with the additional nonetdev option + now I need to try that dde hurd branch :) + samuel.thibault * rf8b9426 gnumach/debian/patches/70_dde.patch: Add + experimental.defs to gnuamch-dev + + +# IRC, freenode, #hurd, 2012-02-19 + + * youpi got dde almost working + it's able to send packets, but apparently not receive them + (e1000) + ok, rtl8139 works + antrik: the wiki instructions are correct + with e1000 I haven't investigated + (Archhurd guys also reported problems with e1000 IIRC... the one I + built a while back works fine though on my T40p with real e1000 NIC) + maybe I should try with current versions... something might got + broken by later changes :-( + at least testing could tell the changeset which breaks it + Mmm, it's very odd + with the debian package, pfinet's call to device_set_filter returns + D_INVALID_OPERATION + and indeed devnode.c returns that + ah but it's libmachdev which is supposed to answer here + youpi: so, regarding the failing device_set_filter... I guess you + are using some wrong combination of gnumach and pfinet + no it's actually that my pfinet was not using bpf + I've now fixed it + the DDE drivers rely on zhengda's modified pfinet, which uses + devnode, but also switched to using proper BPF filters. so you also need + his BPF additions/fixes in gnumach + OK + that's the latter + I had already fixed the devnode part + but hadn't seen that the filter was different + err... did I say gnumach? that of course doesn't come into play + here + so yes, you just need a pfinet using BPF + libmachdev does ;) + I'm just using pfinet from zhengda's DDE branch... I think devnode + and BPF are the only modifications + there's also a libpcap modification in the incubator + probably for tcpdump etc. + libpcap is used by the modified pfinet to compile the filter rule + why does pfinet need to compile the rule ? + it's libbpf which is used in the dde driver + it doesn't strictly need to... but I guess zhengda considered it + more elegant to put the source rule in pfinet on compile it live, rather + than the compiled blob + I probably discussed this with him myself a few years back... but + my memory on this is rather hazy ;-) + err... and compile it live + ah, right, it's only used when asking pfinet to change its filter + but it does not need it for the default filter + which is hardcoded + I see + when would pfinet change its filter? + * youpi now completely converting his hurd box to debian packages with dde + support + on SIOCSIFADDR apparently + to set "arp or (ip host %s)", + well, that sounds like the default filter... + the default filter does not choose an IP + oh, right... pfinet has to readjust the filter when setting the IP + arg we lack support for kernel options for gnumach in update-grub + again, I have a vague recollection of discussing this + * youpi crosses fingers + yay, works + so we *do* need libpcap in pfinet to set proper rules... though I + guess it can also work with a static catchall rule (like it did before + zhengda's changes), only less efficient + well in the past we were already catching everything anyway, so at + least it's not a regression :) + right + + +# [[PCI_Arbiter]] + +## IRC, freenode, #hurd, 2012-02-21 + + since all drivers need is interrupts, io ports and iomem + the latter was already available through /dev/mem + io ports through the i386 rpcs + the changes provide both interrupts, and physical-contiguous + allocation + it should be way enough + youpi: ok + youpi: thanks for the details :) + braunr: this was mentioned in the context of the interrupt + forwarding interface... the original one implemented by zhengda isn't + suitable for a PCI server; but the ones proposed by youpi and tschwinge + would work + same for the physical memory interface: the current implementation + doesn't allow delegation; but I already said that it's wrong + + +# IRC, freenode, #hurd, 2012-02-20 + + I was a bit wary of including the ton of dde headers in hurd-dev + maybe adding another package for that + but that would have delayed introducing the dde binaries + probably we can do that for next upload + i can try to work on it, if is feasible (ie if the dde drivers + can currently be built from outside the hurd source tree) + it should be, it's a matter of pointing its makefile to a place + where the make scripts and include headers are + (and the libraries) + ok + youpi: you mean DDEKit headers? + pinotree: actually it doesn't matter where the dde-ified Linux + drivers are built -- libdde_linux26 and the actual drivers use a + completetly different build system anyways + in fact we concluded at some point that they should live in a + separate repository -- but that change never happened + only the base stuff (ddekit, libmachdev etc.) belong in the Hurd + source tree + antrik: yes + antrik: err, not really completely different + the actual drivers' Makefile include the libdde_linux26 mk files + the build itself is separate, though + youpi: yes, I mean both libdde_linux26 and the drivers use a build + system that is completely distinct from the Hurd one + ah, yes + libdde_linux26 should however be shipped in the system + ideally libdde_linux26 and all the drivers should be built in one + go I'd say... + it should be easily feasible to also have a separate driver too + e.g. to quickly try a 2.6 driver + youpi: I'm not sure about that. it's not even dynamically linked + IIRC?... + with scripts to build it + it's not + but that doesn't mean it can't be separate + .a files are usually shipped in -dev packages + youpi: ideally we should try to come with a build system that + reuses the original Linux makefile snippets to build all the drivers + automatically without any manual per-driver work + there's usually no modification of the drivers themselves? + but yeah + "ideally", when somebody takes the time to do it + unfortunately, it's necessary to include one particular + Hurd/DDE-specific header file in each driver :-( + can't it be done through gcc's -include option? + zhengda didn't find a way to avoid this... though I still hope + that it must be possible somehow + I think the problem is that it has to be included *after* the + other Linux headers. don't remember the details though + uh + well, a good script can add a line after the last occurrence of + #include + yeah... rather hacky, but might work + even with a bit of grep, tail, cut, and sed it should work :) + note that this is Hurd-specific; the L4 guys didn't need that + what is it? + don't remember off-hand + + +# IRC, freenode, #hurd, 2012-02-22 + + antrik: AIUI, it should be possible to include all network drivers + in just one binary? + that'd permit to use it in d-i + and completely replace the mach drivers + we just need to make sure to include at least what the mach drivers + cover + (all DDE network drivers, I mean) + of course that doesn't hinder from people to carefully separate + drivers in several binaries if they wish + antrik: it does link at least, I'll give a try later + yes it works! + that looks like a plan + throw all network drivers in a /hurd/dde_net + settrans it on /dev/dde_net, and settrans devnode on /dev/eth[0-9] + I'm also uploading a version of hurd which includes headers & + libraries, so you just need a make in dde_{e100,e1000,etc,net} + (uploading it with the dde driver itself :) ) + btw, a nice thing is that we don't really care that all drivers are + stuffed into a single binary, since it's a normal process only the useful + pages are mapped and actually take memory :) + is that really a nice thing though? compared to other systems I + mean + I know on linux it only loads the modules I need, for example. It's + definitely a step up for hurd though :D + that's actually precisely what I mean + you don't need to load only the modules you need + you just load them all + and paging eliminates automatically what's not useful + even parts of the driver that your device will not need + ooh + awesome + (actually, it's not even loaded, but the pci tables of the drivers + are loaded, then paged out) + + +# IRC, freenode, #hurd, 2012-02-24 + + antrik_: about the #include , I see the issue, it's + about jiffies + it wouldn't be a very good thing to have a jiffies variable which + we'd have to update 100times per second + so ZhengDa preferred to make jiffies a macro which calls a function + which reads the mapped time + +[[Mapped-time_interface|microkernel/mach/gnumach/interface/device/time]]. + + however, that break any use of the work "jiffies", e.g. structure + members & such + actually it's not only after headers that the #include has to be + done, but after any code what uses the word "jiffies" for something else + than the variable + pb is: it has to be done *before* any code that uses the word + "jiffies" for the variable, e.g. inline functions in headers + in l4dde, there's already the jiffies variable so it's not a + problem + + +# IRC, OFTC, #debian-hurd, 2012-02-27 + + I plan to do some light performance testing w.r.t. DDE + Ethernet. That is DDE vs. Mach, etc. + that'd be good, indeed + I'm getting 4MiB/s with dde + I don't remember with mach + Yes. It just shouldn't regress too much. + Aha, OK. + + +## IRC, freenode, #hurd, 2012-02-27 + + tschwinge: nttcp tells me ~80Mbps for mach-rtl8139, ~72Mbps for + dde-rtl8139, ~72Mbps for dde-e1000 + civodul: ↑ btw + youpi: so the dde network device is not much slower than the + kernel-one? + youpi: yes, looks good + rather almost the same speed + apparently + that’s quite a deal. + what speed should it have as maximum? + (means: does the mach version get out all that’s possible?) + differently put: What speed would GNU/Linux get? + I'm checking that right now + cool! + we need those numbers for the moth after the next + Mmm, I'm not sure you really want the linux number :) + 1.6Gbps :) + oh… + let me check with udp rather than tcp + so the Hurd is a “tiny bit” worse at using the network… + it might simply be an issue with tcp tuning in pfinet + hm, yes + tcp is not that cheap + and has some pretty advanced stuff for getting to high speeds + well, I'm not thinking about being cheap + but using more recent tuning + that does not believe only 1Mbps network cards exist :) + like adaptive windows and such? + :) + yes + * ArneBab remembers that TCP was invented when the connections went over + phone lines - by audio :) + yep + what’s the system load while doing the test? + yes, udp seems not so bad + ah, cool! + it's very variable (300-3000Mbps), but like on linux + that pushing it into user space has so low cost is pretty nice. + * ArneBab thinks that that’s a point where Hurd pays off + that's actually what AST said to fosdem + he doesn't care about putting an RPC for each and every port i/o + because hardware is slow anyway + jupp + but it is important to see that in real life + + +# IRC, freenode, #hurd, 2012-04-01 + + antrik: I wonder whether you could actually not route the IRQs to a + non-zero ring, AIUI you can in the x86 IDT table + youpi: you mean having a userspace server for each (non-timer) + interrupt? + youpi: how would a userspace IRQ handler interact with the + scheduler? + antrik: it doesn't necessarily have to + provided that it's trusted + youpi: how would you do CPU time accounting if there is no + interaction with the scheduler?... + antrik: you don't necessarily want to care about it + youpi: well, that would mean that all drivers handling interrupts + would have to be trusted to not use more than a very small part of CPU + time... + yes + which is usually needed for interrupt handlers anyway + youpi: nah, the bottom handler only has to do very basic stuff; + afterwards, we can pass off to "normal" driver processes, scheduled just + like other processes... but that requires some interaction between the + IRQ handler and the scheduler I think + the IRQ handler can wake up a thread, yes + no need for anything special there + so the userspace IRQ server would just decide what process to wake + up, and then call the scheduler to do a normal task switch? I guess + that's possible; but I'm not sure it would buy much... + it would permit userland to quickly react to the IRQ + such as acknowledge it to the hardware etc. + yeah, but my point is that I don't see much benefit in having this + part of the code isolated in a userspace process... it has to be trusted + anyways, and it's pretty trivial too + I never said it was a good idea + + +# IRC, freenode, #hurd, 2012-04-06 + + oh i forgot about my work on pcap + is devnode (or devopen or whatever) in the upstream repository now + ? + can't say for sure, but I'd be surprised... don't remember seeing + any movement in that regard :-( + wasn't it needed for dde ? + hm... good point + + +## IRC, freenode, #hurd, 2013-09-20 + + i should take some time to integrate my pcap changes into the + libpcap debian package at least + braunr: if upstream is active, i'd say to go there directly + the problem with that approach is that netdde is still not part of + our upstream code + don't understand the relation + i don't want to send the pcap guys code for an interface that is + still not considered upstream ... + + +# IRC, freenode, #hurd, 2012-08-14 + + it's amazing how much code just gets reimplemented needlessly ... + libddekit has its own mutex, condition, semaphore etc.. objects + with the *exact* same comment about the dequeueing-on-timeout + problem found in libpthread + *sigh* + + +## IRC, freenode, #hurd, 2012-08-18 + + hum, leaks and potential deadlocks in libddekit/thread.c :/ + + +## IRC, freenode, #hurd, 2012-08-18 + + nice, dde relies on a race to start .. + + +## IRC, freenode, #hurd, 2012-08-21 + +In context of [[libpthread]]. + + hm, i thought my pthreads patches introduced a deadlock, but + actually this one is present in the current upstream/debian code :/ + (the deadlock occurs when receiving data fast with sftp) + either in netdde or pfinet + + +## IRC, freenode, #hurd, 2013-02-28 + + (which needs the same kinds of fixes that libpthread got) + actually i'm not sure why he didn't simply reuse the pthread + functions :/ + which kind of fixes? + cancellation? + timeouts + cancellation too but that's less an issue + I'm not sure it really needs timeout work + on what RPC? + pfinet is just using the mach interface + i don't know but it clearly copies some of the previous pthread + code from pthread_cond_timedwait + see libddekit/thread.c:_sem_timedwait_internal + I recognize the comment indeed :) + I guess he thought he might need some particular semantic that + libpthread may not provide + also, now that i think about it, he couldn't have used libpthread, + could he ? + and there was no condition_timedwait in cthreads + there is a deadlock in netdde + it occurs sometimes, at high network speeds + (well high, 4 MiB/s or more) + + +## IRC, freenode, #hurd, 2013-11-20 + + for example, netdde needs more reviewing and polishing + it is known to deadlock sometimes + what deadlocks ? + i'm not sure + ah, netdde + right + yes + I'm seeing that to on one of my vms + nasty one + i know something is wrong with the condition_wait_timeout function + for example + breaks sysvinit shutdown + because it was taken without modification from libpthread + it might be that, or something else + well, dhclient hangs releasing the lease + that's still on my todo list + so I'm pretty sure it's related + hm + maybe + :/ + + +## IRC, freenode, #hurd, 2014-02-11 + + teythoon: looks like a netdde/pfinet freeze/deadlock + yes a netdde deadlock + i really have to fix that too one day :( + hehe :) + the netdde locking privimites are copies of the "old" pthread + ones, instead of reusing pthread + primitives* + + +## IRC, freenode, #hurd, 2014-03-08 + + what to do if network freezes? + gg0: depends on what caused the freeze + gg0: you could try to kill the netdde process + it's just apt-get'ing, download phase + yess kill netdde + there are known deadlocks in netdde + + +# IRC, freenode, #hurd, 2012-08-18 + + hm looks like if netdde crashes, the kernel doesn't handle it + cleanly, and we can't attach another netdde instance + +[[!message-id "877gu8klq3.fsf@kepler.schwinge.homeip.net"]] + + +# DDE for Filesystems + +## IRC, freenode, #hurd, 2012-10-07 + + * pinotree wonders whether the dde layer could aldo theorically support + also file systems + pinotree: yeah, I also brought up the idea of creating a DDE + extension or DDE-like wrapper for Linux filesystems a while back... don't + know enough about it though to decide whether it's doable + OTOH, I'm not sure it would be worthwhile. we still should + probably have a native (not GPLv2-only) implementation for the main FS at + least; so the wrapper would only be for accessing external + partitions/media... + + +## IRC, freenode, #hurd, 2013-12-03 + + how about porting linux block device layer via dde as mcsim wanted to + do? then all linux filesystems could be brought in, right? + gg0: that should be done, but we need to correctly deal with + multiple pci devices in userspace and arbitration + wouldn't adding support to passive translator into Linux + filesystems be quite some work ? IIRC ext2fs needs a special "owner = + hurd" mode to handle them + + +# [[virtio]] -- cgit v1.2.3