IRC, freenode, #hurd, 2011-07-25

[failed GNU/Hurd project]
< antrik> gnu_srs1: I wouldn't say he was on track. just one of the many
  many people who insist on picking a hard task; realizing that indeed it's
  hard; and going into hiding
< antrik> we see that happen every couple of months
< cluck> maybe we need a "hurd 101" 
< cluck> getting a teacher and setting up a regularly held "class" for hurd
  noobs
< Tekk_> cluck: what would that include?
< cluck> explaining core concepts, giving out "homework" (small tasks), etc

Anatomy of a Hurd system.

< cluck> that way "the big guys" could focus on the hard stuff and have an
  army of code monkeys at their disposal to write speced stuff
< cluck> (then again this idea would heavily depend on available "teachers"
  and "students", which, going by gsoc numbers, may not be all that
  helpful)
< Tekk_> cluck: gsoc isn't an accurate indicator
< Tekk_> cluck: I'm not allowed to participate in gsoc but I'd join :P
< antrik> cluck: we don't need code monkeys... we need hackers
< Tekk_`> antrik: code monkeys involve into hackers
< Tekk_`> under the right conditions
< cluck> antrik: jokes aside some sort of triage system/training ground for
  newcomers could be helpful

IRC, freenode, #hurd, 2013-01-20

<zacts> so once I have written my first translators, and really understand
  that, what kinds of projects would you recommend to an operating
  systems/hurd newbie.
<zacts> I am reading the minix book now as I have it, but I'm waiting on
  getting the modern operating systems book by the same author.
<zacts> I was initially going to start working on minix, but their focus
  seems to be on embedded, and I want to work on a system that is more
  general purpose, and I like the philosophy of freedom surrounding the
  hurd.
<zacts> I like how the hurd design allows more freedom for users of the
  operating system, but I would also like to incorporate ideas from minix
  on the hurd. mainly, rebootless updates of servers/translators.
<neal> then you should study how translators work
<neal> how ipc works
<neal> and understand exactly what state is stored where
<zacts> ok

IRC, freenode, #hurd, 2013-10-12

<ahungry> Hi all, can anyone expand on
  https://www.gnu.org/software/hurd/contributing.html - if I proceed with
  the quick start and have the system running in a virtual image, how do I
  go from there to being able to start tweaking the source (and recompiling
  ) in a meaningful way?
<ahungry> Would I modify the source, compile within the VM and then what
  would be the next step to actually test my new changes?
<braunr> ahungry: we use debian
<braunr> i suggest formatting your changes into patches, importing them
  into debian packages, rebuilding those packages, and installing them over
  the upstream ones
<ahungry> what about modifications to mach itself?  or say I wanted to try
  to work on the wifi drives - I would build the translator or module or
  whatever and just add to the running instance of hurd?
<ahungry> s/drives/drivers
<braunr> same thing
<braunr> although
<braunr> during development, it's obviously a bit too expensive to rebuild
  complete packages each time
<braunr> you can use the hurd on top of a gnumach kernel built completely
  from upstream sources
<braunr> you need a few debian patches for the hurd itself
<braunr> a lot of them for glibc
<braunr> i usually create a temporary local branch with the debian patches
  i need to make my code run
<braunr> and then create the true development branch itself from that one
<braunr> drivers are a a dark corner of the hurd
<braunr> i wouldn't recommend starting there
<braunr> but if you did, yes, you'd write a server to run drivers, and
  start it
<braunr> you'd probably write a translator (which is a special kind of
  server), yes
<ahungry> braunr: thanks for all the info, hittin the sack now but ill have
  to set up a box and try to contribute

Documentation

IRC, freenode, #hurd, 2013-11-04

<stargater> i think the problem my hurd have not more developers or
  contubutors is the project idears and management , eg, the most problem
  is the mach kernel and documatation and the missing subsystem goals
  (driver, etc)
<stargater> no i think you and other have a clue but this is not
  tranzparent when i read the webpage 
<teythoon> well, fwiw I agree, the documentation is lacking
<braunr> about what ?
<braunr> something that doesn't exist ?
<braunr> like smp or a generic device driver framework ?
<teythoon> no, high level concepts, design stuff
<braunr> what ?
<braunr> how come ?
<teythoon> not even the gnumach documentation is complete
<braunr> for example ?
<braunr> see http://www.sceen.net/~rbraun/doc/mach/
<braunr> which is my personal collection of docs on mach/hurd
<braunr> and it's lacking at least one paper
<braunr> well two, since i can't find the original article about the hurd
  in pdf format
<braunr> project ideas are clearly listed in the project ideas page
<stargater> braunr: do you think the mach kernel decumatation a compleat?
  and you think its good documentatition about "how write a drive for mach"
  and you think a answare is found why dont work smp and why is have no
  arm, x64 support ?
<braunr> stargater:
  http://darnassus.sceen.net/~hurd-web/community/gsoc/project_ideas/
<braunr> the page is even named "project ideas"
<braunr> the mach kernel is probably the most documented in the world
<braunr> even today
<braunr> and if there is no documentation about "how to write drivers for
  mach", that's because we don't want in kernel drivers any more
<braunr> and the state of our driver framework is practically non existent
<braunr> it's basically netdde
<braunr> partial support for network drivers from linux
<braunr> that's all
<braunr> we need to improve that
<braunr> someone needs to do the job
<braunr> noone has for now
<braunr> that's all
<braunr> why would we document something that doesn't exist ?
<braunr> only stupid project managers with no clue about the real world do
  that
<braunr> (or great ones who already know everything there is to know before
  writing code, but that's rare)
<braunr> stargater: the answer about smp, architectures etc.. is the same
<stargater> spirit and magic are nice ;-) braunr sorry, that is only my
  meanig and i will help, so i ask and say what i think. when you say, hurd
  and mach are good and we on the right way, then its ok for me . i wonder
  why not more developer help hurd. and i can read and see the project page
  fro side a first time user/developer
<braunr> i didn't say they're good
<braunr> they're not, they need to be improved
<braunr> clearly
<stargater> ok, then sorry
<braunr> i wondered about that too, and my conclusion is that people aren't
  interested that much in system architectures
<braunr> and those who are considered the hurd too old to be interesting,
  and don't learn about it
<braunr> consider*
<braunr> stargater: why are you interested in the hurd ?
<braunr> that's a question everyone intending to work on it should ask
<stargater> the spirit of free software and new and other operation system,
  with focus to make good stuff with less code and working code for ever
  and everone can it used 
<braunr> well, if the focus was really to produce good stuff, the hurd
  wouldn't be so crappy
<braunr> it is now, but it wasn't in the past
<stargater> a good point whas more documentation in now and in the future,
  eg, i like the small project http://wiki.osdev.org/ and i like to see
  more how understanding mach and hurd 
<nalaginrut> I love osdev much, it taught me a lot ;-D
<braunr> osdev is a great source for beginners
<braunr> teythoon: what else did you find lacking ?
<teythoon> braunr: in my opinion the learning curve of Hurd development is
  quite steep at the beginning
<teythoon> yes, documentation exists, but it is distributed all over the
  internets
<braunr> teythoon: hm ok
<braunr> yes the learning curve is too hard
<braunr> that's an entry barrier

IRC, freenode, #hurd, 2014-02-04

<bwright> Does the GNU Mach kernel have concepts of capabilities?
<braunr> yes
<braunr> see ports, port rights and port names
<bwright> Does it follow the take grant approch
<bwright> approach*
<braunr> probably
<bwright> Can for example I take an endpoint that I retype from untyped
  memory and mint it such that it only has read access and pass that to the
  cspace of another task over ipc.
<bwright> Where that read minted cap enforces it may onnly wait on that ep.
<braunr> ep ?
<braunr> ah
<bwright> Endpoint.
<braunr> probably
<bwright> Alright cool.
<braunr> it's a bit too abstract for me to answer reliably
<braunr> ports are message queues
<braunr> port rights are capabilities to ports
<bwright> Not sure exactly how it would be implemented but essentially you
  would have a guarded page table with 2 levels, 2^pow slots.
<braunr> port names are integers referring to port rights
<braunr> we don't care about the implementation of page tables
<bwright> Each slot contains a kernel object, which in itself may be more
  page tabels that store more caps.
<braunr> it's not l4 :p
<braunr> mach is more of a hybrid
<bwright> It isn't a page table for memory.
<braunr> it manages virtual memory
<bwright> Ah ok.
<braunr> whatever, we don't care about the implementation
<bwright> So if I want to say port an ethernet driver over.
<braunr> whether memory or capabilities, mach manages them
<bwright> Can I forward the interrupts through to my new process?
<braunr> yes
<braunr> it has been implemented for netdde
<braunr> these are debian specific patches for the time being though
<bwright> Great, and shared memory set ups are all nice and dandy.
<braunr> yes, the mach vm takes care of that
<bwright> Can I forward page faults?
<bwright> Or does mach actually handle the faults?
<bwright> (Sorry for so many questions just comparing what I know from my
  microkernel knowledge to mach and gnu mach)
<braunr> mach handles them but translates them to requests to userspace
  pagers
<bwright> (Still have a mach paper to read)
<bwright> Alright that sounds sane.
<bwright> Does GNU mach have benchmarks on its IPC times?
<braunr> no but expect them to suck :)
<bwright> Isn't it fixable though?
<braunr> mach ipc is known to be extremely heavy in comparison with modern
  l4-like kernels
<braunr> not easily
<bwright> Yeah so I know that IPC is an issue but never dug into why it is
  bad on Mach.
<bwright> So what design decision really screwed up IPC speed?
<braunr> for one because they're completely async, and also because they
  were designed for network clusters, meaning data is typed inside messages
<bwright> Oh weird
<bwright> So how is type marshalled in the message?
<braunr> in its own field
<braunr> messages have their own header
<braunr> and each data field inside has its own header
<bwright> Oh ok, so I can see this being heavy.
<bwright> So the big advantage is for RPC
<bwright> It would make things nice in that case.
<bwright> Is it possible to send an IPC without the guff though?
<bwright> Or would this break the model mach is trying to achieve?
<bwright> I am assuming Mach wanted something where you couldn't tell if a
  process was local or not.
<bwright> So I am assuming then that IPC is costly for system calls from a
  user process.
<bwright> You have some sort of blocking wait on the call to the service
  that dispatches the syscall.
<bwright> I am assuming the current variants of GNU/Hurd run on glibc.
<bwright> It would be interesting to possibly replace that with UlibC or do
  a full port of the FlexSC exceptionless system calls.
<bwright> Could get rid of some of the bottlenecks in hurd assuming it is
  very IPC heavy.
<bwright> And that won't break the async model.
<bwright> Actually should be simpler if it is already designed for that.
<bwright> But would break the "distributed" vibe unless you had the faults
  to those shared pages hit a page faulter that sent them over the network
  on write.
<bwright> </end probably stupid ideas>
<kilobug> bwright: a lot of POSIX compatibility is handled by the glibc,
  "porting" another libc to the Hurd will be a titanic task
<bwright> In theory exceptionless system calls work fine on glibc, it is
  just harder to get them working.
<bwright> has not been done or was not explored in the paper.
<bwright> Something about it having a few too many annoying assumptions.
<bwright> Would be interesting to run some benchmarks on hurd and figure
  out where the bottlenecks really are.
<bwright> At least for an exercise in writing good benchmarks :P
<bwright> I have a paper on the design of hurd I should read actually.
<bwright> After I get through this l4 ref man.
<braunr> the main bottleneck is scalability
<braunr> there are a lot of global locks
<braunr> and servers are prone to spawning lots of threads
<braunr> because, despite the fact mach provides async ipc, the hurd mostly
  uses sync ipc
<braunr> so the way to handle async notifications is to receive messages
  and spawn threads as needed
<bwright> Lets take a senario
<braunr> beyond that, core algorithms such as scanning pages in pagers, are
  suboptimal
<bwright> I want to get a file and send it across the network.
<bwright> How many copies of the data occur?
<braunr> define send
<braunr> ouch :)
<braunr> disk drivers are currently in the kernel
<bwright> I read a block from disk, I pass this to my file system it passes
  it to the app and it sends to the lwip or whatever interface then out the
  ethernet card.
<braunr> and "block device drivers" in userspace (storeio) are able to
  redirect file system servers directly to those in kernel drivers
<braunr> so
<braunr> kernel -> fs -> client -> pfinet -> netdde (user space network
  drivers on debian hurd)
<bwright> Alright. Hopefully each arrow is not a copy :p
<braunr> it is
<bwright> My currently multiserver does this same thing with zero copy.
<braunr> because buffers are usually small
<braunr> yes but zero copy requires some care
<bwright> Which is possible.
<braunr> and usually, posix clients don't care about that
<bwright> Yes it requires a lot of care.
<bwright> POSIX ruins this
<bwright> Absolutely.
<braunr> they assume read/write copy data, or that the kernel is directly
  able to access data
<bwright> But there are some things you can take care with
<bwright> And not break posix and still have this work.
<braunr> pfinet handles ethernet packets one at a time, and 1500 isn't
  worth zero copying
<bwright> This depends though right?
<braunr> i'm not saying it's not possible
<braunr> i'm saying most often, there are copies
<bwright> So if I have high throughput I can load up lots of packets and
  the data section can then be sectioned with scatter gather 
<braunr> again, the current interface doesn't provide that
<bwright> Alright yeah that is what I expected which is fine.
<bwright> It will be POSIX compliant which is the main goal.
<braunr> not really scatter gather here but rather segment offloading for
  example
<braunr> ah you're working on something like that too :)
<bwright> Yeah I am an intern :)
<bwright> Have it mostly working, just lots of pain.
<bwright> Have you read the netmap paper?
<bwright> Really interesting.
<braunr> not sure i have
<braunr> unless it has another full name
<bwright> 14.86 million packets per second out of the ethernet card :p
<bwright> SMOKES everything else.
<bwright> Implemented in Linux and FreeBSD now.
<bwright> Packets are UDP 1 byte MTU I think
<bwright> 1 byte data *
<bwright> To be correct :p
<braunr> right, i see
<bwright> Break posix again
<bwright> "More Extend"
<braunr> i've actually worked on a proprietary implementation of such a
  thing where i'm currently working
<bwright> Bloody useful for high frequency trading etc.
<bwright> Final year as an undergraduate this year doing my thesis which
  should be fun, going to be something OS hopefully.
<bwright> Very fun field lots of weird and crazy problems.