summaryrefslogtreecommitdiff
path: root/service_solahart_jakarta_selatan__082122541663/hurd_101.mdwn
blob: 89223d926c81f0fb194b639d7250386d83f30739 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
[[!meta copyright="Copyright © 2011, 2013, 2014 Free Software Foundation,
Inc."]]

[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
id="license" text="Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License, Version 1.2 or
any later version published by the Free Software Foundation; with no Invariant
Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
is included in the section entitled [[GNU Free Documentation
License|/fdl]]."]]"""]]

(See Wikipedia page for the meaning of [[!wikipedia "101_(term)"]].)

Not the first time that something like this is proposed...


# IRC, freenode, #hurd, 2011-07-25

    [failed GNU/Hurd project]
    < antrik> gnu_srs1: I wouldn't say he was on track. just one of the many
      many people who insist on picking a hard task; realizing that indeed it's
      hard; and going into hiding
    < antrik> we see that happen every couple of months
    < cluck> maybe we need a "hurd 101" 
    < cluck> getting a teacher and setting up a regularly held "class" for hurd
      noobs
    < Tekk_> cluck: what would that include?
    < cluck> explaining core concepts, giving out "homework" (small tasks), etc

[[service_solahart_jakarta_selatan__082122541663/Anatomy_of_a_hurd_system]].

    < cluck> that way "the big guys" could focus on the hard stuff and have an
      army of code monkeys at their disposal to write speced stuff
    < cluck> (then again this idea would heavily depend on available "teachers"
      and "students", which, going by gsoc numbers, may not be all that
      helpful)
    < Tekk_> cluck: gsoc isn't an accurate indicator
    < Tekk_> cluck: I'm not allowed to participate in gsoc but I'd join :P
    < antrik> cluck: we don't need code monkeys... we need hackers
    < Tekk_`> antrik: code monkeys involve into hackers
    < Tekk_`> under the right conditions
    < cluck> antrik: jokes aside some sort of triage system/training ground for
      newcomers could be helpful


# IRC, freenode, #hurd, 2013-01-20

    <zacts> so once I have written my first translators, and really understand
      that, what kinds of projects would you recommend to an operating
      systems/hurd newbie.
    <zacts> I am reading the minix book now as I have it, but I'm waiting on
      getting the modern operating systems book by the same author.
    <zacts> I was initially going to start working on minix, but their focus
      seems to be on embedded, and I want to work on a system that is more
      general purpose, and I like the philosophy of freedom surrounding the
      hurd.
    <zacts> I like how the hurd design allows more freedom for users of the
      operating system, but I would also like to incorporate ideas from minix
      on the hurd. mainly, rebootless updates of servers/translators.
    <neal> then you should study how translators work
    <neal> how ipc works
    <neal> and understand exactly what state is stored where
    <zacts> ok


# IRC, freenode, #hurd, 2013-10-12

    <ahungry> Hi all, can anyone expand on
      https://www.gnu.org/software/hurd/contributing.html - if I proceed with
      the quick start and have the system running in a virtual image, how do I
      go from there to being able to start tweaking the source (and recompiling
      ) in a meaningful way?
    <ahungry> Would I modify the source, compile within the VM and then what
      would be the next step to actually test my new changes?
    <braunr> ahungry: we use debian
    <braunr> i suggest formatting your changes into patches, importing them
      into debian packages, rebuilding those packages, and installing them over
      the upstream ones
    <ahungry> what about modifications to mach itself?  or say I wanted to try
      to work on the wifi drives - I would build the translator or module or
      whatever and just add to the running instance of hurd?
    <ahungry> s/drives/drivers
    <braunr> same thing
    <braunr> although
    <braunr> during development, it's obviously a bit too expensive to rebuild
      complete packages each time
    <braunr> you can use the hurd on top of a gnumach kernel built completely
      from upstream sources
    <braunr> you need a few debian patches for the hurd itself
    <braunr> a lot of them for glibc
    <braunr> i usually create a temporary local branch with the debian patches
      i need to make my code run
    <braunr> and then create the true development branch itself from that one
    <braunr> drivers are a a dark corner of the hurd
    <braunr> i wouldn't recommend starting there
    <braunr> but if you did, yes, you'd write a server to run drivers, and
      start it
    <braunr> you'd probably write a translator (which is a special kind of
      server), yes
    <ahungry> braunr: thanks for all the info, hittin the sack now but ill have
      to set up a box and try to contribute 


# Documentation

## IRC, freenode, #hurd, 2013-11-04

    <stargater> i think the problem my hurd have not more developers or
      contubutors is the project idears and management , eg, the most problem
      is the mach kernel and documatation and the missing subsystem goals
      (driver, etc)
    <stargater> no i think you and other have a clue but this is not
      tranzparent when i read the webpage 
    <teythoon> well, fwiw I agree, the documentation is lacking
    <braunr> about what ?
    <braunr> something that doesn't exist ?
    <braunr> like smp or a generic device driver framework ?
    <teythoon> no, high level concepts, design stuff
    <braunr> what ?
    <braunr> how come ?
    <teythoon> not even the gnumach documentation is complete
    <braunr> for example ?
    <braunr> see http://www.sceen.net/~rbraun/doc/mach/
    <braunr> which is my personal collection of docs on mach/hurd
    <braunr> and it's lacking at least one paper
    <braunr> well two, since i can't find the original article about the hurd
      in pdf format
    <braunr> project ideas are clearly listed in the project ideas page
    <stargater> braunr: do you think the mach kernel decumatation a compleat?
      and you think its good documentatition about "how write a drive for mach"
      and you think a answare is found why dont work smp and why is have no
      arm, x64 support ?
    <braunr> stargater:
      http://darnassus.sceen.net/~hurd-web/community/gsoc/project_ideas/
    <braunr> the page is even named "project ideas"
    <braunr> the mach kernel is probably the most documented in the world
    <braunr> even today
    <braunr> and if there is no documentation about "how to write drivers for
      mach", that's because we don't want in kernel drivers any more
    <braunr> and the state of our driver framework is practically non existent
    <braunr> it's basically netdde
    <braunr> partial support for network drivers from linux
    <braunr> that's all
    <braunr> we need to improve that
    <braunr> someone needs to do the job
    <braunr> noone has for now
    <braunr> that's all
    <braunr> why would we document something that doesn't exist ?
    <braunr> only stupid project managers with no clue about the real world do
      that
    <braunr> (or great ones who already know everything there is to know before
      writing code, but that's rare)
    <braunr> stargater: the answer about smp, architectures etc.. is the same
    <stargater> spirit and magic are nice ;-) braunr sorry, that is only my
      meanig and i will help, so i ask and say what i think. when you say, hurd
      and mach are good and we on the right way, then its ok for me . i wonder
      why not more developer help hurd. and i can read and see the project page
      fro side a first time user/developer
    <braunr> i didn't say they're good
    <braunr> they're not, they need to be improved
    <braunr> clearly
    <stargater> ok, then sorry
    <braunr> i wondered about that too, and my conclusion is that people aren't
      interested that much in system architectures
    <braunr> and those who are considered the hurd too old to be interesting,
      and don't learn about it
    <braunr> consider*
    <braunr> stargater: why are you interested in the hurd ?
    <braunr> that's a question everyone intending to work on it should ask
    <stargater> the spirit of free software and new and other operation system,
      with focus to make good stuff with less code and working code for ever
      and everone can it used 
    <braunr> well, if the focus was really to produce good stuff, the hurd
      wouldn't be so crappy
    <braunr> it is now, but it wasn't in the past
    <stargater> a good point whas more documentation in now and in the future,
      eg, i like the small project http://wiki.osdev.org/ and i like to see
      more how understanding mach and hurd 
    <nalaginrut> I love osdev much, it taught me a lot ;-D
    <braunr> osdev is a great source for beginners
    <braunr> teythoon: what else did you find lacking ?
    <teythoon> braunr: in my opinion the learning curve of Hurd development is
      quite steep at the beginning
    <teythoon> yes, documentation exists, but it is distributed all over the
      internets
    <braunr> teythoon: hm ok
    <braunr> yes the learning curve is too hard
    <braunr> that's an entry barrier


# IRC, freenode, #hurd, 2014-02-04

[[!tag open_issue_documentation]]

    <bwright> Does the GNU Mach kernel have concepts of capabilities?
    <braunr> yes
    <braunr> see ports, port rights and port names
    <bwright> Does it follow the take grant approch
    <bwright> approach*
    <braunr> probably
    <bwright> Can for example I take an endpoint that I retype from untyped
      memory and mint it such that it only has read access and pass that to the
      cspace of another task over ipc.
    <bwright> Where that read minted cap enforces it may onnly wait on that ep.
    <braunr> ep ?
    <braunr> ah
    <bwright> Endpoint.
    <braunr> probably
    <bwright> Alright cool.
    <braunr> it's a bit too abstract for me to answer reliably
    <braunr> ports are message queues
    <braunr> port rights are capabilities to ports
    <bwright> Not sure exactly how it would be implemented but essentially you
      would have a guarded page table with 2 levels, 2^pow slots.
    <braunr> port names are integers referring to port rights
    <braunr> we don't care about the implementation of page tables
    <bwright> Each slot contains a kernel object, which in itself may be more
      page tabels that store more caps.
    <braunr> it's not l4 :p
    <braunr> mach is more of a hybrid
    <bwright> It isn't a page table for memory.
    <braunr> it manages virtual memory
    <bwright> Ah ok.
    <braunr> whatever, we don't care about the implementation
    <bwright> So if I want to say port an ethernet driver over.
    <braunr> whether memory or capabilities, mach manages them
    <bwright> Can I forward the interrupts through to my new process?
    <braunr> yes
    <braunr> it has been implemented for netdde
    <braunr> these are debian specific patches for the time being though
    <bwright> Great, and shared memory set ups are all nice and dandy.
    <braunr> yes, the mach vm takes care of that
    <bwright> Can I forward page faults?
    <bwright> Or does mach actually handle the faults?
    <bwright> (Sorry for so many questions just comparing what I know from my
      microkernel knowledge to mach and gnu mach)
    <braunr> mach handles them but translates them to requests to userspace
      pagers
    <bwright> (Still have a mach paper to read)
    <bwright> Alright that sounds sane.
    <bwright> Does GNU mach have benchmarks on its IPC times?
    <braunr> no but expect them to suck :)
    <bwright> Isn't it fixable though?
    <braunr> mach ipc is known to be extremely heavy in comparison with modern
      l4-like kernels
    <braunr> not easily
    <bwright> Yeah so I know that IPC is an issue but never dug into why it is
      bad on Mach.
    <bwright> So what design decision really screwed up IPC speed?
    <braunr> for one because they're completely async, and also because they
      were designed for network clusters, meaning data is typed inside messages
    <bwright> Oh weird
    <bwright> So how is type marshalled in the message?
    <braunr> in its own field
    <braunr> messages have their own header
    <braunr> and each data field inside has its own header
    <bwright> Oh ok, so I can see this being heavy.
    <bwright> So the big advantage is for RPC
    <bwright> It would make things nice in that case.
    <bwright> Is it possible to send an IPC without the guff though?
    <bwright> Or would this break the model mach is trying to achieve?
    <bwright> I am assuming Mach wanted something where you couldn't tell if a
      process was local or not.
    <bwright> So I am assuming then that IPC is costly for system calls from a
      user process.
    <bwright> You have some sort of blocking wait on the call to the service
      that dispatches the syscall.
    <bwright> I am assuming the current variants of GNU/Hurd run on glibc.
    <bwright> It would be interesting to possibly replace that with UlibC or do
      a full port of the FlexSC exceptionless system calls.
    <bwright> Could get rid of some of the bottlenecks in hurd assuming it is
      very IPC heavy.
    <bwright> And that won't break the async model.
    <bwright> Actually should be simpler if it is already designed for that.
    <bwright> But would break the "distributed" vibe unless you had the faults
      to those shared pages hit a page faulter that sent them over the network
      on write.
    <bwright> </end probably stupid ideas>
    <kilobug> bwright: a lot of POSIX compatibility is handled by the glibc,
      "porting" another libc to the Hurd will be a titanic task
    <bwright> In theory exceptionless system calls work fine on glibc, it is
      just harder to get them working.
    <bwright> has not been done or was not explored in the paper.
    <bwright> Something about it having a few too many annoying assumptions.
    <bwright> Would be interesting to run some benchmarks on hurd and figure
      out where the bottlenecks really are.
    <bwright> At least for an exercise in writing good benchmarks :P
    <bwright> I have a paper on the design of hurd I should read actually.
    <bwright> After I get through this l4 ref man.
    <braunr> the main bottleneck is scalability
    <braunr> there are a lot of global locks
    <braunr> and servers are prone to spawning lots of threads
    <braunr> because, despite the fact mach provides async ipc, the hurd mostly
      uses sync ipc
    <braunr> so the way to handle async notifications is to receive messages
      and spawn threads as needed
    <bwright> Lets take a senario
    <braunr> beyond that, core algorithms such as scanning pages in pagers, are
      suboptimal
    <bwright> I want to get a file and send it across the network.
    <bwright> How many copies of the data occur?
    <braunr> define send
    <braunr> ouch :)
    <braunr> disk drivers are currently in the kernel
    <bwright> I read a block from disk, I pass this to my file system it passes
      it to the app and it sends to the lwip or whatever interface then out the
      ethernet card.
    <braunr> and "block device drivers" in userspace (storeio) are able to
      redirect file system servers directly to those in kernel drivers
    <braunr> so
    <braunr> kernel -> fs -> client -> pfinet -> netdde (user space network
      drivers on debian hurd)
    <bwright> Alright. Hopefully each arrow is not a copy :p
    <braunr> it is
    <bwright> My currently multiserver does this same thing with zero copy.
    <braunr> because buffers are usually small
    <braunr> yes but zero copy requires some care
    <bwright> Which is possible.
    <braunr> and usually, posix clients don't care about that
    <bwright> Yes it requires a lot of care.
    <bwright> POSIX ruins this
    <bwright> Absolutely.
    <braunr> they assume read/write copy data, or that the kernel is directly
      able to access data
    <bwright> But there are some things you can take care with
    <bwright> And not break posix and still have this work.
    <braunr> pfinet handles ethernet packets one at a time, and 1500 isn't
      worth zero copying
    <bwright> This depends though right?
    <braunr> i'm not saying it's not possible
    <braunr> i'm saying most often, there are copies
    <bwright> So if I have high throughput I can load up lots of packets and
      the data section can then be sectioned with scatter gather 
    <braunr> again, the current interface doesn't provide that
    <bwright> Alright yeah that is what I expected which is fine.
    <bwright> It will be POSIX compliant which is the main goal.
    <braunr> not really scatter gather here but rather segment offloading for
      example
    <braunr> ah you're working on something like that too :)
    <bwright> Yeah I am an intern :)
    <bwright> Have it mostly working, just lots of pain.
    <bwright> Have you read the netmap paper?
    <bwright> Really interesting.
    <braunr> not sure i have
    <braunr> unless it has another full name
    <bwright> 14.86 million packets per second out of the ethernet card :p
    <bwright> SMOKES everything else.
    <bwright> Implemented in Linux and FreeBSD now.
    <bwright> Packets are UDP 1 byte MTU I think
    <bwright> 1 byte data *
    <bwright> To be correct :p
    <braunr> right, i see
    <bwright> Break posix again
    <bwright> "More Extend"
    <braunr> i've actually worked on a proprietary implementation of such a
      thing where i'm currently working
    <bwright> Bloody useful for high frequency trading etc.
    <bwright> Final year as an undergraduate this year doing my thesis which
      should be fun, going to be something OS hopefully.
    <bwright> Very fun field lots of weird and crazy problems.