path: root/open_issues/profiling.mdwn

[[!meta copyright="Copyright © 2010, 2011, 2013, 2014, 2016 Free Software
Foundation, Inc."]]

[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
id="license" text="Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License, Version 1.2 or
any later version published by the Free Software Foundation; with no Invariant
Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
is included in the section entitled [[GNU Free Documentation
License|/fdl]]."]]"""]]

[[!meta title="Profiling, Tracing"]]

*Profiling* ([[!wikipedia Profiling_(computer_programming) desc="Wikipedia
article"]]) is a tool for tracing where CPU time is spent.  This is usually
done for [[performance analysis|performance]] reasons.

  * [[hurd/debugging/rpctrace]]

  * [[gprof]]

  * [[profiling GNU Mach|microkernel/mach/gnumach/profiling]]

  * [[glibc]]'s sotruss

  * [[ltrace]]

  * [[latrace]]

  * [[community/gsoc/project_ideas/dtrace]]

    Have a look at this, integrate it into the main trees.

  * [[LTTng]]

  * [[SystemTap]]

  * ... or some other Linux thing.


# IRC, freenode, #hurd, 2013-06-17

    <congzhang> is that possible we develop rpc msg analyse tool? make it clear
      view system at different level?
    <congzhang> hurd was dynamic system, how can we just read log line by line
    <kilobug> congzhang: well, you can use rpctrace and then analyze the logs,
      but rpctrace is quite intrusive and will slow down things (like strace or
      similar)
    <kilobug> congzhang: I don't know if a low-overhead solution could be made
      or not
    <congzhang> that's the problem
    <congzhang> when real system run, the msg cross different server, and then
      the debug action should not intrusive the process itself
    <congzhang> we observe the system and analyse os
    <congzhang> when rms choose microkernel, it's expect to accelerate the
      progress, but not
    <congzhang> microkernel make debug a litter hard
    <kilobug> well, it's not limited to microkernels, debugging/tracing is
      intrusive and slow things down, it's an universal law of compsci
    <kilobug> no, it makes debugging easier
    <congzhang> I don't think so
    <kilobug> you can gdb the various services (like ext2fs or pfinet) more
      easily
    <kilobug> and rpctrace isn't any worse than strace
    <congzhang> how easy when debug lpc
    <kilobug> lpc ?
    <congzhang> because cross context
    <congzhang> classic function call
    <congzhang> when find the bug source, I don't care performance, I wan't to
      know it's right or wrong by design, If it work as I expect 
    <congzhang> I optimize it latter
    <congzhang> I have an idea, but don't know weather it's usefull or not
    <braunr> rpctrace is a lot less instrusive than ptrace based tools
    <braunr> congzhang: debugging is not made hard by the design choice, but by
      implementation details
    <braunr> as a simple counter example, someone often cited usb development
      on l3 being made a lot easier than on a monolithic kernel
    <congzhang> Collect the trace information first, and then layout the msg by
      graph, when something wrong, I focus the trouble rpc, and found what
      happen around
    <braunr> "by graph" ?
    <congzhang> yes
    <congzhang> braunr: directed graph or something similar
    <braunr> and not caring about performance when debugging is actually stupid
    <braunr> i've seen it on many occasions, people not being able to use
      debugging tools because they were far too inefficient and slow
    <braunr> why a graph ?
    <braunr> what you want is the complete trace, taking into account cross
      address space boundaries
    <congzhang> yes
    <braunr> well it's linear
    <braunr> switching server
    <congzhang> by independent process view it's linear
    <congzhang> it's linear on cpu's view too
    <congzhang> yes, I need complete trace, and dynamic control at microkernel
      level
    <congzhang> os, if server crash, and then I know what's other doing, from
      the graph
    <congzhang> graph needn't to be one, if the are not connect together, time
      sort them
    <congzhang> when hurd was complete ok, some tools may be help too
    <braunr> i don't get what you want on that graph
    <congzhang> sorry, I need a context
    <congzhang> like uml sequence diagram, I need what happen one by one
    <congzhang> from server's view and from the function's view
    <braunr> that's still linear
    <braunr> so please stop using the word graph
    <braunr> you want a trace
    <braunr> a simple call trace
    <congzhang> yes, and a tool
    <braunr> with some work gdb could do it
    <congzhang> you mean under  some microkernel infrastructure help 
    <congzhang> ?
    <braunr> if needed
    <congzhang> braunr: will that be easy?
    <braunr> not too hard
    <braunr> i've had this idea for a long time actually
    <braunr> another reason i insist on migrating threads (or rather, binding
      server and client threads)
    <congzhang> braunr: that's  great
    <braunr> the current problem we have when using gdb is that we don't know
      which server thread is handling the request of which client
    <braunr> we can guess it
    <braunr> but it's not always obvious
    <congzhang> I read the talk, know some of your idea
    <congzhang> make things happen like classic kernel, just from function
      ,sure:)
    <braunr> that's it
    <congzhang> I think you and other do a lot of work to improve the mach and
      hurd, buT we lack the design document and the diagram, one diagram was
      great than one thousand words
    <braunr> diagrams are made after the prototypes that prove they're doable
    <braunr> i'm not a researcher
    <braunr> and we have little time
    <braunr> the prototype is the true spec
    <congzhang> that's why i wan't cllector the trace info and show, you can
      know what happen and how happen, maybe just suitable for newbie, hope
      more young hack like it
    <braunr> once it's done, everything else is just sugar candy around it


# IRC, freenode, #hurd, 2014-01-05

    <teythoon> braunr: do you speak ocaml ?
    <teythoon> i had this awesome idea for a universal profiling framework for
      c
    <teythoon> universal as in not os dependent, so it can be easily used on
      hurd or in gnu mach
    <teythoon> it does a source transformation, instrumenting what you are
      interested in
    <teythoon> for this transformation, coccinelle is used
    <teythoon> i have a prototype to measure how often a field in a struct is
      accessed
    <teythoon> unfortunately, coccinelle hangs while processing kern/slab.c :/
    <youpi> teythoon:  I do speak ocaml
    <teythoon> awesome :)
    <teythoon> unfortunately, i do not :/
    <teythoon> i should probably get in touch with the coccinelle devs, most
      likely the problem is that coccinelle runs in circles somewhere
    <youpi> it's not so complex actually
    <youpi> possibly,  yes
    <teythoon> do you know coccinelle ?
    <youpi> the only really peculiar thing in ocaml is lambda calculus
    <youpi> +c
    <youpi> I know a bit, although I've never really written an semantic patch
      myself
    <teythoon> i'm okay with that
    <youpi> but I can understand them
    <youpi> then ocaml should be fine for you :)
    <youpi> just ask the few bits that you don't understand :)
    <teythoon> yeah, i haven't really made an effort yet
    <youpi> writing ocaml is a bit more difficult because you need to
      understand the syntax, but for putting printfs it should be easy enough
    <youpi> if you get a backtrace with ocamldebug (it basically works like
      gdb), I can probably explain you what might be happening


## IRC, freenode, #hurd, 2014-01-06

    <teythoon> braunr: i'm not doing microoptimizations, i'm developing a
      profiler :p
    <braunr> teythoon: nice :)
    <teythoon> i thought you might like it
    <braunr> teythoon: you may want to look at
      http://pdos.csail.mit.edu/multicore/dprof/
    <braunr> from the same people who brought radixvm
    <teythoon> which data structure should i test it with next ?
    <braunr> uh, no idea :)
    <braunr> the ipc ones i suppose
    <teythoon> yeah, or the task related ones
    <braunr> but be careful, there many "inline" versions of many ipc functions
      in the fast paths
    <braunr> and when they say inline, they really mean they copied it
    <braunr> +are
    <teythoon> but i have a microbenchmark for ipc performance
    <braunr> you sure have been busy ;p
    <braunr> it's funny you're working on a profiler at the same time a
      collegue of mine said he was interested in writing one in x15 :)
    <teythoon> i don't think inlining is a problem for my tool
    <teythoon> well, you can use my tool for x15
    <braunr> i told him he could look at what you did
    <braunr> so i expect he'll ask soon
    <teythoon> cool :)
    <teythoon> my tool uses coccinelle to instrument c code, so this works in
      any environment
    <teythoon> one just needs a little glue and a method to get the data
    <braunr> seems reasonable
    <teythoon> for gnumach, i just stuff a tiny bit of code into the kdb

    <teythoon> hm debians bigmem patch with my code transformation makes
      gnumach hang early on
    <teythoon> i don't even get a single message from gnumach
    <braunr> ouch
    <teythoon> or it is somethign else entirely
    <teythoon> it didn't even work without my patches o_O
    <teythoon> weird
    <teythoon> uh oh, the kmem_cache array is not properly aligned
    <teythoon> braunr: http://paste.debian.net/74588/
    <braunr> teythoon: do you mean, with your patch ?
    <braunr> i'm not sure i understand
    <braunr> are you saying gnumach doesn't start because of an alignment issue
      ?
    <teythoon> no, that's unrelated
    <teythoon> i skipped the bigmem patch, have a running gnumach with
      instrumentation
    <braunr> hum, what is that aliased column ?
    <teythoon> but, despite my efforts with __attribute__((align(64))), i see
      lot's of accesses to kmem_cache objects which are not properly aligned
    <braunr> is that reported by the performance counters ?
    <teythoon> no
    <teythoon> http://paste.debian.net/74593/
    <braunr> aer those the previous lines accessed by other unrelated code ?
    <braunr> previous bytes in the same line*
    <teythoon> this is a patch generated to instrument the code
    <teythoon> so i instrument field access of the form i->a
    <teythoon> but if one does &i->a, my approach will no longer keep track of
      any access through that pointer
    <teythoon> so i do not count that as an access but as creating an alias for
      that field
    <braunr> ok
    <teythoon> so if that aliased count is not zero, the tool might
      underestimate the access count
    <teythoon> hm
    <teythoon> static struct kmem_cache kalloc_caches[KALLOC_NR_CACHES]
      __attribute__((align(64)));
    <teythoon> but
    <teythoon> nm gnumach|grep kalloc_caches
    <teythoon> c0226e20 b kalloc_caches
    <teythoon> ah, that's fine
    <braunr> yes
    <teythoon> nevr mind
    <braunr> don't we have a macro for the cache line size ?
    <teythoon> ah, there are a great many more kmem_caches around and noone
      told me ...
    <braunr> teythoon: eh :)
    <braunr> aren't you familiar with type-specific caches ?
    <teythoon> no, i'm not familiar with anything in gnumach-land
    <braunr> well, it's the regular slab allocator, carrying the same ideas
      since 1994
    <braunr> it's pretty much the same in linux and other modern unices
    <teythoon> ok
    <braunr> the main difference is likely that we allocate our caches
      statically because we have no kernel modules and know we'll never destroy
      them, only reap them
    <teythoon> is there a macro for the cache line size ?
    <teythoon> there is one burried in the linux source
    <teythoon> L1_CACHE_BYTES from linux/src/include/asm-i386/cache.h
    <braunr> there is one in kern/slab.h
    <teythoon> but it is out of date
    <teythoon> there is ?
    <braunr> but it's commented out
    <braunr> only used when SLAB_USE_CPU_POOLS is defined
    <braunr> but the build system should give you CPU_L1_SHIFT
    <teythoon> hm
    <braunr> and we probably should define CPU_L1_SIZE from that
      unconditionnally in config.h or a general param.h file if there is one
    <braunr> the architecture-specific one perhaps
    <braunr> although it's exported to userland so maybe not


## IRC, freenode, #hurd, 2014-01-07

    <teythoon> braunr: linux defines ____cacheline_aligned :
      http://lxr.free-electrons.com/source/include/linux/cache.h#L20
    <teythoon> where would i put a similar definition in gnumach ?
    <taylanub> .oO( four underscores ?!? )
    <teythoon> heh
    <teythoon> yes, four
    <braunr> teythoon: yes :)

    <teythoon> are kmem_cache objects ever allocated dynamically in gnumach ?
    <braunr> no
    <teythoon> hm
    <braunr> i figured that, since there are no kernel modules, there is no
      need to allocate them dynamically, since they're never destroyed
    <teythoon> so i aligned all statically declarations with
      __attribute__((align(1 << CPU_L1_SHIFT)))
    <teythoon> but i still see 77% of all accesses being to objects that are
      not properly aligned o_O
    <teythoon> ah
    <teythoon> >,<
    <braunr> you could add an assertion in kmem_cache_init to find out what's
      wrong
    <teythoon> *aligned
    <braunr> eh :)
    <braunr> right
    <teythoon> grr
    <teythoon> sweet, the kmem_caches are now all properly aligned :)
    <braunr> :)

    <braunr> hm
    <braunr> i guess i should change what vmstat reports as "cache" from the
      cached objects to the external ones (which map files and not anonymous
      memory)
    <teythoon> braunr: http://paste.debian.net/74869/
    <teythoon> turned out that struct kmem_cache was actually an easy target
    <teythoon> no bitfields, no embedded structs that were addressed as such
      (and not aliased)
    <braunr> :)


## IRC, freenode, #hurd, 2014-01-09

    <teythoon> braunr: i didn't quite get what you and youpi were talking about
      wrt to the alignment attribute
    <teythoon> define a type for struct kmem_cache with the alignment attribute
      ? is that possible ?
    <teythoon> ah, like it's done for kmem_cpu_pool
    <braunr> teythoon: that's it :)
    <braunr> note that aligning a struct doesn't change what sizeof returns
    <teythoon> heh, that save's one a whole lot of trouble indeed
    <braunr> you have to align a member inside for that
    <teythoon> why would it change the size ?
    <braunr> imagine an array of such structs
    <teythoon> ah
    <teythoon> right
    <teythoon> but it fits into two cachelines exactly
    <braunr> that wouldn't be a problem with an array either
    <teythoon> so an array of those will still be aligned element-wise
    <teythoon> yes
    <braunr> and it's often used like that, just as i did for the cpu pools
    <braunr> but then one is tempted to think the size of each element has
      changed too
    <braunr> and then use that technique for, say, reserving a whole cache line
      for one variable
    <teythoon> ah, now i get that remark ;)
    <braunr> :)

    <teythoon> braunr: i annotated struct kmem_cache in slab.h with
      __cacheline_aligned and it did not have the desired effect
    <braunr> can you show the diff please ?
    <teythoon> http://paste.debian.net/75192/
    <braunr> i don't know why :/
    <teythoon> that's how it's done for kmem_cpu_pool
    <braunr> i'll try it here
    <teythoon> wait
    <teythoon> i made a typo
    <teythoon> >,<
    <teythoon> __cachline_aligned
    <teythoon> bad one
    <braunr> uh :)
    <braunr> i don't see it
    <braunr> ah yes
    <braunr> missing e
    <teythoon> yep, works like a charme :)
    <teythoon> nice, good to know :)
    <braunr> :)
    <teythoon> given the previous discussion, shall i send it to the list or
      commit it right away ?
    <braunr> i'd say go ahead and commit