# Kernel Profiling

WAR STORY: Kernel Profiling http://sourceware.org/systemtap/wiki/WSKernelProfile?highlight=((WarStories)) http://sourceware.org/systemtap/examples/process/pf2.stp

In , you can revise the wildcards used in the probe to target all kernel functions. This can be useful if you are interested in indentifying what the kernel is performing over a specific time period.

However, doing so can cause considerable stress on the machine. In addition, this does not provide any indication of whether a specific function is being called too often during small time increments.

This section describes how to profile the kernel properly. does this by providing a list of the top ten kernel functions called within a specific time period, and how many times each function was called during that time.

kernelprof.stp

    #! /usr/bin/env stap

    global profile, pcount
    probe timer.profile {
    pcount <<< 1
      fn = probefunc ()
      if (fn != "") profile[fn] <<< 1
    }
    probe timer.ms(5000) {
      printf ("\n--- %d samples recorded:\n", @count(pcount))
      foreach (f in profile- limit 10) {
        printf ("%-30s\t%6d\n", f, @count(profile[f]))
      }
      delete profile
      delete pcount
    }

records and outputs kernel functions called every 5 seconds. You can change this setting by editing `probe timer.ms(5000)` accordingly. contains an excerpt of the output over a 20-second period:

Sample Output

    [...]
    --- 10002 samples recorded:
    sys_recvfrom                         1
    memmove                              1
    __copy_from_user_ll                 17
    __copy_to_user_ll                   15
    mwait_idle                        5868
    link_path_walk                       1
    kfree                                1
    fget_light                           1
    audit_syscall_exit                   1
    __d_lookup                           1

    --- 10002 samples recorded:
    sysfs_read_file                      1
    free_poll_entry                      1
    syscall_exit_work                    1
    profile_hit                          1
    do_page_fault                        2
    _read_lock                           3
    kmap_atomic                          4
    strncpy_from_user                    1
    find_vma_prepare                     1
    __copy_from_user_ll                 12

    --- 10000 samples recorded:
    system_call                          4
    unix_stream_sendmsg                  1
    __d_path                             1
    do_page_fault                        2
    kmap_atomic                          1
    find_vma                             1
    __copy_to_user_ll                   16
    __copy_from_user_ll                 10
    mwait_idle                        5759
    memcpy_fromiovec                     1

    --- 10004 samples recorded:
    syscall_exit                         1
    __d_path                             1
    fput                                 2
    do_page_fault                        4
    strncpy_from_user                    1
    may_open                             1
    ide_outb                             1
    ide_outsw                            1
    __copy_to_user_ll                   11
    __copy_from_user_ll                 14
    [...]

allows you to identify interesting kernel-specific performance facts about the system, such as kernel-intensive workloads and idle times (in this case, `mwait_idle`. Note that does not count user-space functions, although it counts all kernel functions as part of the sample (even if the script cannot identify from which specific kernel function a call originated.