# Kernel Profiling

In countcallssect, you can revise the wildcards used in the probe to target all kernel functions. This can be useful if you are interested in indentifying what the kernel is performing over a specific time period.

However, doing so can cause considerable stress on the machine. In addition, this does not provide any indication of whether a specific function is being called too often during small time increments.

This section describes how to profile the kernel properly. kernelprof does this by providing a list of the top ten kernel functions called within a specific time period, and how many times each function was called during that time.

## kernelprof.stp

    #! /usr/bin/env stap

    global profile, pcount
    probe timer.profile {
    pcount <<< 1
      fn = probefunc ()
      if (fn != "") profile[fn] <<< 1
    }
    probe timer.ms(5000) {
      printf ("\n--- %d samples recorded:\n", @count(pcount))
      foreach (f in profile- limit 10) {
        printf ("%-30s\t%6d\n", f, @count(profile[f]))
      }
      delete profile
      delete pcount
    }

kernelprof records and outputs kernel functions called every 5 seconds. You can change this setting by editing `probe timer.ms(5000)` accordingly. kernelprofoutput contains an excerpt of the output over a 20-second period:

## kernelprof Sample Output

    [...]
    --- 10002 samples recorded:
    sys_recvfrom                         1
    memmove                              1
    __copy_from_user_ll                 17
    __copy_to_user_ll                   15
    mwait_idle                        5868
    link_path_walk                       1
    kfree                                1
    fget_light                           1
    audit_syscall_exit                   1
    __d_lookup                           1

    --- 10002 samples recorded:
    sysfs_read_file                      1
    free_poll_entry                      1
    syscall_exit_work                    1
    profile_hit                          1
    do_page_fault                        2
    _read_lock                           3
    kmap_atomic                          4
    strncpy_from_user                    1
    find_vma_prepare                     1
    __copy_from_user_ll                 12

    --- 10000 samples recorded:
    system_call                          4
    unix_stream_sendmsg                  1
    __d_path                             1
    do_page_fault                        2
    kmap_atomic                          1
    find_vma                             1
    __copy_to_user_ll                   16
    __copy_from_user_ll                 10
    mwait_idle                        5759
    memcpy_fromiovec                     1

    --- 10004 samples recorded:
    syscall_exit                         1
    __d_path                             1
    fput                                 2
    do_page_fault                        4
    strncpy_from_user                    1
    may_open                             1
    ide_outb                             1
    ide_outsw                            1
    __copy_to_user_ll                   11
    __copy_from_user_ll                 14
    [...]

kernelprof allows you to identify interesting kernel-specific performance facts about the system, such as kernel-intensive workloads and idle times (in this case, `mwait_idle`. Note that kernelprof does not count user-space functions, although it counts all kernel functions as part of the sample (even if the script cannot identify from which specific kernel function a call originated.