This directory contains sets of scripts for testing plain kprobes or
systemtap probes on a set of kernel functions. It tries to bisect the
list if a probe on a set of kernel functions crashes the system.

Here's a workflow:

1) First put a list of functions to probe in a file, by default called
'probes.all'. If you'd like a list of all kernel functions, you can
run 'gen_code_all.sh'.

2a) To test plain kprobes, run:

# ./kprobes_test.py

2b) To test systemtap probes, run:

# ./stap_probes_test.py

3) Wait for the script to finish. Feel free to watch kprobes_test.log
for progress. If the system crashes, restart it. Either
'kprobe_test.py' or 'stap_probes_test.py' should get automatically run
when the system gets restarted.

If you'd like help monitoring the target system, you can use
'monitor_system.py', which will try to automatically reboot a remote
system if the system can't be reached.

4) When the script (either ./kprobes_test.py or ./stap_probes_test.py)
finishes, you'll see the following output files.

- probes.unregistered is a file that contains the list of kernel
  functions where a probe couldn't be registered.
- probes.untriggered is a file that contains the list of kernel
  functions where a probe could be registered, but the kernel
  function wasn't actually called.  So, we don't really know whether
  these functions passed or failed.
- probes.passed is a file that contains the list of kernel functions
  where a probe was registered and the kerenel function was called
  with no crash.
- probes.failed is a file that contains the list of kernel functions
  where a probe was registered and the kernel function was called
  that caused a crash.


Here are the original instructions for 'gen_passlist', which tested
systemtap probe safety.

====

To start a kprobes test run:
   runtest passlist.exp

----
Here is an implementation of generating passlist for safe-mode probes
based on the discussion in former thread
(http://sourceware.org/ml/systemtap/2006-q3/msg00574.html).

Its main idea is:
 1) Fetch a group of probe points from probes.pending, probe them and
    run some workloads(e.g. runltp) parallely meanwhile.

 2) If the probe test ends without crash, those actually triggered
    probe points are moved into probes.passed and those untriggered 
    are into probes.untriggered; 
    If the probe test crashes the system, it will be resumed
    automatically after system reboot. Those probe points which have 
    been triggered are also moved into probes.passed, but those 
    untriggered ones are moved into probes.failed.

 3) Repeat the above until probes.pending becomes empty, then:
    Normally, probes.pending is reinitialized from probes.failed and
    probes.untriggered, then start the next iteration; 
    But if max running level (now 3) is reached, or probes.pending,
    probes.failed and probes.untriggered are all empty, stop the 
    whole test.

To be able to resume after a crash, this test will append its startup
codes to the end of /etc/rc.d/rc.local at the beginning and remove 
them at the end of the test.

The gen_tapset_all.sh is used to generate a "probes.all" file based
on current tapset of systemtap.

It is suggested to use a script in a remote server to restart the 
test machine automatically in case it is crashed by the probe test.

Instructions:
 0) Please jump to 6) directly if you use all default settings
 1) Define your list of probe points in a file named "probes.all";
    otherwise, it will be automatically extracted from 
      stap -p2 -e 'probe kernel.function("*"){}
 2) Define your own workloads by changing the "benchs" list variable; 
    otherwise, several LTP testcases will be used
 3) Define how may times you want the test to be iterated by changing
    MAX_RUNNING_LEVEL variable; otherwise, 3 is the default
 4) Define the group size for different iteration level by changing 
    proper_current_size() function; otherwise, they will be set in 
    a decreasing order based on the length of probes.all
 5) Remove /stp_genpasslist_running if you want to start a test from
    the beginning after a system crash
 6) Start a test:
      runtest passlist.exp
