Search this site

The goal of this project is to make Darwin more usable by providing an installation ISO, documentation, and add-on software. You are welcome to join #puredarwin on irc.freenode.net if you would like to join PureDarwin development and to add to this site.

Recent site activity
For the curious‎ > ‎

DTrace

This page is about the DTrace facility in "Darwin-land".

Contents

  1. 1 What is DTrace?
    1. 1.1 Pros
    2. 1.2 Cons
  2. 2 Examples
    1. 2.1 A real world example: show missing files
    2. 2.2 A cliché example: "Hello world!"
      1. 2.2.1 Inline
      2. 2.2.2 In a file
    3. 2.3 Detect new process (successfully started)
    4. 2.4 Detect sigkill
    5. 2.5 Detect promiscuous mode changes
    6. 2.6 Watch setuid/setgid calls
    7. 2.7 Classic examples
  3. 3 D Programming language
    1. 3.1 A generic D script example
    2. 3.2 Probes and providers
      1. 3.2.1 Probe syntax
      2. 3.2.2 Listing probes
      3. 3.2.3 Available providers
      4. 3.2.4 Default variables
      5. 3.2.5 Macro variables
    3. 3.3 Predicate (optional)
    4. 3.4 Action (optional)
    5. 3.5 Kernel destructive actions
      1. 3.5.1 "-w" flag required for...
  4. 4 References

What is DTrace?

The dtrace command is a generic front-end to the DTrace facility.   The

command  implements  a  simple  interface to invoke the D language com-

piler, the ability to retrieve buffered trace data from the DTrace ker-

nel  facility,  and  a set of basic routines to format and print traced

data. said `man dtrace'.

Simply, DTrace is
  • a dynamic debugger and analysis tool (first appeared in Solaris 10 and OpenSolaris, then in Mac OS X 10.5)
  • a D programming language interpreter 
  • a monitoring tool (ie: Observing devices, such as disk or network activity) 
  • a profiling tool (performance analysis) 
  • a process, library, user function, (some) kernel function, the operating system (itself) inspector tool 
  • a bug finder
  • scalable 
DTrace may also be used as (or be a part of) 
  • a malware investigation tool 
  • a process deciphering tool 
  • a Troubleshooting software bugs 
  • a partial intrusion detection system 
  • an educative software (in order to explorer and understand how this operating system work)
Notes: Although PureDarwin is focused on i386, DTrace supports most of common architectures (-arch i386|x86_64|ppc|ppc64).
In Mac OS X 10.5, Instruments.app found in developer tools is another front-end to the DTrace facility.

Warning: If the operating system becomes too busy or if you attempt to trace too many events (ie: all function entry related to the kernel), DTrace can drop events and even abort tracing and execution. Consequently, security auditing with DTrace is impacted and more, system() action is not synchronous (so unreliable for security purpose).

Pros

  • Every process, or specific processes, can be traced "simultaneously".
  • Output is fully flexible.
  • Nanosecond timestamp for function execution measurements.
  • Minimum performance impact when used smartly.
  • No need to restart anything in order to trace.
  • Relatively safe.

Cons

  • Partially supported (around 20K probes available instead of 60K) on Darwin (dtrace -l | wc -l returns 22466).
  • DTrace cannot fetch low levels hardware data.
  • Performance impact when probe count goes under 36,000 probes (every kernel function entry/return) and 100,000 probes function in user-land.
    Consequently, not really suited for security auditing, because DTrace may drop events if the system is really busy or simply (but dramatically in this case) abort.
  • Need to be run as root (not sure it should belong to "cons" part).

Examples

A real world example: show missing files

On Linux systems, "strace -eopen -f ... 2>&1 | grep ENOENT" can be used to see which files a process tries to open. Using dtrace, the equivalent is:

  1. Open two terminals
  2. In the first terminal, run:
    opensnoop -ex
  3. In the second terminal, run the command that you want to inspect
In the first terminal, you will see the files that are tried to be opened but fail to open (e.g., ENOENT).

A cliché example: "Hello world!"

Inline

dtrace -n "BEGIN { trace(\"Hello World\!\"); exit(0);}"

dtrace: description 'BEGIN ' matched 1 probe

CPU     ID                    FUNCTION:NAME

  0      1                           :BEGIN   Hello World!

In a file

Edit and save Hello.d:

BEGIN 

{ 

trace("Hello World!"); 

exit(0); 

}


Then run the script:

dtrace -s Hello.d 

dtrace: script 'Hello.d' matched 1 probe

CPU     ID                    FUNCTION:NAME

  0      1                           :BEGIN   Hello World!

Detect new process (successfully started)

proc:::exec-success
{
printf("%s(pid=%d) started by uid - %d\n",execname, pid, uid);
}

Detect sigkill

proc:::signal-send
/args[2] == SIGKILL/
{
  printf("to %s",args[1]->pr_fname);
  printf(" at %d", timestamp);
  printf(" by (%d)", uid);
  printf(" from %s\n", execname);
}

Detect promiscuous mode changes

fbt:mach_kernel:ifnet_set_promiscuous:*

{

   printf("%s", execname);

}

Watch setuid/setgid calls

/* '*' == {entry|return} */

fbt:mach_kernel:setuid:*
{
   printf("%s", execname);
}

fbt:mach_kernel:setgid:*
{
   printf("%s", execname);
}

Classic examples

DTrace one-liners
The DTraceToolkit (~100 scripts)

D Programming language

It seems to have been based on C language for some parts.
(please add text)
  1. D programs are compiled in user-land
  2. And sent to the DTrace virtual machine in the kernel-land
  3. Then run inside the kernel's address space (not in the DTrace process, nor in the target process)
Most of the times, you will use copyin or copyinstr in order to copy data from user-land to kernel-land:


     copyin()      Copies len bytes of data from the user-space address uaddr

                   to the kernel-space address kaddr.


     copyinstr()   Copies a NUL-terminated string, at most len bytes long,

                   from user-space address uaddr to kernel-space address

                   kaddr.  The number of bytes actually copied, including the

                   terminating NUL, is returned in *done.

A generic D script example

probe description
/predicate/
{
   action
}

Probes and providers

A provider is a collection of probe.

Probe syntax

Four fields describe a probe:

provider:module:function:name

Examples:
fbt:mach_kernel:setuid:entry
syscall::open*:entry

Notes: Some special characters *, ?, [, and ] are allowed.
* is the well known wildcard extension. 
? character is for any single character in the name. 
[ and ] characters are used to specify a set of specific characters in the name.

Listing probes

dtrace -l lists all available probes.

   ID   PROVIDER            MODULE                          FUNCTION NAME
    1     dtrace                                                     BEGIN
    2     dtrace                                                     END
    3     dtrace                                                     ERROR
    4   lockstat       mach_kernel                      lck_mtx_lock adaptive-acquire
[...]
22527 plockstat1 libSystem.B.dylib    pthread_rwlock_unlock$UNIX2003 rw-release

Available providers

Some providers (non exhaustive list):

 Probe name Description
 fbt Function boundary tracing
 io Physical I/O requests/events
 mach_trap Kernel traps
 proc Process functions
 profile Sample count activity?
 syscall Kernel system calls
 vminfo 

Default variables

FIXME
 Variable name
 Description
 arg0...argN Function arguments and return value
 curpsinfo Structure of current process information
 errno Error code
 execname Processname
 pid Process id
 probefunc Probe function name
 probename Probe name
 tid Thread id
 timestamp Nanoseconds since boot

Macro variables

FIXME
 Macro name
 Description

 $[0-9]+

 macro arguments

 $egid

 effective group id

 $euid

 effective user id

 $gid real group id
 $pid process id
 $pgid process group id
 $ppid parent process id
 $projid project id
 $sid session id
 $target target process id
 $taskid task id
 $uid real user id

Predicate (optional)

When the predicate evaluates to true, "action" is executed.

 Predicate Name
 Description
 cpu == 0 true if the probe executes on cpu0
 ppid !=0 true if the parent process id is not 0
 arg0 == 0 true if first argument is 0
 ppid !=0 && arg0 == 0 ...

Action (optional)

C-style functions and semicolon (;) separation for the "body".

Examples: printf(), ustack(), trace, ...

Kernel destructive actions

Understand "destructive for the running OS".

"-w" flag required for...

-w  permit destructive actions

  • void stop()
  • voir raise()
  • void breakpoint(void)
  • void chill(int nanoseconds)
  • void panic(void)

panic: We are hanging here... 

It can force a kernel crash dump.
FIXME

References

man dtrace
man dtruss
man procsystime

Jon Haslam. http://wikis.sun.com/display/DTrace/Security, Sun Microsystems, 2008.

Glenn Brunette & Jon Haslam. Enhancing Security Awareness with DTrace, Sun Microsystems, 2006.

How To Use DTrace. http://www.sun.com/software/solaris/howtoguides/dtracehowto.jsp, Sun Microsystems, 2005.

Solaris Dynamic Tracing Guide. http://docs.sun.com/app/docs/doc/817-6223, Sun Microsystems, 2005.