I work at Joyent – a cloud computing company – doing performance analysis of small to large cloud environments. Most of our systems have DTrace: the first widely used implementation of Dynamic Tracing, which we use for monitoring and performance analysis. But we also have some Linux, which I’ve been analyzing in the following situations:
- Competitive performance analysis: measuring the runtime of certain codepaths in Linux vs SmartOS.
- KVM performance tuning: we run Linux as guests of the KVM we ported to Illumos, and as part of that work I’ve studied performance from the host (via DTrace) and from the guest (via SystemTap).
- Real customer issues: replicated in a KVM lab environment for safe study with SystemTap.
I would prefer to log in to the KVM guests and run DTrace, but it’s not available on those Linux systems. (In the future, we may have a way to reach into the guest using DTrace from the host to trace its internals, be it Linux or Windows.) In the meantime, I need to do some serious performance analysis on Linux (including kernel code-path latency) so I’m using SystemTap.