Introduction to "perf top"
Perf top is a great tool for tuning vpp data plane code. As with any tool, it pays to know how it works and to construct test conditions which play to its strengths. It's a statistical profiler. Every so often, the underlying collection mechanism samples the current PC. Collection mechanisms vary by platform, and have different capabilities.
Typically, CPU hardware samples the current PC when a counter corresponding to a specific CPU event underflows. As in: "remember the current PC when a respectable number of cpu clock ticks have occurred." Statistically, an instruction which stalls frequently - for example, due to a missing prefetch - will show up frequently. Other instructions which do not stall will show up infrequently.