Any interesting stories you could tell us, without violating the NDA? What do you measure usually? Time to complete something? Operations per second? Does the high resolution timer matter in the process of measurement or a simple difference between start time and stop time is usually good enough? The NDA applies even when hardware starts shipping?
All work that is done under NDA is always under NDA (depends, all NDAs are a bit different, some actually stipulate results can be released). Technically you could probably get some stuff released once product ships, but legal is a freaking nightmare to deal with. Not worth the effort, so I just treat everything as forever under NDA unless I am working on a released product.
What I measure depends on the benchmark used. Can be time spent, or transactions/s or some bandwidth measurement. My main tool is VMmark (
www.vmmark.com). I am on the team that developed this benchmark. This sort of prompted me to want to improve my programming skills as I am writing a lot of python and bash scripts for this. This benchmark measures transactions/s and response time for those transactions. We set a limit on response time and pack as many VMs on the host that can still function under that response time.
Depending on what the goal is I sometimes will collect low level stats on the CPU scheduler of ESX to see if there is something funky going on (this happens when new architecture is developed like AMD Zen, see below whitepaper). Or maybe look at the network, disk, or memory, etc.
Some examples whitepapers:
Optane PMEM
https://www.vmware.com/content/dam/...er/performance/pmem-balanced-profile-perf.pdf
Also contributed to this whitepaper
https://www.vmware.com/content/dam/...erformance/vsphere70u2-cpu-sched-amd-epyc.pdf
Haven't written any other whitepapers recently.
Usually I am just validating performance of vSphere on new CPUs. I am currently working on some upcoming servers which will be launched in the next year. See the CPU forums for ideas on what those might be...
Timers are kind of a moot point now. When I first started ~15 years ago this was a big issue. We always had to make sure time didn't drift by checking time sync for all systems at start and end of testing (or throughout the test for long running benchmarks). This was a big issue when the system was under heavy load. These days time is able to be kept very accurately. We still will check time sync, but I have never seen any drift in the past ~10 years on any system. I personally don't deal with real-time systems, probably much more of a concern in that space.