My company publishes simulation software that regularly performs prolonged CPU (and GPU, and occasionally I/O) intensive tasks. Our userbase are engineers, of whom some are embedded in large companies, of which some still subscribe to fairly archaic mindsets regarding IT that better belong in the 90s.
More than a few times now we have experienced situations where client computers which are extremely highly specced (some are commodity machines with 5 digit dollar values) are running simulations anywhere between 2-30 times slower than the same simulation on our internal machines. The client machines outspec our internal machines in basically every aspect (CPU, GPU, RAM, disk).
In a few rare instances we have been able to identify the antivirus software as the culprit, but this has typically been more due to serendipitous circumstances than any solid detective work on our part. We do not have a lot of experience with antivirus but from what we've observed, a lot of active scanning antivirus software does not appear to take up CPU itself, or even show up as a task manager process, but (presumably) the CPU utilisation is assigned to our process instead, causing these massive performance slowdowns.
On other occassions (sometimes when rapidly accessing anywhere between 1k-100k binary files) it will simply appear to frequently cause the CPU to throttle down to 0% usage, so the system effectively idles for prolonged durations (whereas on our internal machines the CPU maintains a steady 100% usage until the task is finished)
On occasion when the antivirus is turned off we have seen the performance immediately come up to the level expected. Unfortunately, it is very difficult to ask the IT departments of these companies to do this based on a hunch (and understandably so). In some circumstances they are also disinterested in genuinely supporting their engineers more than the absolute minimum so this becomes an unpleasant interdepartmental fight.
Is there a way we can reliably detect if the antivirus is causing performance issues so that we can better determine if it is worth having the argument?
Depending on the client we may or may not have administrator privileges, so I would welcome solutions or suggestions for either.