Quantum computing systems need to be benchmarked in terms of practical tasks they would be expected to do.
A team of researchers at Cambridge Quantum Computing (CQC) and IBM has proposed “application-motivated” circuit classes for benchmarking: deep (relevant for state preparation in the variational quantum eigensolver algorithm), shallow (inspired by IQP-type circuits that might be useful for near term quantum machine learning), and square (inspired by the quantum volume benchmark).
They have quantified the performance of a quantum computing system in running circuits from these classes using several figures of merit, all of which require exponential classical computing resources and a polynomial number of classical samples (bitstrings) from the system.
The team studied how performance varies with the compilation strategy used and the device on which the circuit is run.
Using systems made available by IBM Quantum, they examined their performance, showing that noise-aware compilation strategies may be beneficial, and that device connectivity and noise levels play a crucial role in the performance of the system according to our benchmarks.
The paper can be read there.