{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3b8ae811",
   "metadata": {},
   "source": [
    "# Profiling notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1097bd1e",
   "metadata": {},
   "source": [
    "This notebook collects and compares the run time for several notebooks. These notebooks are specified in the `metrics.py`, in the `EXPERIMENT_NOTEBOOKS` variable.\n",
    "Simply run the whole notebook, and the results will be displayed in tables at the end.\n",
    "\n",
    "Each notebook listed in the `EXPERIMENT_NOTEBOOKS` (see `metrics.py`) table must have the `run_experiment()` defined. Optionally, you can define the `close_experiment()` function.\n",
    "\n",
    "This profiler will profile the `run_experiment` function, and after that's done, call the `close_experiment` (if it exists). The `close_experiment` function is not mandatory, but if there are any resources that need to be closed, you can implement that here. The profiler measures the times listed in the `METHODS` variable (see `metrics.py`), and the total time."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13ae83c1",
   "metadata": {},
   "source": [
    "After the profiling is done, the notebook generates a file in this directory for each notebook. This file contains the detailed profiling report. For the notebook `<notebook>.ipynb` it generates `<notebook>.ipynb.prof` file, which can be opened with snakeviz (`pip install snakeviz`): `snakeviz <notebook>.ipynb.prof`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04139070",
   "metadata": {},
   "source": [
    "## Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e2521e52",
   "metadata": {},
   "outputs": [],
   "source": [
    "# `benchmark_mode` sets whether we run the schedules in benchmark mode.\n",
    "# If it's benchmark mode, we override the reference measurements file\n",
    "# with the current timing values, and that will be those will be the new reference values.\n",
    "benchmark_mode = True\n",
    "profiling_reference_filename = \"profiling_reference_values.pickle\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "90faba6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The end result table will display each cell in different colors.\n",
    "# Each value's \"sigma\" is practically it's measurement error,\n",
    "# and if the current time is above/below\n",
    "# the `reference value±sigma*sigma_multiplier_threshold`\n",
    "# the cell will be displayed in different colors.\n",
    "sigma_multiplier_threshold = 2.0  # 2.0 is a reasonable value."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf4f5061",
   "metadata": {},
   "source": [
    "## Loading reference data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "758b1d4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reference values for profiling.\n",
    "# Each notebook has a reference timing value.\n",
    "import pickle\n",
    "from os.path import exists\n",
    "\n",
    "if not benchmark_mode:\n",
    "    if not exists(profiling_reference_filename):\n",
    "        raise RuntimeError(\n",
    "            f\"Reference file '{profiling_reference_filename}' does not exist! \"\n",
    "            f\"Make sure this file is created by first running the profiling with 'benchmark_mode=True'!\"\n",
    "        )\n",
    "    with open(profiling_reference_filename, \"rb\") as f:\n",
    "        reference = pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e840374",
   "metadata": {},
   "source": [
    "## Running the profiling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "109b43ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "simple_binned_acquisition: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:32<00:00,  3.28s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated `simple_binned_acquisition.prof` profiling file\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7680000000.0 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7680000000.0'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7680000000.0 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7680000000.0'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7680000000.0 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7680000000.0'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7680000000.0 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7680000000.0'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7680000000.0 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7680000000.0'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7680000000.0 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7680000000.0'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7680000000.0 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7680000000.0'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7680000000.0 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7680000000.0'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7680000000.0 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7680000000.0'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7680000000.0 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7680000000.0'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/backends/circuit_to_device.py:414: RuntimeWarning: Clock 'q0.ro' has conflicting frequency definitions: 7700100502.512563 Hz in the schedule and 7600000000.0 Hz in the device config. The clock is set to '7700100502.512563'. Ensure the schedule clock resource matches the device config clock frequency or set the clock frequency in the device config to np.NaN to omit this warning.\n",
      "  warnings.warn(\n",
      "resonator_spectroscopy: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:12<00:00,  1.27s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated `resonator_spectroscopy.prof` profiling file\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "random_gates: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:29<00:00,  2.94s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated `random_gates.prof` profiling file\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loops_with_measurements:   0%|                                                                                                                                                                               | 0/10 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.22.2.dev11+ge0c4cd4c7.dirty\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/__init__.py\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loops_with_measurements:  10%|████████████████▋                                                                                                                                                      | 1/10 [00:04<00:39,  4.35s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.22.2.dev11+ge0c4cd4c7.dirty\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/__init__.py\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loops_with_measurements:  20%|█████████████████████████████████▍                                                                                                                                     | 2/10 [00:08<00:34,  4.35s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.22.2.dev11+ge0c4cd4c7.dirty\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/__init__.py\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loops_with_measurements:  30%|██████████████████████████████████████████████████                                                                                                                     | 3/10 [00:13<00:30,  4.42s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.22.2.dev11+ge0c4cd4c7.dirty\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/__init__.py\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loops_with_measurements:  40%|██████████████████████████████████████████████████████████████████▊                                                                                                    | 4/10 [00:17<00:25,  4.31s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.22.2.dev11+ge0c4cd4c7.dirty\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/__init__.py\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loops_with_measurements:  50%|███████████████████████████████████████████████████████████████████████████████████▌                                                                                   | 5/10 [00:21<00:21,  4.34s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.22.2.dev11+ge0c4cd4c7.dirty\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/__init__.py\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loops_with_measurements:  60%|████████████████████████████████████████████████████████████████████████████████████████████████████▏                                                                  | 6/10 [00:26<00:17,  4.34s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.22.2.dev11+ge0c4cd4c7.dirty\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/__init__.py\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loops_with_measurements:  70%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉                                                  | 7/10 [00:30<00:13,  4.39s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.22.2.dev11+ge0c4cd4c7.dirty\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/__init__.py\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loops_with_measurements:  80%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                 | 8/10 [00:34<00:08,  4.32s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.22.2.dev11+ge0c4cd4c7.dirty\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/__init__.py\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loops_with_measurements:  90%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎                | 9/10 [00:39<00:04,  4.40s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.22.2.dev11+ge0c4cd4c7.dirty\n",
      "/home/gabor/projects/quantify-scheduler/quantify_scheduler/__init__.py\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loops_with_measurements: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:43<00:00,  4.34s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated `loops_with_measurements.prof` profiling file\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "multidim_batched_sweep:   0%|                                                                                                                                                                                | 0/10 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting batched measurement...\n",
      "Iterative settable(s) [outer loop(s)]:\n",
      "\t --- (None) --- \n",
      "Batched settable(s):\n",
      "\t repetition, spectator_rotation, phase \n",
      "Batch size limit: 640\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "141480aa649940e4b0951b26e6ed4253",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Completed:   0%|           [ elapsed time: 00:00 | time left: ? ] it"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "multidim_batched_sweep:  10%|████████████████▊                                                                                                                                                       | 1/10 [00:06<00:57,  6.41s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting batched measurement...\n",
      "Iterative settable(s) [outer loop(s)]:\n",
      "\t --- (None) --- \n",
      "Batched settable(s):\n",
      "\t repetition, spectator_rotation, phase \n",
      "Batch size limit: 640\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b30795cff15c455d820fcfb987708b64",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Completed:   0%|           [ elapsed time: 00:00 | time left: ? ] it"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "multidim_batched_sweep:  20%|█████████████████████████████████▌                                                                                                                                      | 2/10 [00:12<00:51,  6.47s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting batched measurement...\n",
      "Iterative settable(s) [outer loop(s)]:\n",
      "\t --- (None) --- \n",
      "Batched settable(s):\n",
      "\t repetition, spectator_rotation, phase \n",
      "Batch size limit: 640\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dc8fd78f554f489ba288404c4c8f07a8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Completed:   0%|           [ elapsed time: 00:00 | time left: ? ] it"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "multidim_batched_sweep:  30%|██████████████████████████████████████████████████▍                                                                                                                     | 3/10 [00:19<00:45,  6.44s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting batched measurement...\n",
      "Iterative settable(s) [outer loop(s)]:\n",
      "\t --- (None) --- \n",
      "Batched settable(s):\n",
      "\t repetition, spectator_rotation, phase \n",
      "Batch size limit: 640\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d61924d12a2a4c98b6aa23591c2c0c81",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Completed:   0%|           [ elapsed time: 00:00 | time left: ? ] it"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "multidim_batched_sweep:  40%|███████████████████████████████████████████████████████████████████▏                                                                                                    | 4/10 [00:25<00:38,  6.40s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting batched measurement...\n",
      "Iterative settable(s) [outer loop(s)]:\n",
      "\t --- (None) --- \n",
      "Batched settable(s):\n",
      "\t repetition, spectator_rotation, phase \n",
      "Batch size limit: 640\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "399151534ace41bc8bef30587784a139",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Completed:   0%|           [ elapsed time: 00:00 | time left: ? ] it"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "multidim_batched_sweep:  50%|████████████████████████████████████████████████████████████████████████████████████                                                                                    | 5/10 [00:32<00:32,  6.43s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting batched measurement...\n",
      "Iterative settable(s) [outer loop(s)]:\n",
      "\t --- (None) --- \n",
      "Batched settable(s):\n",
      "\t repetition, spectator_rotation, phase \n",
      "Batch size limit: 640\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "719c9d4f3dd9424fa68b12218a77a6ef",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Completed:   0%|           [ elapsed time: 00:00 | time left: ? ] it"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "multidim_batched_sweep:  60%|████████████████████████████████████████████████████████████████████████████████████████████████████▊                                                                   | 6/10 [00:38<00:25,  6.46s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting batched measurement...\n",
      "Iterative settable(s) [outer loop(s)]:\n",
      "\t --- (None) --- \n",
      "Batched settable(s):\n",
      "\t repetition, spectator_rotation, phase \n",
      "Batch size limit: 640\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ea0d685a29684b01978cca485ed3c923",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Completed:   0%|           [ elapsed time: 00:00 | time left: ? ] it"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "multidim_batched_sweep:  70%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                                  | 7/10 [00:45<00:19,  6.45s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting batched measurement...\n",
      "Iterative settable(s) [outer loop(s)]:\n",
      "\t --- (None) --- \n",
      "Batched settable(s):\n",
      "\t repetition, spectator_rotation, phase \n",
      "Batch size limit: 640\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "862bb6f998404e7cb297d2280a808d9d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Completed:   0%|           [ elapsed time: 00:00 | time left: ? ] it"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "multidim_batched_sweep:  80%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍                                 | 8/10 [00:51<00:12,  6.44s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting batched measurement...\n",
      "Iterative settable(s) [outer loop(s)]:\n",
      "\t --- (None) --- \n",
      "Batched settable(s):\n",
      "\t repetition, spectator_rotation, phase \n",
      "Batch size limit: 640\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3a7977a4f47940cc89efdab6bfd94a49",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Completed:   0%|           [ elapsed time: 00:00 | time left: ? ] it"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "multidim_batched_sweep:  90%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏                | 9/10 [00:58<00:06,  6.49s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting batched measurement...\n",
      "Iterative settable(s) [outer loop(s)]:\n",
      "\t --- (None) --- \n",
      "Batched settable(s):\n",
      "\t repetition, spectator_rotation, phase \n",
      "Batch size limit: 640\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b7a323884c794e539842cbf64bea0b58",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Completed:   0%|           [ elapsed time: 00:00 | time left: ? ] it"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "multidim_batched_sweep: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [01:04<00:00,  6.45s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generated `multidim_batched_sweep.prof` profiling file\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "import metrics\n",
    "\n",
    "measured_data = metrics.measure_experiment_runtimes()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "34fd0713",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('simple_binned_acquisition',\n",
       "  [(1.2875609662, 0.07378692750853963),\n",
       "   (0.09055239879999999, 0.0014548760144358257),\n",
       "   (0.2917579839000001, 0.08109964024850098),\n",
       "   (0.0281971761, 0.00028751424373139227),\n",
       "   (0.0105310604, 0.0008908094153272372),\n",
       "   (0.0025370000000001017, 2.9103830456733704e-11)],\n",
       "  (1.726121195200001, 0.06709376242028985)),\n",
       " ('resonator_spectroscopy',\n",
       "  [(0.22818396000000002, 0.06190343802191572),\n",
       "   (0.061770644400000005, 0.0007160129325833485),\n",
       "   (0.05975772509999999, 0.05384526784258865),\n",
       "   (0.0013046132, 3.643896816815789e-05),\n",
       "   (0.0099989413, 0.0002736500742273392),\n",
       "   (0.48784000000000305, 9.125060374972142e-09)],\n",
       "  (0.5871041078, 0.0818179918891472)),\n",
       " ('random_gates',\n",
       "  [(1.0911051992, 0.06307970276291672),\n",
       "   (0.09293269380000001, 0.001303993266840501),\n",
       "   (0.26497131230000004, 0.0052465345296674026),\n",
       "   (0.0006509752000000001, 8.432517000273423e-06),\n",
       "   (0.0045667877, 6.299631062860032e-05),\n",
       "   (0.00026014000000001476, 6.301164130741662e-12)],\n",
       "  (1.4190985734000008, 0.07205644318149013)),\n",
       " ('loops_with_measurements',\n",
       "  [(2.6091251336, 0.009564247905489043),\n",
       "   (0.0659306276, 0.000477378713084575),\n",
       "   (0.0423343382, 0.0010128287973002726),\n",
       "   (0.0006680734000000001, 3.868058515638048e-05),\n",
       "   (0.004808833, 7.261637788406787e-05),\n",
       "   (20.23210000000001, 4.1295309247228556e-07)],\n",
       "  (2.733087237000003, 0.009196191665407858)),\n",
       " ('multidim_batched_sweep',\n",
       "  [(3.0812415328, 0.10435291773775217),\n",
       "   (0.2621062487, 0.002803255145907753),\n",
       "   (nan, nan),\n",
       "   (0.0008072722000000001, 9.233291447792206e-06),\n",
       "   (0.0092235426, 0.0002691701188643441),\n",
       "   (0.0660454400000041, 1.3170890159654386e-09)],\n",
       "  (3.9227151624000003, 0.09037171447641731))]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "measured_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a02eac5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "if benchmark_mode:\n",
    "    with open(profiling_reference_filename, \"wb\") as f:\n",
    "        pickle.dump(measured_data, f)\n",
    "    reference = measured_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebf6126f",
   "metadata": {},
   "source": [
    "## Displaying the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "dc64e1bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('simple_binned_acquisition',\n",
       "  [(1.2875609662, 0.07378692750853963),\n",
       "   (0.09055239879999999, 0.0014548760144358257),\n",
       "   (0.2917579839000001, 0.08109964024850098),\n",
       "   (0.0281971761, 0.00028751424373139227),\n",
       "   (0.0105310604, 0.0008908094153272372),\n",
       "   (0.0025370000000001017, 2.9103830456733704e-11)],\n",
       "  (1.726121195200001, 0.06709376242028985)),\n",
       " ('resonator_spectroscopy',\n",
       "  [(0.22818396000000002, 0.06190343802191572),\n",
       "   (0.061770644400000005, 0.0007160129325833485),\n",
       "   (0.05975772509999999, 0.05384526784258865),\n",
       "   (0.0013046132, 3.643896816815789e-05),\n",
       "   (0.0099989413, 0.0002736500742273392),\n",
       "   (0.48784000000000305, 9.125060374972142e-09)],\n",
       "  (0.5871041078, 0.0818179918891472)),\n",
       " ('random_gates',\n",
       "  [(1.0911051992, 0.06307970276291672),\n",
       "   (0.09293269380000001, 0.001303993266840501),\n",
       "   (0.26497131230000004, 0.0052465345296674026),\n",
       "   (0.0006509752000000001, 8.432517000273423e-06),\n",
       "   (0.0045667877, 6.299631062860032e-05),\n",
       "   (0.00026014000000001476, 6.301164130741662e-12)],\n",
       "  (1.4190985734000008, 0.07205644318149013)),\n",
       " ('loops_with_measurements',\n",
       "  [(2.6091251336, 0.009564247905489043),\n",
       "   (0.0659306276, 0.000477378713084575),\n",
       "   (0.0423343382, 0.0010128287973002726),\n",
       "   (0.0006680734000000001, 3.868058515638048e-05),\n",
       "   (0.004808833, 7.261637788406787e-05),\n",
       "   (20.23210000000001, 4.1295309247228556e-07)],\n",
       "  (2.733087237000003, 0.009196191665407858)),\n",
       " ('multidim_batched_sweep',\n",
       "  [(3.0812415328, 0.10435291773775217),\n",
       "   (0.2621062487, 0.002803255145907753),\n",
       "   (nan, nan),\n",
       "   (0.0008072722000000001, 9.233291447792206e-06),\n",
       "   (0.0092235426, 0.0002691701188643441),\n",
       "   (0.0660454400000041, 1.3170890159654386e-09)],\n",
       "  (3.9227151624000003, 0.09037171447641731))]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2c06a609",
   "metadata": {},
   "outputs": [],
   "source": [
    "import metrics\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ea1af4bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "table = []\n",
    "header = []\n",
    "table_diff = []\n",
    "header_diff = []\n",
    "\n",
    "header.append(\"\")\n",
    "header_diff.append(\"\")\n",
    "for method in metrics.METHODS:\n",
    "    header.append(method[1])\n",
    "    header_diff.append(method[1])\n",
    "header.append(\"total\")\n",
    "header_diff.append(\"total\")\n",
    "\n",
    "for row_id, (experiment_notebook, times, total_time) in enumerate(measured_data):\n",
    "    row = []\n",
    "    row_diff = []\n",
    "    row.append(experiment_notebook)\n",
    "    row_diff.append(experiment_notebook)\n",
    "    for column_id, time in enumerate(times):\n",
    "        expected_value = time[0]\n",
    "        sigma = time[1]\n",
    "        row.append(f\"{expected_value:.2g} ± {sigma:.2g} s\")\n",
    "\n",
    "        time_diff = expected_value - reference[row_id][1][column_id][0]\n",
    "        row_diff.append(f\"{time_diff:.2g} ± {sigma:.2g} s\")\n",
    "\n",
    "    row.append(f\"{total_time[0]:.2g} ± {total_time[1]:.2g} s\")\n",
    "\n",
    "    total_time_diff = total_time[0] - reference[row_id][2][0]\n",
    "    row_diff.append(f\"{total_time_diff:.2g} ± {total_time[1]:.2g} s\")\n",
    "\n",
    "    table.append(row)\n",
    "    table_diff.append(row_diff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "fe75788c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def diff_to_style(current, ref):\n",
    "    green = \"#d0ffd0\"\n",
    "    red = \"#ffd0d0\"\n",
    "    val, sigma = current[0], current[1]\n",
    "    ref_val, ref_sigma = ref[0], ref[1]\n",
    "    if (val - sigma * sigma_multiplier_threshold) > (\n",
    "        ref_val + ref_sigma * sigma_multiplier_threshold\n",
    "    ):\n",
    "        return f\"background-color: {red}\"\n",
    "    if (val + sigma * sigma_multiplier_threshold) < (\n",
    "        ref_val - ref_sigma * sigma_multiplier_threshold\n",
    "    ):\n",
    "        return f\"background-color: {green}\"\n",
    "    return \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a6b824c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "style_table = []\n",
    "\n",
    "for row_id, (experiment_notebook, times, total_time) in enumerate(measured_data):\n",
    "    row = []\n",
    "    row.append(\"\")\n",
    "    for column_id, time in enumerate(times):\n",
    "        if row_id < len(reference) and column_id < len(reference[row_id][1]):\n",
    "            row.append(diff_to_style(time, reference[row_id][1][column_id]))\n",
    "        else:\n",
    "            row.append(\"\")\n",
    "    if row_id < len(reference):\n",
    "        row.append(diff_to_style(total_time, reference[row_id][2]))\n",
    "    else:\n",
    "        row.append(\"\")\n",
    "    style_table.append(row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2271fdaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "style_table = np.array(style_table)\n",
    "style_properties = {\"border\": \"1px solid gray\"}\n",
    "styles = [\n",
    "    dict(\n",
    "        selector=\"caption\",\n",
    "        props=[(\"text-align\", \"center\"), (\"font-size\", \"200%\"), (\"color\", \"black\")],\n",
    "    )\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d74afb03",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(table, columns=header)\n",
    "df = df.style.set_properties(**style_properties).apply(lambda _: style_table, axis=None)\n",
    "df = df.set_caption(\"Measured times\").set_table_styles(styles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bb38048d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_diff = pd.DataFrame(table_diff, columns=header)\n",
    "df_diff = df_diff.style.set_properties(**style_properties).apply(lambda _: style_table, axis=None)\n",
    "df_diff = df_diff.set_caption(\"Measured diffs to reference\").set_table_styles(styles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "822cb265",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_877e9 caption {\n",
       "  text-align: center;\n",
       "  font-size: 200%;\n",
       "  color: black;\n",
       "}\n",
       "#T_877e9_row0_col0, #T_877e9_row0_col1, #T_877e9_row0_col2, #T_877e9_row0_col3, #T_877e9_row0_col4, #T_877e9_row0_col5, #T_877e9_row0_col6, #T_877e9_row0_col7, #T_877e9_row1_col0, #T_877e9_row1_col1, #T_877e9_row1_col2, #T_877e9_row1_col3, #T_877e9_row1_col4, #T_877e9_row1_col5, #T_877e9_row1_col6, #T_877e9_row1_col7, #T_877e9_row2_col0, #T_877e9_row2_col1, #T_877e9_row2_col2, #T_877e9_row2_col3, #T_877e9_row2_col4, #T_877e9_row2_col5, #T_877e9_row2_col6, #T_877e9_row2_col7, #T_877e9_row3_col0, #T_877e9_row3_col1, #T_877e9_row3_col2, #T_877e9_row3_col3, #T_877e9_row3_col4, #T_877e9_row3_col5, #T_877e9_row3_col6, #T_877e9_row3_col7, #T_877e9_row4_col0, #T_877e9_row4_col1, #T_877e9_row4_col2, #T_877e9_row4_col3, #T_877e9_row4_col4, #T_877e9_row4_col5, #T_877e9_row4_col6, #T_877e9_row4_col7 {\n",
       "  border: 1px solid gray;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_877e9\">\n",
       "  <caption>Measured times</caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_877e9_level0_col0\" class=\"col_heading level0 col0\" ></th>\n",
       "      <th id=\"T_877e9_level0_col1\" class=\"col_heading level0 col1\" >compile</th>\n",
       "      <th id=\"T_877e9_level0_col2\" class=\"col_heading level0 col2\" >prepare</th>\n",
       "      <th id=\"T_877e9_level0_col3\" class=\"col_heading level0 col3\" >schedule</th>\n",
       "      <th id=\"T_877e9_level0_col4\" class=\"col_heading level0 col4\" >run</th>\n",
       "      <th id=\"T_877e9_level0_col5\" class=\"col_heading level0 col5\" >process</th>\n",
       "      <th id=\"T_877e9_level0_col6\" class=\"col_heading level0 col6\" >schedule_duration</th>\n",
       "      <th id=\"T_877e9_level0_col7\" class=\"col_heading level0 col7\" >total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_877e9_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_877e9_row0_col0\" class=\"data row0 col0\" >simple_binned_acquisition</td>\n",
       "      <td id=\"T_877e9_row0_col1\" class=\"data row0 col1\" >1.3 ± 0.074 s</td>\n",
       "      <td id=\"T_877e9_row0_col2\" class=\"data row0 col2\" >0.091 ± 0.0015 s</td>\n",
       "      <td id=\"T_877e9_row0_col3\" class=\"data row0 col3\" >0.29 ± 0.081 s</td>\n",
       "      <td id=\"T_877e9_row0_col4\" class=\"data row0 col4\" >0.028 ± 0.00029 s</td>\n",
       "      <td id=\"T_877e9_row0_col5\" class=\"data row0 col5\" >0.011 ± 0.00089 s</td>\n",
       "      <td id=\"T_877e9_row0_col6\" class=\"data row0 col6\" >0.0025 ± 2.9e-11 s</td>\n",
       "      <td id=\"T_877e9_row0_col7\" class=\"data row0 col7\" >1.7 ± 0.067 s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_877e9_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_877e9_row1_col0\" class=\"data row1 col0\" >resonator_spectroscopy</td>\n",
       "      <td id=\"T_877e9_row1_col1\" class=\"data row1 col1\" >0.23 ± 0.062 s</td>\n",
       "      <td id=\"T_877e9_row1_col2\" class=\"data row1 col2\" >0.062 ± 0.00072 s</td>\n",
       "      <td id=\"T_877e9_row1_col3\" class=\"data row1 col3\" >0.06 ± 0.054 s</td>\n",
       "      <td id=\"T_877e9_row1_col4\" class=\"data row1 col4\" >0.0013 ± 3.6e-05 s</td>\n",
       "      <td id=\"T_877e9_row1_col5\" class=\"data row1 col5\" >0.01 ± 0.00027 s</td>\n",
       "      <td id=\"T_877e9_row1_col6\" class=\"data row1 col6\" >0.49 ± 9.1e-09 s</td>\n",
       "      <td id=\"T_877e9_row1_col7\" class=\"data row1 col7\" >0.59 ± 0.082 s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_877e9_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_877e9_row2_col0\" class=\"data row2 col0\" >random_gates</td>\n",
       "      <td id=\"T_877e9_row2_col1\" class=\"data row2 col1\" >1.1 ± 0.063 s</td>\n",
       "      <td id=\"T_877e9_row2_col2\" class=\"data row2 col2\" >0.093 ± 0.0013 s</td>\n",
       "      <td id=\"T_877e9_row2_col3\" class=\"data row2 col3\" >0.26 ± 0.0052 s</td>\n",
       "      <td id=\"T_877e9_row2_col4\" class=\"data row2 col4\" >0.00065 ± 8.4e-06 s</td>\n",
       "      <td id=\"T_877e9_row2_col5\" class=\"data row2 col5\" >0.0046 ± 6.3e-05 s</td>\n",
       "      <td id=\"T_877e9_row2_col6\" class=\"data row2 col6\" >0.00026 ± 6.3e-12 s</td>\n",
       "      <td id=\"T_877e9_row2_col7\" class=\"data row2 col7\" >1.4 ± 0.072 s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_877e9_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_877e9_row3_col0\" class=\"data row3 col0\" >loops_with_measurements</td>\n",
       "      <td id=\"T_877e9_row3_col1\" class=\"data row3 col1\" >2.6 ± 0.0096 s</td>\n",
       "      <td id=\"T_877e9_row3_col2\" class=\"data row3 col2\" >0.066 ± 0.00048 s</td>\n",
       "      <td id=\"T_877e9_row3_col3\" class=\"data row3 col3\" >0.042 ± 0.001 s</td>\n",
       "      <td id=\"T_877e9_row3_col4\" class=\"data row3 col4\" >0.00067 ± 3.9e-05 s</td>\n",
       "      <td id=\"T_877e9_row3_col5\" class=\"data row3 col5\" >0.0048 ± 7.3e-05 s</td>\n",
       "      <td id=\"T_877e9_row3_col6\" class=\"data row3 col6\" >20 ± 4.1e-07 s</td>\n",
       "      <td id=\"T_877e9_row3_col7\" class=\"data row3 col7\" >2.7 ± 0.0092 s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_877e9_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_877e9_row4_col0\" class=\"data row4 col0\" >multidim_batched_sweep</td>\n",
       "      <td id=\"T_877e9_row4_col1\" class=\"data row4 col1\" >3.1 ± 0.1 s</td>\n",
       "      <td id=\"T_877e9_row4_col2\" class=\"data row4 col2\" >0.26 ± 0.0028 s</td>\n",
       "      <td id=\"T_877e9_row4_col3\" class=\"data row4 col3\" >nan ± nan s</td>\n",
       "      <td id=\"T_877e9_row4_col4\" class=\"data row4 col4\" >0.00081 ± 9.2e-06 s</td>\n",
       "      <td id=\"T_877e9_row4_col5\" class=\"data row4 col5\" >0.0092 ± 0.00027 s</td>\n",
       "      <td id=\"T_877e9_row4_col6\" class=\"data row4 col6\" >0.066 ± 1.3e-09 s</td>\n",
       "      <td id=\"T_877e9_row4_col7\" class=\"data row4 col7\" >3.9 ± 0.09 s</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7661591c4f10>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# If the cell is green (or red), the current time\n",
    "# is significantly less (or more) than the reference time.\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f25477aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_e253c caption {\n",
       "  text-align: center;\n",
       "  font-size: 200%;\n",
       "  color: black;\n",
       "}\n",
       "#T_e253c_row0_col0, #T_e253c_row0_col1, #T_e253c_row0_col2, #T_e253c_row0_col3, #T_e253c_row0_col4, #T_e253c_row0_col5, #T_e253c_row0_col6, #T_e253c_row0_col7, #T_e253c_row1_col0, #T_e253c_row1_col1, #T_e253c_row1_col2, #T_e253c_row1_col3, #T_e253c_row1_col4, #T_e253c_row1_col5, #T_e253c_row1_col6, #T_e253c_row1_col7, #T_e253c_row2_col0, #T_e253c_row2_col1, #T_e253c_row2_col2, #T_e253c_row2_col3, #T_e253c_row2_col4, #T_e253c_row2_col5, #T_e253c_row2_col6, #T_e253c_row2_col7, #T_e253c_row3_col0, #T_e253c_row3_col1, #T_e253c_row3_col2, #T_e253c_row3_col3, #T_e253c_row3_col4, #T_e253c_row3_col5, #T_e253c_row3_col6, #T_e253c_row3_col7, #T_e253c_row4_col0, #T_e253c_row4_col1, #T_e253c_row4_col2, #T_e253c_row4_col3, #T_e253c_row4_col4, #T_e253c_row4_col5, #T_e253c_row4_col6, #T_e253c_row4_col7 {\n",
       "  border: 1px solid gray;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_e253c\">\n",
       "  <caption>Measured diffs to reference</caption>\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_e253c_level0_col0\" class=\"col_heading level0 col0\" ></th>\n",
       "      <th id=\"T_e253c_level0_col1\" class=\"col_heading level0 col1\" >compile</th>\n",
       "      <th id=\"T_e253c_level0_col2\" class=\"col_heading level0 col2\" >prepare</th>\n",
       "      <th id=\"T_e253c_level0_col3\" class=\"col_heading level0 col3\" >schedule</th>\n",
       "      <th id=\"T_e253c_level0_col4\" class=\"col_heading level0 col4\" >run</th>\n",
       "      <th id=\"T_e253c_level0_col5\" class=\"col_heading level0 col5\" >process</th>\n",
       "      <th id=\"T_e253c_level0_col6\" class=\"col_heading level0 col6\" >schedule_duration</th>\n",
       "      <th id=\"T_e253c_level0_col7\" class=\"col_heading level0 col7\" >total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_e253c_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_e253c_row0_col0\" class=\"data row0 col0\" >simple_binned_acquisition</td>\n",
       "      <td id=\"T_e253c_row0_col1\" class=\"data row0 col1\" >0 ± 0.074 s</td>\n",
       "      <td id=\"T_e253c_row0_col2\" class=\"data row0 col2\" >0 ± 0.0015 s</td>\n",
       "      <td id=\"T_e253c_row0_col3\" class=\"data row0 col3\" >0 ± 0.081 s</td>\n",
       "      <td id=\"T_e253c_row0_col4\" class=\"data row0 col4\" >0 ± 0.00029 s</td>\n",
       "      <td id=\"T_e253c_row0_col5\" class=\"data row0 col5\" >0 ± 0.00089 s</td>\n",
       "      <td id=\"T_e253c_row0_col6\" class=\"data row0 col6\" >0 ± 2.9e-11 s</td>\n",
       "      <td id=\"T_e253c_row0_col7\" class=\"data row0 col7\" >0 ± 0.067 s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_e253c_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_e253c_row1_col0\" class=\"data row1 col0\" >resonator_spectroscopy</td>\n",
       "      <td id=\"T_e253c_row1_col1\" class=\"data row1 col1\" >0 ± 0.062 s</td>\n",
       "      <td id=\"T_e253c_row1_col2\" class=\"data row1 col2\" >0 ± 0.00072 s</td>\n",
       "      <td id=\"T_e253c_row1_col3\" class=\"data row1 col3\" >0 ± 0.054 s</td>\n",
       "      <td id=\"T_e253c_row1_col4\" class=\"data row1 col4\" >0 ± 3.6e-05 s</td>\n",
       "      <td id=\"T_e253c_row1_col5\" class=\"data row1 col5\" >0 ± 0.00027 s</td>\n",
       "      <td id=\"T_e253c_row1_col6\" class=\"data row1 col6\" >0 ± 9.1e-09 s</td>\n",
       "      <td id=\"T_e253c_row1_col7\" class=\"data row1 col7\" >0 ± 0.082 s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_e253c_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_e253c_row2_col0\" class=\"data row2 col0\" >random_gates</td>\n",
       "      <td id=\"T_e253c_row2_col1\" class=\"data row2 col1\" >0 ± 0.063 s</td>\n",
       "      <td id=\"T_e253c_row2_col2\" class=\"data row2 col2\" >0 ± 0.0013 s</td>\n",
       "      <td id=\"T_e253c_row2_col3\" class=\"data row2 col3\" >0 ± 0.0052 s</td>\n",
       "      <td id=\"T_e253c_row2_col4\" class=\"data row2 col4\" >0 ± 8.4e-06 s</td>\n",
       "      <td id=\"T_e253c_row2_col5\" class=\"data row2 col5\" >0 ± 6.3e-05 s</td>\n",
       "      <td id=\"T_e253c_row2_col6\" class=\"data row2 col6\" >0 ± 6.3e-12 s</td>\n",
       "      <td id=\"T_e253c_row2_col7\" class=\"data row2 col7\" >0 ± 0.072 s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_e253c_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_e253c_row3_col0\" class=\"data row3 col0\" >loops_with_measurements</td>\n",
       "      <td id=\"T_e253c_row3_col1\" class=\"data row3 col1\" >0 ± 0.0096 s</td>\n",
       "      <td id=\"T_e253c_row3_col2\" class=\"data row3 col2\" >0 ± 0.00048 s</td>\n",
       "      <td id=\"T_e253c_row3_col3\" class=\"data row3 col3\" >0 ± 0.001 s</td>\n",
       "      <td id=\"T_e253c_row3_col4\" class=\"data row3 col4\" >0 ± 3.9e-05 s</td>\n",
       "      <td id=\"T_e253c_row3_col5\" class=\"data row3 col5\" >0 ± 7.3e-05 s</td>\n",
       "      <td id=\"T_e253c_row3_col6\" class=\"data row3 col6\" >0 ± 4.1e-07 s</td>\n",
       "      <td id=\"T_e253c_row3_col7\" class=\"data row3 col7\" >0 ± 0.0092 s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_e253c_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_e253c_row4_col0\" class=\"data row4 col0\" >multidim_batched_sweep</td>\n",
       "      <td id=\"T_e253c_row4_col1\" class=\"data row4 col1\" >0 ± 0.1 s</td>\n",
       "      <td id=\"T_e253c_row4_col2\" class=\"data row4 col2\" >0 ± 0.0028 s</td>\n",
       "      <td id=\"T_e253c_row4_col3\" class=\"data row4 col3\" >nan ± nan s</td>\n",
       "      <td id=\"T_e253c_row4_col4\" class=\"data row4 col4\" >0 ± 9.2e-06 s</td>\n",
       "      <td id=\"T_e253c_row4_col5\" class=\"data row4 col5\" >0 ± 0.00027 s</td>\n",
       "      <td id=\"T_e253c_row4_col6\" class=\"data row4 col6\" >0 ± 1.3e-09 s</td>\n",
       "      <td id=\"T_e253c_row4_col7\" class=\"data row4 col7\" >0 ± 0.09 s</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7660f9d3c040>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# All data is (current_time - reference_time).\n",
    "# If the cell is green (or red), the current time\n",
    "# is significantly less (or more) than the reference time.\n",
    "df_diff"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.21"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
