1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright (c) 2022 Marvell. 3 4dpdk-test-mldev Application 5=========================== 6 7The ``dpdk-test-mldev`` tool is a Data Plane Development Kit (DPDK) application 8that allows testing various mldev use cases. 9This application has a generic framework to add new mldev based test cases 10to verify functionality 11and measure the performance of inference execution on DPDK ML devices. 12 13 14Application and Options 15----------------------- 16 17The application has a number of command line options: 18 19.. code-block:: console 20 21 dpdk-test-mldev [EAL Options] -- [application options] 22 23 24EAL Options 25~~~~~~~~~~~ 26 27The following are the EAL command-line options that can be used 28with the ``dpdk-test-mldev`` application. 29See the DPDK Getting Started Guides for more information on these options. 30 31``-c <COREMASK>`` or ``-l <CORELIST>`` 32 Set the hexadecimal bitmask of the cores to run on. 33 The corelist is a list of cores to use. 34 35``-a <PCI_ID>`` 36 Attach a PCI based ML device. 37 Specific to drivers using a PCI based ML device. 38 39``--vdev <driver>`` 40 Add a virtual mldev device. 41 Specific to drivers using a ML virtual device. 42 43 44Application Options 45~~~~~~~~~~~~~~~~~~~ 46 47The following are the command-line options supported by the test application. 48 49``--test <name>`` 50 Name of the test to execute. 51 ML tests are divided into three groups: Device, Model and Inference tests. 52 Test name should be one of the following supported tests. 53 54 **ML Device Tests** :: 55 56 device_ops 57 58 **ML Model Tests** :: 59 60 model_ops 61 62 **ML Inference Tests** :: 63 64 inference_ordered 65 inference_interleave 66 67``--dev_id <n>`` 68 Set the device ID of the ML device to be used for the test. 69 Default value is ``0``. 70 71``--socket_id <n>`` 72 Set the socket ID of the application resources. 73 Default value is ``SOCKET_ID_ANY``. 74 75``--models <model_list>`` 76 Set the list of model files to be used for the tests. 77 Application expects the ``model_list`` in comma separated form 78 (i.e. ``--models model_A.bin,model_B.bin``). 79 Maximum number of models supported by the test is ``8``. 80 81``--filelist <file_list>`` 82 Set the list of model, input, output and reference files to be used for the tests. 83 Application expects the ``file_list`` to be in comma separated form 84 (i.e. ``--filelist <model,input,output>[,reference]``). 85 86 Multiple filelist entries can be specified when running the tests with multiple models. 87 Both quantized and dequantized outputs are written to the disk. 88 Dequantized output file would have the name specified by the user through ``--filelist`` option. 89 A suffix ``.q`` is appended to quantized output filename. 90 Maximum number of filelist entries supported by the test is ``8``. 91 92``--repetitions <n>`` 93 Set the number of inference repetitions to be executed in the test per each model. 94 Default value is ``1``. 95 96``--burst_size <n>`` 97 Set the burst size to be used when enqueuing / dequeuing inferences. 98 Default value is ``1``. 99 100``--queue_pairs <n>`` 101 Set the number of queue-pairs to be used for inference enqueue and dequeue operations. 102 Default value is ``1``. 103 104``--queue_size <n>`` 105 Set the size of queue-pair to be created for inference enqueue / dequeue operations. 106 Queue size would translate into ``rte_ml_dev_qp_conf::nb_desc`` field during queue-pair creation. 107 Default value is ``1``. 108 109``--debug`` 110 Enable the tests to run in debug mode. 111 112``--help`` 113 Print help message. 114 115 116ML Device Tests 117--------------- 118 119ML device tests are functional tests to validate ML device API. 120Device tests validate the ML device handling configure, close, start and stop APIs. 121 122 123Application Options 124~~~~~~~~~~~~~~~~~~~ 125 126Supported command line options for the ``device_ops`` test are following:: 127 128 --debug 129 --test 130 --dev_id 131 --socket_id 132 --queue_pairs 133 --queue_size 134 135 136DEVICE_OPS Test 137~~~~~~~~~~~~~~~ 138 139Device ops test validates the device configuration and reconfiguration support. 140The test configures ML device based on the options 141``--queue_pairs`` and ``--queue_size`` specified by the user, 142and later reconfigures the ML device with the number of queue pairs and queue size 143based on the maximum specified through the device info. 144 145 146Example 147^^^^^^^ 148 149Command to run ``device_ops`` test: 150 151.. code-block:: console 152 153 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 154 --test=device_ops 155 156Command to run ``device_ops`` test with user options: 157 158.. code-block:: console 159 160 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 161 --test=device_ops --queue_pairs <M> --queue_size <N> 162 163 164ML Model Tests 165-------------- 166 167Model tests are functional tests to validate ML model API. 168Model tests validate the functioning of load, start, stop and unload ML models. 169 170 171Application Options 172~~~~~~~~~~~~~~~~~~~ 173 174Supported command line options for the ``model_ops`` test are following:: 175 176 --debug 177 --test 178 --dev_id 179 --socket_id 180 --models 181 182List of model files to be used for the ``model_ops`` test can be specified 183through the option ``--models <model_list>`` as a comma separated list. 184Maximum number of models supported in the test is ``8``. 185 186.. note:: 187 188 * The ``--models <model_list>`` is a mandatory option for running this test. 189 * Options not supported by the test are ignored if specified. 190 191 192MODEL_OPS Test 193~~~~~~~~~~~~~~ 194 195The test is a collection of multiple sub-tests, 196each with a different order of slow-path operations 197when handling with `N` number of models. 198 199**Sub-test A:** 200executes the sequence of load / start / stop / unload for a model in order, 201followed by next model. 202 203.. _figure_mldev_model_ops_subtest_a: 204 205.. figure:: img/mldev_model_ops_subtest_a.* 206 207 Execution sequence of model_ops subtest A. 208 209**Sub-test B:** 210executes load for all models, followed by a start for all models. 211Upon successful start of all models, stop is invoked for all models followed by unload. 212 213.. _figure_mldev_model_ops_subtest_b: 214 215.. figure:: img/mldev_model_ops_subtest_b.* 216 217 Execution sequence of model_ops subtest B. 218 219**Sub-test C:** 220loads all models, followed by a start and stop of all models in order. 221Upon completion of stop, unload is invoked for all models. 222 223.. _figure_mldev_model_ops_subtest_c: 224 225.. figure:: img/mldev_model_ops_subtest_c.* 226 227 Execution sequence of model_ops subtest C. 228 229**Sub-test D:** 230executes load and start for all models available. 231Upon successful start of all models, stop is executed for the models. 232 233.. _figure_mldev_model_ops_subtest_d: 234 235.. figure:: img/mldev_model_ops_subtest_d.* 236 237 Execution sequence of model_ops subtest D. 238 239 240Example 241^^^^^^^ 242 243Command to run ``model_ops`` test: 244 245.. code-block:: console 246 247 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 248 --test=model_ops --models model_1.bin,model_2.bin,model_3.bin, model_4.bin 249 250 251ML Inference Tests 252------------------ 253 254Inference tests are a set of tests to validate end-to-end inference execution on ML device. 255These tests executes the full sequence of operations required to run inferences 256with one or multiple models. 257 258 259Application Options 260~~~~~~~~~~~~~~~~~~~ 261 262Supported command line options for inference tests are following:: 263 264 --debug 265 --test 266 --dev_id 267 --socket_id 268 --filelist 269 --repetitions 270 --burst_size 271 --queue_pairs 272 --queue_size 273 274List of files to be used for the inference tests can be specified 275through the option ``--filelist <file_list>`` as a comma separated list. 276A filelist entry would be of the format 277``--filelist <model_file,input_file,output_file>[,reference_file]`` 278and is used to specify the list of files required to test with a single model. 279Multiple filelist entries are supported by the test, one entry per model. 280Maximum number of file entries supported by the test is ``8``. 281 282When ``--burst_size <num>`` option is specified for the test, 283enqueue and dequeue burst would try to enqueue or dequeue 284``num`` number of inferences per each call respectively. 285 286In the inference test, a pair of lcores are mapped to each queue pair. 287Minimum number of lcores required for the tests is equal to ``(queue_pairs * 2 + 1)``. 288 289.. note:: 290 291 * The ``--filelist <file_list>`` is a mandatory option for running inference tests. 292 * Options not supported by the tests are ignored if specified. 293 294 295INFERENCE_ORDERED Test 296~~~~~~~~~~~~~~~~~~~~~~ 297 298This is a functional test for validating the end-to-end inference execution on ML device. 299This test configures ML device and queue pairs 300as per the queue-pair related options (queue_pairs and queue_size) specified by the user. 301Upon successful configuration of the device and queue pairs, 302the first model specified through the filelist is loaded to the device 303and inferences are enqueued by a pool of worker threads to the ML device. 304Total number of inferences enqueued for the model are equal to the repetitions specified. 305A dedicated pool of worker threads would dequeue the inferences from the device. 306The model is unloaded upon completion of all inferences for the model. 307The test would continue loading and executing inference requests for all models 308specified through ``filelist`` option in an ordered manner. 309 310.. _figure_mldev_inference_ordered: 311 312.. figure:: img/mldev_inference_ordered.* 313 314 Execution of inference_ordered on single model. 315 316 317Example 318^^^^^^^ 319 320Example command to run ``inference_ordered`` test: 321 322.. code-block:: console 323 324 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 325 --test=inference_ordered --filelist model.bin,input.bin,output.bin 326 327Example command to run ``inference_ordered`` test with a specific burst size: 328 329.. code-block:: console 330 331 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 332 --test=inference_ordered --filelist model.bin,input.bin,output.bin \ 333 --burst_size 12 334 335Example command to run ``inference_ordered`` test with multiple queue-pairs and queue size: 336 337.. code-block:: console 338 339 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 340 --test=inference_ordered --filelist model.bin,input.bin,output.bin \ 341 --queue_pairs 4 --queue_size 16 342 343 344INFERENCE_INTERLEAVE Test 345~~~~~~~~~~~~~~~~~~~~~~~~~ 346 347This is a stress test for validating the end-to-end inference execution on ML device. 348The test configures the ML device and queue pairs 349as per the queue-pair related options (queue_pairs and queue_size) specified by the user. 350Upon successful configuration of the device and queue pairs, 351all models specified through the filelist are loaded to the device. 352Inferences for multiple models are enqueued by a pool of worker threads in parallel. 353Inference execution by the device is interleaved between multiple models. 354Total number of inferences enqueued for a model are equal to the repetitions specified. 355An additional pool of threads would dequeue the inferences from the device. 356Models would be unloaded upon completion of inferences for all models loaded. 357 358.. _figure_mldev_inference_interleave: 359 360.. figure:: img/mldev_inference_interleave.* 361 362 Execution of inference_interleave on single model. 363 364 365Example 366^^^^^^^ 367 368Example command to run ``inference_interleave`` test: 369 370.. code-block:: console 371 372 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 373 --test=inference_interleave --filelist model.bin,input.bin,output.bin 374 375Example command to run ``inference_interleave`` test with multiple models: 376 377.. code-block:: console 378 379 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 380 --test=inference_interleave --filelist model_A.bin,input_A.bin,output_A.bin \ 381 --filelist model_B.bin,input_B.bin,output_B.bin 382 383Example command to run ``inference_interleave`` test 384with a specific burst size, multiple queue-pairs and queue size: 385 386.. code-block:: console 387 388 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 389 --test=inference_interleave --filelist model.bin,input.bin,output.bin \ 390 --queue_pairs 8 --queue_size 12 --burst_size 16 391 392 393Debug mode 394---------- 395 396ML tests can be executed in debug mode by enabling the option ``--debug``. 397Execution of tests in debug mode would enable additional prints. 398