1.. SPDX-License-Identifier: BSD-3-Clause 2 Copyright (c) 2022 Marvell. 3 4dpdk-test-mldev Application 5=========================== 6 7The ``dpdk-test-mldev`` tool is a Data Plane Development Kit (DPDK) application 8that allows testing various mldev use cases. 9This application has a generic framework to add new mldev based test cases 10to verify functionality 11and measure the performance of inference execution on DPDK ML devices. 12 13 14Application and Options 15----------------------- 16 17The application has a number of command line options: 18 19.. code-block:: console 20 21 dpdk-test-mldev [EAL Options] -- [application options] 22 23 24EAL Options 25~~~~~~~~~~~ 26 27The following are the EAL command-line options that can be used 28with the ``dpdk-test-mldev`` application. 29See the DPDK Getting Started Guides for more information on these options. 30 31``-c <COREMASK>`` or ``-l <CORELIST>`` 32 Set the hexadecimal bitmask of the cores to run on. 33 The corelist is a list of cores to use. 34 35``-a <PCI_ID>`` 36 Attach a PCI based ML device. 37 Specific to drivers using a PCI based ML device. 38 39``--vdev <driver>`` 40 Add a virtual mldev device. 41 Specific to drivers using a ML virtual device. 42 43 44Application Options 45~~~~~~~~~~~~~~~~~~~ 46 47The following are the command-line options supported by the test application. 48 49``--test <name>`` 50 Name of the test to execute. 51 ML tests are divided into three groups: Device, Model and Inference tests. 52 Test name should be one of the following supported tests. 53 54 **ML Device Tests** :: 55 56 device_ops 57 58 **ML Model Tests** :: 59 60 model_ops 61 62 **ML Inference Tests** :: 63 64 inference_ordered 65 inference_interleave 66 67``--dev_id <n>`` 68 Set the device ID of the ML device to be used for the test. 69 Default value is ``0``. 70 71``--socket_id <n>`` 72 Set the socket ID of the application resources. 73 Default value is ``SOCKET_ID_ANY``. 74 75``--models <model_list>`` 76 Set the list of model files to be used for the tests. 77 Application expects the ``model_list`` in comma separated form 78 (i.e. ``--models model_A.bin,model_B.bin``). 79 Maximum number of models supported by the test is ``8``. 80 81``--filelist <file_list>`` 82 Set the list of model, input, output and reference files to be used for the tests. 83 Application expects the ``file_list`` to be in comma separated form 84 (i.e. ``--filelist <model,input,output>[,reference]``). 85 86 Multiple filelist entries can be specified when running the tests with multiple models. 87 Both quantized and dequantized outputs are written to the disk. 88 Dequantized output file would have the name specified by the user through ``--filelist`` option. 89 A suffix ``.q`` is appended to quantized output filename. 90 Maximum number of filelist entries supported by the test is ``8``. 91 92``--repetitions <n>`` 93 Set the number of inference repetitions to be executed in the test per each model. 94 Default value is ``1``. 95 96``--burst_size <n>`` 97 Set the burst size to be used when enqueuing / dequeuing inferences. 98 Default value is ``1``. 99 100``--queue_pairs <n>`` 101 Set the number of queue-pairs to be used for inference enqueue and dequeue operations. 102 Default value is ``1``. 103 104``--queue_size <n>`` 105 Set the size of queue-pair to be created for inference enqueue / dequeue operations. 106 Queue size would translate into ``rte_ml_dev_qp_conf::nb_desc`` field during queue-pair creation. 107 Default value is ``1``. 108 109``--batches <n>`` 110 Set the number batches in the input file provided for inference run. 111 When not specified, the test would assume the number of batches 112 is the batch size of the model. 113 114``--debug`` 115 Enable the tests to run in debug mode. 116 117``--help`` 118 Print help message. 119 120 121ML Device Tests 122--------------- 123 124ML device tests are functional tests to validate ML device API. 125Device tests validate the ML device handling configure, close, start and stop APIs. 126 127 128Application Options 129~~~~~~~~~~~~~~~~~~~ 130 131Supported command line options for the ``device_ops`` test are following:: 132 133 --debug 134 --test 135 --dev_id 136 --socket_id 137 --queue_pairs 138 --queue_size 139 140 141DEVICE_OPS Test 142~~~~~~~~~~~~~~~ 143 144Device ops test validates the device configuration and reconfiguration support. 145The test configures ML device based on the options 146``--queue_pairs`` and ``--queue_size`` specified by the user, 147and later reconfigures the ML device with the number of queue pairs and queue size 148based on the maximum specified through the device info. 149 150 151Example 152^^^^^^^ 153 154Command to run ``device_ops`` test: 155 156.. code-block:: console 157 158 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 159 --test=device_ops 160 161Command to run ``device_ops`` test with user options: 162 163.. code-block:: console 164 165 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 166 --test=device_ops --queue_pairs <M> --queue_size <N> 167 168 169ML Model Tests 170-------------- 171 172Model tests are functional tests to validate ML model API. 173Model tests validate the functioning of load, start, stop and unload ML models. 174 175 176Application Options 177~~~~~~~~~~~~~~~~~~~ 178 179Supported command line options for the ``model_ops`` test are following:: 180 181 --debug 182 --test 183 --dev_id 184 --socket_id 185 --models 186 187List of model files to be used for the ``model_ops`` test can be specified 188through the option ``--models <model_list>`` as a comma separated list. 189Maximum number of models supported in the test is ``8``. 190 191.. note:: 192 193 * The ``--models <model_list>`` is a mandatory option for running this test. 194 * Options not supported by the test are ignored if specified. 195 196 197MODEL_OPS Test 198~~~~~~~~~~~~~~ 199 200The test is a collection of multiple sub-tests, 201each with a different order of slow-path operations 202when handling with `N` number of models. 203 204**Sub-test A:** 205executes the sequence of load / start / stop / unload for a model in order, 206followed by next model. 207 208.. _figure_mldev_model_ops_subtest_a: 209 210.. figure:: img/mldev_model_ops_subtest_a.* 211 212 Execution sequence of model_ops subtest A. 213 214**Sub-test B:** 215executes load for all models, followed by a start for all models. 216Upon successful start of all models, stop is invoked for all models followed by unload. 217 218.. _figure_mldev_model_ops_subtest_b: 219 220.. figure:: img/mldev_model_ops_subtest_b.* 221 222 Execution sequence of model_ops subtest B. 223 224**Sub-test C:** 225loads all models, followed by a start and stop of all models in order. 226Upon completion of stop, unload is invoked for all models. 227 228.. _figure_mldev_model_ops_subtest_c: 229 230.. figure:: img/mldev_model_ops_subtest_c.* 231 232 Execution sequence of model_ops subtest C. 233 234**Sub-test D:** 235executes load and start for all models available. 236Upon successful start of all models, stop is executed for the models. 237 238.. _figure_mldev_model_ops_subtest_d: 239 240.. figure:: img/mldev_model_ops_subtest_d.* 241 242 Execution sequence of model_ops subtest D. 243 244 245Example 246^^^^^^^ 247 248Command to run ``model_ops`` test: 249 250.. code-block:: console 251 252 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 253 --test=model_ops --models model_1.bin,model_2.bin,model_3.bin, model_4.bin 254 255 256ML Inference Tests 257------------------ 258 259Inference tests are a set of tests to validate end-to-end inference execution on ML device. 260These tests executes the full sequence of operations required to run inferences 261with one or multiple models. 262 263 264Application Options 265~~~~~~~~~~~~~~~~~~~ 266 267Supported command line options for inference tests are following:: 268 269 --debug 270 --test 271 --dev_id 272 --socket_id 273 --filelist 274 --repetitions 275 --burst_size 276 --queue_pairs 277 --queue_size 278 --batches 279 280List of files to be used for the inference tests can be specified 281through the option ``--filelist <file_list>`` as a comma separated list. 282A filelist entry would be of the format 283``--filelist <model_file,input_file,output_file>[,reference_file]`` 284and is used to specify the list of files required to test with a single model. 285Multiple filelist entries are supported by the test, one entry per model. 286Maximum number of file entries supported by the test is ``8``. 287 288When ``--burst_size <num>`` option is specified for the test, 289enqueue and dequeue burst would try to enqueue or dequeue 290``num`` number of inferences per each call respectively. 291 292In the inference test, a pair of lcores are mapped to each queue pair. 293Minimum number of lcores required for the tests is equal to ``(queue_pairs * 2 + 1)``. 294 295.. note:: 296 297 * The ``--filelist <file_list>`` is a mandatory option for running inference tests. 298 * Options not supported by the tests are ignored if specified. 299 300 301INFERENCE_ORDERED Test 302~~~~~~~~~~~~~~~~~~~~~~ 303 304This is a functional test for validating the end-to-end inference execution on ML device. 305This test configures ML device and queue pairs 306as per the queue-pair related options (queue_pairs and queue_size) specified by the user. 307Upon successful configuration of the device and queue pairs, 308the first model specified through the filelist is loaded to the device 309and inferences are enqueued by a pool of worker threads to the ML device. 310Total number of inferences enqueued for the model are equal to the repetitions specified. 311A dedicated pool of worker threads would dequeue the inferences from the device. 312The model is unloaded upon completion of all inferences for the model. 313The test would continue loading and executing inference requests for all models 314specified through ``filelist`` option in an ordered manner. 315 316.. _figure_mldev_inference_ordered: 317 318.. figure:: img/mldev_inference_ordered.* 319 320 Execution of inference_ordered on single model. 321 322 323Example 324^^^^^^^ 325 326Example command to run ``inference_ordered`` test: 327 328.. code-block:: console 329 330 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 331 --test=inference_ordered --filelist model.bin,input.bin,output.bin 332 333Example command to run ``inference_ordered`` test with a specific burst size: 334 335.. code-block:: console 336 337 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 338 --test=inference_ordered --filelist model.bin,input.bin,output.bin \ 339 --burst_size 12 340 341Example command to run ``inference_ordered`` test with multiple queue-pairs and queue size: 342 343.. code-block:: console 344 345 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 346 --test=inference_ordered --filelist model.bin,input.bin,output.bin \ 347 --queue_pairs 4 --queue_size 16 348 349 350INFERENCE_INTERLEAVE Test 351~~~~~~~~~~~~~~~~~~~~~~~~~ 352 353This is a stress test for validating the end-to-end inference execution on ML device. 354The test configures the ML device and queue pairs 355as per the queue-pair related options (queue_pairs and queue_size) specified by the user. 356Upon successful configuration of the device and queue pairs, 357all models specified through the filelist are loaded to the device. 358Inferences for multiple models are enqueued by a pool of worker threads in parallel. 359Inference execution by the device is interleaved between multiple models. 360Total number of inferences enqueued for a model are equal to the repetitions specified. 361An additional pool of threads would dequeue the inferences from the device. 362Models would be unloaded upon completion of inferences for all models loaded. 363 364.. _figure_mldev_inference_interleave: 365 366.. figure:: img/mldev_inference_interleave.* 367 368 Execution of inference_interleave on single model. 369 370 371Example 372^^^^^^^ 373 374Example command to run ``inference_interleave`` test: 375 376.. code-block:: console 377 378 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 379 --test=inference_interleave --filelist model.bin,input.bin,output.bin 380 381Example command to run ``inference_interleave`` test with multiple models: 382 383.. code-block:: console 384 385 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 386 --test=inference_interleave --filelist model_A.bin,input_A.bin,output_A.bin \ 387 --filelist model_B.bin,input_B.bin,output_B.bin 388 389Example command to run ``inference_interleave`` test 390with a specific burst size, multiple queue-pairs and queue size: 391 392.. code-block:: console 393 394 sudo <build_dir>/app/dpdk-test-mldev -c 0xf -a <PCI_ID> -- \ 395 --test=inference_interleave --filelist model.bin,input.bin,output.bin \ 396 --queue_pairs 8 --queue_size 12 --burst_size 16 397 398 399Debug mode 400---------- 401 402ML tests can be executed in debug mode by enabling the option ``--debug``. 403Execution of tests in debug mode would enable additional prints. 404