diff options
| author | Joshua Bakita <jbakita@cs.unc.edu> | 2025-06-16 19:29:07 -0400 |
|---|---|---|
| committer | Joshua Bakita <jbakita@cs.unc.edu> | 2025-06-17 14:01:49 -0400 |
| commit | 89177fce34edb5ad0059a41548888d05588cc1c5 (patch) | |
| tree | 096dc302bb5e17e3987c45a59ef02c69ec73e9ed /README.md | |
| parent | 03ae77e35d35b2a82f5387d1903cfa954b696edd (diff) | |
Rewrite nvtaskset and implementation of partitioning for unmodified tasks
Rather than requiring libsmctrl.so to be preloaded, we now wrap
libcuda.so.1. All CUDA-using applications will load libcuda.so.1,
ensuring that our wrapper will always be dynamically loaded, no
matter if LD_PRELOAD is enabled, or if a program has been staticly
linked. All that needs to be done is that the location of our
"fake" libcuda.so.1 need to be put within the loader search path.
This can be done by setting LD_LIBRARY_PATH, or by installing
our wrapper into /lib/x86_64-linux-gnu.
The mask can still be set via the LIBSMCTRL_MASK environment
variable, but the easier-to-use nvtaskset tool is now the
recommended way to view or change the supreme TPC mask for any
CUDA-using application. This allows launching a program on the
first two GPCs via a command as simple as:
./nvtaskset -g 0-1 ./a_program a_program_args
(Note that use of the -g option requires the nvdebug kernel module
to first be loaded.)
These changes support the final version of the ECRTS'25 paper.
Note that nvtaskset does not yet fully support multi-GPU systems.
Bugfixes:
- Fix crash that would occur if both libsmctrl.so and libsmctrl.a
were built into an application.
- Correctly use GPU ID when initializing a context in
`libsmctrl_test_gpc_info`.
- Include `nvtaskset` as a prerequisite for
`libsmctrl_test_supreme_mask`.
- Fix malfunction of `libsmctrl_test_gpc_info` if
CUDA_VISIBLE_DEVICES is set.
Other minor changes:
- Adds make target to run all the tests.
- Fixes typos in comments.
- Enables -Wall build option.
- Upgrades supreme mask from 64 to 128 bits.
- Removes `detect_parker_soc()` from the global namespace.
- Adjusts test messages to be more succinct.
- Updates README with overview of how to partition unmodified
applications, more details on the tests, and information on the
new ECRTS'25 paper.
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 83 |
1 files changed, 78 insertions, 5 deletions
| @@ -13,16 +13,37 @@ Please cite this paper in any work which leverages our library. Here's the BibTe | |||
| 13 | year={2023}, | 13 | year={2023}, |
| 14 | month={May}, | 14 | month={May}, |
| 15 | pages={54--66}, | 15 | pages={54--66}, |
| 16 | doi={10.1109/RTAS58335.2023.00012}, | ||
| 16 | _series={RTAS} | 17 | _series={RTAS} |
| 17 | } | 18 | } |
| 18 | ``` | 19 | ``` |
| 19 | 20 | ||
| 20 | Please see [the paper](https://www.cs.unc.edu/~jbakita/rtas23.pdf) and `libsmctrl.h` for details and examples of how to use this library. | 21 | The ability for `libsmctrl` to work on unmodified tasks was developed as part of a follow-up paper: |
| 22 | |||
| 23 | _J. Bakita and J. H. Anderson, "Hardware Compute Partitioning on NVIDIA GPUs for Composable Systems", Proceedings of the 37th Euromicro Conference on Real-Time Systems, pp. 18:1-18:24, July 2025._ | ||
| 24 | |||
| 25 | Please cite this paper in any work which uses this for partitioning unmodified tasks. Here's the BibTeX entry: | ||
| 26 | ``` | ||
| 27 | @inproceedings{bakita2025hardware, | ||
| 28 | title={Hardware Compute Partitioning on {NVIDIA} {GPUs} for Composable Systems}, | ||
| 29 | author={Bakita, Joshua and Anderson, James H}, | ||
| 30 | booktitle={Proceedings of the 37th Euromicro Conference on Real-Time Systems}, | ||
| 31 | year={2025}, | ||
| 32 | month={July}, | ||
| 33 | pages={18:1--18:24}, | ||
| 34 | doi={10.1109/ECRTS.2025.18}, | ||
| 35 | _series={ECRTS} | ||
| 36 | } | ||
| 37 | ``` | ||
| 38 | |||
| 39 | Please see [the first paper](https://www.cs.unc.edu/~jbakita/rtas23.pdf), [the second paper](https://www.cs.unc.edu/~jbakita/ecrts25.pdf) and `libsmctrl.h` for details and examples of how to use this library. | ||
| 21 | We strongly encourage consulting those resources first; the below comments serve merely as an appendum. | 40 | We strongly encourage consulting those resources first; the below comments serve merely as an appendum. |
| 22 | 41 | ||
| 23 | ## Run-time Dependencies | 42 | ## Run-time Dependencies |
| 24 | `libcuda.so`, which is automatically installed by the NVIDIA GPU driver. | 43 | `libcuda.so`, which is automatically installed by the NVIDIA GPU driver. |
| 25 | 44 | ||
| 45 | (Technically `libdl` is also required, but this should never need to be manually installed. This is a dependency of CUDA, and is also part of the GNU C Standard Library starting with version 2.34.) | ||
| 46 | |||
| 26 | ## Building | 47 | ## Building |
| 27 | To build, ensure that you have `gcc` installed and access to the CUDA SDK including `nvcc`. Then run: | 48 | To build, ensure that you have `gcc` installed and access to the CUDA SDK including `nvcc`. Then run: |
| 28 | ``` | 49 | ``` |
| @@ -66,8 +87,52 @@ nvcc benchmark.cu -o benchmark -I/playpen/libsmctl -lsmctrl -lcuda -L/playpen/li | |||
| 66 | ``` | 87 | ``` |
| 67 | The resultant `benchmark` binary should be portable to any system with an equivalent or newer version of the NVIDIA GPU driver installed. | 88 | The resultant `benchmark` binary should be portable to any system with an equivalent or newer version of the NVIDIA GPU driver installed. |
| 68 | 89 | ||
| 90 | ## Use Without Application Modification | ||
| 91 | As an alternative to modifying your application, `libsmctrl` can be installed system-wide, and partitions for each application can be set via the `nvtaskset` tool. | ||
| 92 | The `nvtaskset` tool works very similarly to the Linux CPU-affinity-setting tool `taskset`. | ||
| 93 | |||
| 94 | To install `libsmctrl` system-wide, such that all CUDA-using applications automatically load it, ensure that `patchelf` is installed (`sudo apt install patchelf`), and run: | ||
| 95 | ``` | ||
| 96 | make libcuda.so.1 install | ||
| 97 | ``` | ||
| 98 | Or, if you do not want to modify any system-wide state, and only want `libsmctrl` loaded as part of anything run from this console: | ||
| 99 | ``` | ||
| 100 | make libcuda.so.1 | ||
| 101 | export LD_LIBRARY_PATH=$(pwd) | ||
| 102 | ``` | ||
| 103 | (This works because CUDA is always dynamically loaded from `libcuda.so.1`, and `lbsmctrl` creates a "fake" `libcuda.so.1` in this directory that wraps CUDA. | ||
| 104 | Setting `LD_LIBRARY_PATH` ensures that the wrapped version is the first one loaded. | ||
| 105 | The only difference with running `make install` is that it copies our "fake" `libcuda.so.1` to a location where the loader will automatically find it.) | ||
| 106 | |||
| 107 | And then to start an application within a specific TPC partition, e.g., the first 10 TPCs: | ||
| 108 | ``` | ||
| 109 | ./nvtaskset -t 0-9 my_program my_args | ||
| 110 | ``` | ||
| 111 | Note that this will automatically start NVIDIA MPS, which is a prerequisite to co-run tasks on NVIDIA GPUs without timeslicing. | ||
| 112 | |||
| 113 | And to change the TPCs available for a process ID 1234 to to the first 10 TPCs: | ||
| 114 | ``` | ||
| 115 | ./nvtaskset -tp 0-9 1234 | ||
| 116 | ``` | ||
| 117 | |||
| 118 | Or, to change a process of ID 1234 to only run on GPC 3: | ||
| 119 | ``` | ||
| 120 | ./nvtaskset -gp 3 1234 | ||
| 121 | ``` | ||
| 122 | |||
| 123 | To remove the system-wide installation of `libsmctrl`, run: | ||
| 124 | ``` | ||
| 125 | make remove | ||
| 126 | ``` | ||
| 127 | |||
| 69 | ## Run Tests | 128 | ## Run Tests |
| 70 | To test partitioning: | 129 | |
| 130 | To run them all: | ||
| 131 | ``` | ||
| 132 | make run_tests | ||
| 133 | ``` | ||
| 134 | |||
| 135 | If you prefer to run them individually, to test partitioning: | ||
| 71 | ``` | 136 | ``` |
| 72 | make tests | 137 | make tests |
| 73 | ./libsmctrl_test_global_mask | 138 | ./libsmctrl_test_global_mask |
| @@ -82,18 +147,26 @@ make tests | |||
| 82 | ./libsmctrl_test_next_mask_override | 147 | ./libsmctrl_test_next_mask_override |
| 83 | ``` | 148 | ``` |
| 84 | 149 | ||
| 85 | And if `nvdebug` has been installed: | 150 | To test that `nvtaskset` can dynamically change the mask of a running program: |
| 86 | ``` | 151 | ``` |
| 87 | make tests | 152 | make libsmctrl_test_supreme_mask |
| 153 | ./libsmctrl_test_supreme_mask | ||
| 154 | ``` | ||
| 155 | |||
| 156 | To test that TPC to GPC mappings can be obtained (if `nvdebug` has been installed): | ||
| 157 | ``` | ||
| 158 | make libsmctrl_test_gpc_info | ||
| 88 | ./libsmctrl_test_gpc_info | 159 | ./libsmctrl_test_gpc_info |
| 89 | ``` | 160 | ``` |
| 90 | 161 | ||
| 162 | The `CUDA_VISIBLE_DEVICES` environment variable can be set to run any of the partitioning tests on a different GPU. | ||
| 163 | |||
| 91 | ## Supported GPUs | 164 | ## Supported GPUs |
| 92 | 165 | ||
| 93 | #### Known Working | 166 | #### Known Working |
| 94 | 167 | ||
| 95 | - NVIDIA GPUs from compute capability 3.5 through 8.9, including embedded "Jetson" GPUs | 168 | - NVIDIA GPUs from compute capability 3.5 through 8.9, including embedded "Jetson" GPUs |
| 96 | - CUDA 6.5 through 12.6 | 169 | - CUDA 6.5 through 12.8 |
| 97 | - `x86_64` and Jetson `aarch64` platforms | 170 | - `x86_64` and Jetson `aarch64` platforms |
| 98 | 171 | ||
| 99 | #### Known Issues | 172 | #### Known Issues |
