aboutsummaryrefslogtreecommitdiffstats
path: root/README.md
diff options
context:
space:
mode:
authorJoshua Bakita <jbakita@cs.unc.edu>2025-06-16 19:29:07 -0400
committerJoshua Bakita <jbakita@cs.unc.edu>2025-06-17 14:01:49 -0400
commit89177fce34edb5ad0059a41548888d05588cc1c5 (patch)
tree096dc302bb5e17e3987c45a59ef02c69ec73e9ed /README.md
parent03ae77e35d35b2a82f5387d1903cfa954b696edd (diff)
Rewrite nvtaskset and implementation of partitioning for unmodified tasks
Rather than requiring libsmctrl.so to be preloaded, we now wrap libcuda.so.1. All CUDA-using applications will load libcuda.so.1, ensuring that our wrapper will always be dynamically loaded, no matter if LD_PRELOAD is enabled, or if a program has been staticly linked. All that needs to be done is that the location of our "fake" libcuda.so.1 need to be put within the loader search path. This can be done by setting LD_LIBRARY_PATH, or by installing our wrapper into /lib/x86_64-linux-gnu. The mask can still be set via the LIBSMCTRL_MASK environment variable, but the easier-to-use nvtaskset tool is now the recommended way to view or change the supreme TPC mask for any CUDA-using application. This allows launching a program on the first two GPCs via a command as simple as: ./nvtaskset -g 0-1 ./a_program a_program_args (Note that use of the -g option requires the nvdebug kernel module to first be loaded.) These changes support the final version of the ECRTS'25 paper. Note that nvtaskset does not yet fully support multi-GPU systems. Bugfixes: - Fix crash that would occur if both libsmctrl.so and libsmctrl.a were built into an application. - Correctly use GPU ID when initializing a context in `libsmctrl_test_gpc_info`. - Include `nvtaskset` as a prerequisite for `libsmctrl_test_supreme_mask`. - Fix malfunction of `libsmctrl_test_gpc_info` if CUDA_VISIBLE_DEVICES is set. Other minor changes: - Adds make target to run all the tests. - Fixes typos in comments. - Enables -Wall build option. - Upgrades supreme mask from 64 to 128 bits. - Removes `detect_parker_soc()` from the global namespace. - Adjusts test messages to be more succinct. - Updates README with overview of how to partition unmodified applications, more details on the tests, and information on the new ECRTS'25 paper.
Diffstat (limited to 'README.md')
-rw-r--r--README.md83
1 files changed, 78 insertions, 5 deletions
diff --git a/README.md b/README.md
index f2be718..11ad153 100644
--- a/README.md
+++ b/README.md
@@ -13,16 +13,37 @@ Please cite this paper in any work which leverages our library. Here's the BibTe
13 year={2023}, 13 year={2023},
14 month={May}, 14 month={May},
15 pages={54--66}, 15 pages={54--66},
16 doi={10.1109/RTAS58335.2023.00012},
16 _series={RTAS} 17 _series={RTAS}
17} 18}
18``` 19```
19 20
20Please see [the paper](https://www.cs.unc.edu/~jbakita/rtas23.pdf) and `libsmctrl.h` for details and examples of how to use this library. 21The ability for `libsmctrl` to work on unmodified tasks was developed as part of a follow-up paper:
22
23_J. Bakita and J. H. Anderson, "Hardware Compute Partitioning on NVIDIA GPUs for Composable Systems", Proceedings of the 37th Euromicro Conference on Real-Time Systems, pp. 18:1-18:24, July 2025._
24
25Please cite this paper in any work which uses this for partitioning unmodified tasks. Here's the BibTeX entry:
26```
27@inproceedings{bakita2025hardware,
28 title={Hardware Compute Partitioning on {NVIDIA} {GPUs} for Composable Systems},
29 author={Bakita, Joshua and Anderson, James H},
30 booktitle={Proceedings of the 37th Euromicro Conference on Real-Time Systems},
31 year={2025},
32 month={July},
33 pages={18:1--18:24},
34 doi={10.1109/ECRTS.2025.18},
35 _series={ECRTS}
36}
37```
38
39Please see [the first paper](https://www.cs.unc.edu/~jbakita/rtas23.pdf), [the second paper](https://www.cs.unc.edu/~jbakita/ecrts25.pdf) and `libsmctrl.h` for details and examples of how to use this library.
21We strongly encourage consulting those resources first; the below comments serve merely as an appendum. 40We strongly encourage consulting those resources first; the below comments serve merely as an appendum.
22 41
23## Run-time Dependencies 42## Run-time Dependencies
24`libcuda.so`, which is automatically installed by the NVIDIA GPU driver. 43`libcuda.so`, which is automatically installed by the NVIDIA GPU driver.
25 44
45(Technically `libdl` is also required, but this should never need to be manually installed. This is a dependency of CUDA, and is also part of the GNU C Standard Library starting with version 2.34.)
46
26## Building 47## Building
27To build, ensure that you have `gcc` installed and access to the CUDA SDK including `nvcc`. Then run: 48To build, ensure that you have `gcc` installed and access to the CUDA SDK including `nvcc`. Then run:
28``` 49```
@@ -66,8 +87,52 @@ nvcc benchmark.cu -o benchmark -I/playpen/libsmctl -lsmctrl -lcuda -L/playpen/li
66``` 87```
67The resultant `benchmark` binary should be portable to any system with an equivalent or newer version of the NVIDIA GPU driver installed. 88The resultant `benchmark` binary should be portable to any system with an equivalent or newer version of the NVIDIA GPU driver installed.
68 89
90## Use Without Application Modification
91As an alternative to modifying your application, `libsmctrl` can be installed system-wide, and partitions for each application can be set via the `nvtaskset` tool.
92The `nvtaskset` tool works very similarly to the Linux CPU-affinity-setting tool `taskset`.
93
94To install `libsmctrl` system-wide, such that all CUDA-using applications automatically load it, ensure that `patchelf` is installed (`sudo apt install patchelf`), and run:
95```
96make libcuda.so.1 install
97```
98Or, if you do not want to modify any system-wide state, and only want `libsmctrl` loaded as part of anything run from this console:
99```
100make libcuda.so.1
101export LD_LIBRARY_PATH=$(pwd)
102```
103(This works because CUDA is always dynamically loaded from `libcuda.so.1`, and `lbsmctrl` creates a "fake" `libcuda.so.1` in this directory that wraps CUDA.
104 Setting `LD_LIBRARY_PATH` ensures that the wrapped version is the first one loaded.
105 The only difference with running `make install` is that it copies our "fake" `libcuda.so.1` to a location where the loader will automatically find it.)
106
107And then to start an application within a specific TPC partition, e.g., the first 10 TPCs:
108```
109./nvtaskset -t 0-9 my_program my_args
110```
111Note that this will automatically start NVIDIA MPS, which is a prerequisite to co-run tasks on NVIDIA GPUs without timeslicing.
112
113And to change the TPCs available for a process ID 1234 to to the first 10 TPCs:
114```
115./nvtaskset -tp 0-9 1234
116```
117
118Or, to change a process of ID 1234 to only run on GPC 3:
119```
120./nvtaskset -gp 3 1234
121```
122
123To remove the system-wide installation of `libsmctrl`, run:
124```
125make remove
126```
127
69## Run Tests 128## Run Tests
70To test partitioning: 129
130To run them all:
131```
132make run_tests
133```
134
135If you prefer to run them individually, to test partitioning:
71``` 136```
72make tests 137make tests
73./libsmctrl_test_global_mask 138./libsmctrl_test_global_mask
@@ -82,18 +147,26 @@ make tests
82./libsmctrl_test_next_mask_override 147./libsmctrl_test_next_mask_override
83``` 148```
84 149
85And if `nvdebug` has been installed: 150To test that `nvtaskset` can dynamically change the mask of a running program:
86``` 151```
87make tests 152make libsmctrl_test_supreme_mask
153./libsmctrl_test_supreme_mask
154```
155
156To test that TPC to GPC mappings can be obtained (if `nvdebug` has been installed):
157```
158make libsmctrl_test_gpc_info
88./libsmctrl_test_gpc_info 159./libsmctrl_test_gpc_info
89``` 160```
90 161
162The `CUDA_VISIBLE_DEVICES` environment variable can be set to run any of the partitioning tests on a different GPU.
163
91## Supported GPUs 164## Supported GPUs
92 165
93#### Known Working 166#### Known Working
94 167
95- NVIDIA GPUs from compute capability 3.5 through 8.9, including embedded "Jetson" GPUs 168- NVIDIA GPUs from compute capability 3.5 through 8.9, including embedded "Jetson" GPUs
96- CUDA 6.5 through 12.6 169- CUDA 6.5 through 12.8
97- `x86_64` and Jetson `aarch64` platforms 170- `x86_64` and Jetson `aarch64` platforms
98 171
99#### Known Issues 172#### Known Issues