aboutsummaryrefslogtreecommitdiffstats
path: root/README.md
diff options
context:
space:
mode:
Diffstat (limited to 'README.md')
-rw-r--r--README.md83
1 files changed, 78 insertions, 5 deletions
diff --git a/README.md b/README.md
index f2be718..11ad153 100644
--- a/README.md
+++ b/README.md
@@ -13,16 +13,37 @@ Please cite this paper in any work which leverages our library. Here's the BibTe
13 year={2023}, 13 year={2023},
14 month={May}, 14 month={May},
15 pages={54--66}, 15 pages={54--66},
16 doi={10.1109/RTAS58335.2023.00012},
16 _series={RTAS} 17 _series={RTAS}
17} 18}
18``` 19```
19 20
20Please see [the paper](https://www.cs.unc.edu/~jbakita/rtas23.pdf) and `libsmctrl.h` for details and examples of how to use this library. 21The ability for `libsmctrl` to work on unmodified tasks was developed as part of a follow-up paper:
22
23_J. Bakita and J. H. Anderson, "Hardware Compute Partitioning on NVIDIA GPUs for Composable Systems", Proceedings of the 37th Euromicro Conference on Real-Time Systems, pp. 18:1-18:24, July 2025._
24
25Please cite this paper in any work which uses this for partitioning unmodified tasks. Here's the BibTeX entry:
26```
27@inproceedings{bakita2025hardware,
28 title={Hardware Compute Partitioning on {NVIDIA} {GPUs} for Composable Systems},
29 author={Bakita, Joshua and Anderson, James H},
30 booktitle={Proceedings of the 37th Euromicro Conference on Real-Time Systems},
31 year={2025},
32 month={July},
33 pages={18:1--18:24},
34 doi={10.1109/ECRTS.2025.18},
35 _series={ECRTS}
36}
37```
38
39Please see [the first paper](https://www.cs.unc.edu/~jbakita/rtas23.pdf), [the second paper](https://www.cs.unc.edu/~jbakita/ecrts25.pdf) and `libsmctrl.h` for details and examples of how to use this library.
21We strongly encourage consulting those resources first; the below comments serve merely as an appendum. 40We strongly encourage consulting those resources first; the below comments serve merely as an appendum.
22 41
23## Run-time Dependencies 42## Run-time Dependencies
24`libcuda.so`, which is automatically installed by the NVIDIA GPU driver. 43`libcuda.so`, which is automatically installed by the NVIDIA GPU driver.
25 44
45(Technically `libdl` is also required, but this should never need to be manually installed. This is a dependency of CUDA, and is also part of the GNU C Standard Library starting with version 2.34.)
46
26## Building 47## Building
27To build, ensure that you have `gcc` installed and access to the CUDA SDK including `nvcc`. Then run: 48To build, ensure that you have `gcc` installed and access to the CUDA SDK including `nvcc`. Then run:
28``` 49```
@@ -66,8 +87,52 @@ nvcc benchmark.cu -o benchmark -I/playpen/libsmctl -lsmctrl -lcuda -L/playpen/li
66``` 87```
67The resultant `benchmark` binary should be portable to any system with an equivalent or newer version of the NVIDIA GPU driver installed. 88The resultant `benchmark` binary should be portable to any system with an equivalent or newer version of the NVIDIA GPU driver installed.
68 89
90## Use Without Application Modification
91As an alternative to modifying your application, `libsmctrl` can be installed system-wide, and partitions for each application can be set via the `nvtaskset` tool.
92The `nvtaskset` tool works very similarly to the Linux CPU-affinity-setting tool `taskset`.
93
94To install `libsmctrl` system-wide, such that all CUDA-using applications automatically load it, ensure that `patchelf` is installed (`sudo apt install patchelf`), and run:
95```
96make libcuda.so.1 install
97```
98Or, if you do not want to modify any system-wide state, and only want `libsmctrl` loaded as part of anything run from this console:
99```
100make libcuda.so.1
101export LD_LIBRARY_PATH=$(pwd)
102```
103(This works because CUDA is always dynamically loaded from `libcuda.so.1`, and `lbsmctrl` creates a "fake" `libcuda.so.1` in this directory that wraps CUDA.
104 Setting `LD_LIBRARY_PATH` ensures that the wrapped version is the first one loaded.
105 The only difference with running `make install` is that it copies our "fake" `libcuda.so.1` to a location where the loader will automatically find it.)
106
107And then to start an application within a specific TPC partition, e.g., the first 10 TPCs:
108```
109./nvtaskset -t 0-9 my_program my_args
110```
111Note that this will automatically start NVIDIA MPS, which is a prerequisite to co-run tasks on NVIDIA GPUs without timeslicing.
112
113And to change the TPCs available for a process ID 1234 to to the first 10 TPCs:
114```
115./nvtaskset -tp 0-9 1234
116```
117
118Or, to change a process of ID 1234 to only run on GPC 3:
119```
120./nvtaskset -gp 3 1234
121```
122
123To remove the system-wide installation of `libsmctrl`, run:
124```
125make remove
126```
127
69## Run Tests 128## Run Tests
70To test partitioning: 129
130To run them all:
131```
132make run_tests
133```
134
135If you prefer to run them individually, to test partitioning:
71``` 136```
72make tests 137make tests
73./libsmctrl_test_global_mask 138./libsmctrl_test_global_mask
@@ -82,18 +147,26 @@ make tests
82./libsmctrl_test_next_mask_override 147./libsmctrl_test_next_mask_override
83``` 148```
84 149
85And if `nvdebug` has been installed: 150To test that `nvtaskset` can dynamically change the mask of a running program:
86``` 151```
87make tests 152make libsmctrl_test_supreme_mask
153./libsmctrl_test_supreme_mask
154```
155
156To test that TPC to GPC mappings can be obtained (if `nvdebug` has been installed):
157```
158make libsmctrl_test_gpc_info
88./libsmctrl_test_gpc_info 159./libsmctrl_test_gpc_info
89``` 160```
90 161
162The `CUDA_VISIBLE_DEVICES` environment variable can be set to run any of the partitioning tests on a different GPU.
163
91## Supported GPUs 164## Supported GPUs
92 165
93#### Known Working 166#### Known Working
94 167
95- NVIDIA GPUs from compute capability 3.5 through 8.9, including embedded "Jetson" GPUs 168- NVIDIA GPUs from compute capability 3.5 through 8.9, including embedded "Jetson" GPUs
96- CUDA 6.5 through 12.6 169- CUDA 6.5 through 12.8
97- `x86_64` and Jetson `aarch64` platforms 170- `x86_64` and Jetson `aarch64` platforms
98 171
99#### Known Issues 172#### Known Issues