aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorJoshua Bakita <jbakita@cs.unc.edu>2025-06-16 19:29:07 -0400
committerJoshua Bakita <jbakita@cs.unc.edu>2025-06-17 14:01:49 -0400
commit89177fce34edb5ad0059a41548888d05588cc1c5 (patch)
tree096dc302bb5e17e3987c45a59ef02c69ec73e9ed
parent03ae77e35d35b2a82f5387d1903cfa954b696edd (diff)
Rewrite nvtaskset and implementation of partitioning for unmodified tasks
Rather than requiring libsmctrl.so to be preloaded, we now wrap libcuda.so.1. All CUDA-using applications will load libcuda.so.1, ensuring that our wrapper will always be dynamically loaded, no matter if LD_PRELOAD is enabled, or if a program has been staticly linked. All that needs to be done is that the location of our "fake" libcuda.so.1 need to be put within the loader search path. This can be done by setting LD_LIBRARY_PATH, or by installing our wrapper into /lib/x86_64-linux-gnu. The mask can still be set via the LIBSMCTRL_MASK environment variable, but the easier-to-use nvtaskset tool is now the recommended way to view or change the supreme TPC mask for any CUDA-using application. This allows launching a program on the first two GPCs via a command as simple as: ./nvtaskset -g 0-1 ./a_program a_program_args (Note that use of the -g option requires the nvdebug kernel module to first be loaded.) These changes support the final version of the ECRTS'25 paper. Note that nvtaskset does not yet fully support multi-GPU systems. Bugfixes: - Fix crash that would occur if both libsmctrl.so and libsmctrl.a were built into an application. - Correctly use GPU ID when initializing a context in `libsmctrl_test_gpc_info`. - Include `nvtaskset` as a prerequisite for `libsmctrl_test_supreme_mask`. - Fix malfunction of `libsmctrl_test_gpc_info` if CUDA_VISIBLE_DEVICES is set. Other minor changes: - Adds make target to run all the tests. - Fixes typos in comments. - Enables -Wall build option. - Upgrades supreme mask from 64 to 128 bits. - Removes `detect_parker_soc()` from the global namespace. - Adjusts test messages to be more succinct. - Updates README with overview of how to partition unmodified applications, more details on the tests, and information on the new ECRTS'25 paper.
-rw-r--r--.gitignore2
-rw-r--r--Makefile47
-rw-r--r--README.md83
-rw-r--r--libsmctrl.c212
-rw-r--r--libsmctrl_test_gpc_info.c4
-rw-r--r--libsmctrl_test_mask_shared.cu31
-rw-r--r--nvtaskset.c520
7 files changed, 659 insertions, 240 deletions
diff --git a/.gitignore b/.gitignore
index 5f0fdbe..c42b364 100644
--- a/.gitignore
+++ b/.gitignore
@@ -8,6 +8,8 @@ libsmctrl_test_stream_mask
8libsmctrl_test_stream_mask_override 8libsmctrl_test_stream_mask_override
9libsmctrl_test_next_mask 9libsmctrl_test_next_mask
10libsmctrl_test_next_mask_override 10libsmctrl_test_next_mask_override
11libcuda.so.1
12nvtaskset
11*.pyc 13*.pyc
12*.o 14*.o
13.gdb_history 15.gdb_history
diff --git a/Makefile b/Makefile
index 62ec245..87a5708 100644
--- a/Makefile
+++ b/Makefile
@@ -3,9 +3,11 @@ CUDA ?= /usr/local/cuda
3# Note that CXX and CC are predefined as g++ and cc (respectively) by Make 3# Note that CXX and CC are predefined as g++ and cc (respectively) by Make
4NVCC ?= $(CUDA)/bin/nvcc 4NVCC ?= $(CUDA)/bin/nvcc
5# Everything has to have -lcuda, as it's needed for libsmctrl 5# Everything has to have -lcuda, as it's needed for libsmctrl
6LDFLAGS := -lcuda -I$(CUDA)/include -L$(CUDA)/lib64 6LDFLAGS := -ldl -lcuda -I$(CUDA)/include -L$(CUDA)/lib64
7ARCH = $(shell $(CC) -dumpmachine)
8CFLAGS := -Wall -Wno-parentheses
7 9
8.PHONY: clean tests all 10.PHONY: clean tests all install remove run_tests
9 11
10# ----- Main Library ----- 12# ----- Main Library -----
11libsmctrl.so: libsmctrl.c libsmctrl.h 13libsmctrl.so: libsmctrl.c libsmctrl.h
@@ -14,9 +16,14 @@ libsmctrl.so: libsmctrl.c libsmctrl.h
14# -fPIC is needed even if built as a static library, in case we are linked into 16# -fPIC is needed even if built as a static library, in case we are linked into
15# another shared library 17# another shared library
16libsmctrl.a: libsmctrl.c libsmctrl.h 18libsmctrl.a: libsmctrl.c libsmctrl.h
17 $(CC) $< -c -o libsmctrl.o -fPIC $(CFLAGS) $(LDFLAGS) 19 $(CC) $< -c -o libsmctrl.o -fPIC -DLIBSMCTRL_STATIC $(CFLAGS) $(LDFLAGS)
18 ar rcs $@ libsmctrl.o 20 ar rcs $@ libsmctrl.o
19 21
22# ----- CUDA Wrapper -----
23libcuda.so.1: libsmctrl.c libsmctrl.h
24 $(CC) $< -shared -o $@ -fPIC -DLIBSMCTRL_WRAPPER $(CFLAGS) $(LDFLAGS)
25 patchelf libcuda.so.1 --add-needed libcuda.so
26
20# ----- Utilities ----- 27# ----- Utilities -----
21# Use static linking with tests to avoid LD_LIBRARY_PATH issues 28# Use static linking with tests to avoid LD_LIBRARY_PATH issues
22nvtaskset: nvtaskset.c libsmctrl.so libsmctrl.a 29nvtaskset: nvtaskset.c libsmctrl.so libsmctrl.a
@@ -29,7 +36,7 @@ libsmctrl_test_gpc_info: libsmctrl_test_gpc_info.c libsmctrl.a testbench.h
29libsmctrl_test_mask_shared.o: libsmctrl_test_mask_shared.cu testbench.h 36libsmctrl_test_mask_shared.o: libsmctrl_test_mask_shared.cu testbench.h
30 $(NVCC) -ccbin $(CXX) $< -c -g 37 $(NVCC) -ccbin $(CXX) $< -c -g
31 38
32libsmctrl_test_supreme_mask: libsmctrl_test_supreme_mask.c libsmctrl.a libsmctrl_test_mask_shared.o 39libsmctrl_test_supreme_mask: libsmctrl_test_supreme_mask.c libsmctrl.a libsmctrl_test_mask_shared.o libcuda.so.1 nvtaskset
33 $(NVCC) -ccbin $(CXX) $@.c -o $@ libsmctrl_test_mask_shared.o -g -L. -l:libsmctrl.a $(LDFLAGS) 40 $(NVCC) -ccbin $(CXX) $@.c -o $@ libsmctrl_test_mask_shared.o -g -L. -l:libsmctrl.a $(LDFLAGS)
34 41
35libsmctrl_test_global_mask: libsmctrl_test_global_mask.c libsmctrl.a libsmctrl_test_mask_shared.o 42libsmctrl_test_global_mask: libsmctrl_test_global_mask.c libsmctrl.a libsmctrl_test_mask_shared.o
@@ -52,7 +59,7 @@ tests: libsmctrl_test_gpc_info libsmctrl_test_supreme_mask \
52 libsmctrl_test_stream_mask_override libsmctrl_test_next_mask \ 59 libsmctrl_test_stream_mask_override libsmctrl_test_next_mask \
53 libsmctrl_test_next_mask_override 60 libsmctrl_test_next_mask_override
54 61
55all: libsmctrl.so nvtaskset tests 62all: libsmctrl.so libcuda.so.1 nvtaskset tests
56 63
57clean: 64clean:
58 rm -f libsmctrl.so libsmctrl.o libsmctrl.a libsmctrl_test_gpc_info \ 65 rm -f libsmctrl.so libsmctrl.o libsmctrl.a libsmctrl_test_gpc_info \
@@ -60,4 +67,32 @@ clean:
60 libsmctrl_test_global_mask \ 67 libsmctrl_test_global_mask \
61 libsmctrl_test_stream_mask libsmctrl_test_stream_mask_override \ 68 libsmctrl_test_stream_mask libsmctrl_test_stream_mask_override \
62 libsmctrl_test_next_mask libsmctrl_test_next_mask_override \ 69 libsmctrl_test_next_mask libsmctrl_test_next_mask_override \
63 nvtaskset 70 nvtaskset libcuda.so.1
71
72install: libcuda.so.1
73 @# Check that CUDA is installed first
74 test -f /lib/$(ARCH)/libcuda.so.*.*
75 @# Change libcuda.so link to bypass libcuda.so.1
76 sudo ln -sf /lib/$(ARCH)/libcuda.so.*.* /lib/$(ARCH)/libcuda.so
77 @# Remove libcuda.so.1 symlink
78 sudo rm /lib/$(ARCH)/libcuda.so.1
79 @# Install wrapper as libcuda.so.1
80 sudo cp libcuda.so.1 /lib/$(ARCH)/libcuda.so.1
81
82remove:
83 @# Test that our library in installed first
84 test ! -L /lib/$(ARCH)/libcuda.so.1
85 @# Overwrite install with original symlinks
86 sudo ln -sf libcuda.so.1 /lib/$(ARCH)/libcuda.so
87 sudo ln -sf /lib/$(ARCH)/libcuda.so.*.* /lib/$(ARCH)/libcuda.so.1
88
89run_tests: tests
90 ./libsmctrl_test_global_mask
91 ./libsmctrl_test_next_mask
92 ./libsmctrl_test_stream_mask
93 ./libsmctrl_test_next_mask_override
94 ./libsmctrl_test_stream_mask_override
95 @# Must set LD_LIBRARY_PATH in case make install has not been run
96 LD_LIBRARY_PATH=. ./libsmctrl_test_supreme_mask
97 ./libsmctrl_test_gpc_info
98 @ echo "All tests passed!"
diff --git a/README.md b/README.md
index f2be718..11ad153 100644
--- a/README.md
+++ b/README.md
@@ -13,16 +13,37 @@ Please cite this paper in any work which leverages our library. Here's the BibTe
13 year={2023}, 13 year={2023},
14 month={May}, 14 month={May},
15 pages={54--66}, 15 pages={54--66},
16 doi={10.1109/RTAS58335.2023.00012},
16 _series={RTAS} 17 _series={RTAS}
17} 18}
18``` 19```
19 20
20Please see [the paper](https://www.cs.unc.edu/~jbakita/rtas23.pdf) and `libsmctrl.h` for details and examples of how to use this library. 21The ability for `libsmctrl` to work on unmodified tasks was developed as part of a follow-up paper:
22
23_J. Bakita and J. H. Anderson, "Hardware Compute Partitioning on NVIDIA GPUs for Composable Systems", Proceedings of the 37th Euromicro Conference on Real-Time Systems, pp. 18:1-18:24, July 2025._
24
25Please cite this paper in any work which uses this for partitioning unmodified tasks. Here's the BibTeX entry:
26```
27@inproceedings{bakita2025hardware,
28 title={Hardware Compute Partitioning on {NVIDIA} {GPUs} for Composable Systems},
29 author={Bakita, Joshua and Anderson, James H},
30 booktitle={Proceedings of the 37th Euromicro Conference on Real-Time Systems},
31 year={2025},
32 month={July},
33 pages={18:1--18:24},
34 doi={10.1109/ECRTS.2025.18},
35 _series={ECRTS}
36}
37```
38
39Please see [the first paper](https://www.cs.unc.edu/~jbakita/rtas23.pdf), [the second paper](https://www.cs.unc.edu/~jbakita/ecrts25.pdf) and `libsmctrl.h` for details and examples of how to use this library.
21We strongly encourage consulting those resources first; the below comments serve merely as an appendum. 40We strongly encourage consulting those resources first; the below comments serve merely as an appendum.
22 41
23## Run-time Dependencies 42## Run-time Dependencies
24`libcuda.so`, which is automatically installed by the NVIDIA GPU driver. 43`libcuda.so`, which is automatically installed by the NVIDIA GPU driver.
25 44
45(Technically `libdl` is also required, but this should never need to be manually installed. This is a dependency of CUDA, and is also part of the GNU C Standard Library starting with version 2.34.)
46
26## Building 47## Building
27To build, ensure that you have `gcc` installed and access to the CUDA SDK including `nvcc`. Then run: 48To build, ensure that you have `gcc` installed and access to the CUDA SDK including `nvcc`. Then run:
28``` 49```
@@ -66,8 +87,52 @@ nvcc benchmark.cu -o benchmark -I/playpen/libsmctl -lsmctrl -lcuda -L/playpen/li
66``` 87```
67The resultant `benchmark` binary should be portable to any system with an equivalent or newer version of the NVIDIA GPU driver installed. 88The resultant `benchmark` binary should be portable to any system with an equivalent or newer version of the NVIDIA GPU driver installed.
68 89
90## Use Without Application Modification
91As an alternative to modifying your application, `libsmctrl` can be installed system-wide, and partitions for each application can be set via the `nvtaskset` tool.
92The `nvtaskset` tool works very similarly to the Linux CPU-affinity-setting tool `taskset`.
93
94To install `libsmctrl` system-wide, such that all CUDA-using applications automatically load it, ensure that `patchelf` is installed (`sudo apt install patchelf`), and run:
95```
96make libcuda.so.1 install
97```
98Or, if you do not want to modify any system-wide state, and only want `libsmctrl` loaded as part of anything run from this console:
99```
100make libcuda.so.1
101export LD_LIBRARY_PATH=$(pwd)
102```
103(This works because CUDA is always dynamically loaded from `libcuda.so.1`, and `lbsmctrl` creates a "fake" `libcuda.so.1` in this directory that wraps CUDA.
104 Setting `LD_LIBRARY_PATH` ensures that the wrapped version is the first one loaded.
105 The only difference with running `make install` is that it copies our "fake" `libcuda.so.1` to a location where the loader will automatically find it.)
106
107And then to start an application within a specific TPC partition, e.g., the first 10 TPCs:
108```
109./nvtaskset -t 0-9 my_program my_args
110```
111Note that this will automatically start NVIDIA MPS, which is a prerequisite to co-run tasks on NVIDIA GPUs without timeslicing.
112
113And to change the TPCs available for a process ID 1234 to to the first 10 TPCs:
114```
115./nvtaskset -tp 0-9 1234
116```
117
118Or, to change a process of ID 1234 to only run on GPC 3:
119```
120./nvtaskset -gp 3 1234
121```
122
123To remove the system-wide installation of `libsmctrl`, run:
124```
125make remove
126```
127
69## Run Tests 128## Run Tests
70To test partitioning: 129
130To run them all:
131```
132make run_tests
133```
134
135If you prefer to run them individually, to test partitioning:
71``` 136```
72make tests 137make tests
73./libsmctrl_test_global_mask 138./libsmctrl_test_global_mask
@@ -82,18 +147,26 @@ make tests
82./libsmctrl_test_next_mask_override 147./libsmctrl_test_next_mask_override