diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 83 |
1 files changed, 78 insertions, 5 deletions
| @@ -13,16 +13,37 @@ Please cite this paper in any work which leverages our library. Here's the BibTe | |||
| 13 | year={2023}, | 13 | year={2023}, |
| 14 | month={May}, | 14 | month={May}, |
| 15 | pages={54--66}, | 15 | pages={54--66}, |
| 16 | doi={10.1109/RTAS58335.2023.00012}, | ||
| 16 | _series={RTAS} | 17 | _series={RTAS} |
| 17 | } | 18 | } |
| 18 | ``` | 19 | ``` |
| 19 | 20 | ||
| 20 | Please see [the paper](https://www.cs.unc.edu/~jbakita/rtas23.pdf) and `libsmctrl.h` for details and examples of how to use this library. | 21 | The ability for `libsmctrl` to work on unmodified tasks was developed as part of a follow-up paper: |
| 22 | |||
| 23 | _J. Bakita and J. H. Anderson, "Hardware Compute Partitioning on NVIDIA GPUs for Composable Systems", Proceedings of the 37th Euromicro Conference on Real-Time Systems, pp. 18:1-18:24, July 2025._ | ||
| 24 | |||
| 25 | Please cite this paper in any work which uses this for partitioning unmodified tasks. Here's the BibTeX entry: | ||
| 26 | ``` | ||
| 27 | @inproceedings{bakita2025hardware, | ||
| 28 | title={Hardware Compute Partitioning on {NVIDIA} {GPUs} for Composable Systems}, | ||
| 29 | author={Bakita, Joshua and Anderson, James H}, | ||
| 30 | booktitle={Proceedings of the 37th Euromicro Conference on Real-Time Systems}, | ||
| 31 | year={2025}, | ||
| 32 | month={July}, | ||
| 33 | pages={18:1--18:24}, | ||
| 34 | doi={10.1109/ECRTS.2025.18}, | ||
| 35 | _series={ECRTS} | ||
| 36 | } | ||
| 37 | ``` | ||
| 38 | |||
| 39 | Please see [the first paper](https://www.cs.unc.edu/~jbakita/rtas23.pdf), [the second paper](https://www.cs.unc.edu/~jbakita/ecrts25.pdf) and `libsmctrl.h` for details and examples of how to use this library. | ||
| 21 | We strongly encourage consulting those resources first; the below comments serve merely as an appendum. | 40 | We strongly encourage consulting those resources first; the below comments serve merely as an appendum. |
| 22 | 41 | ||
| 23 | ## Run-time Dependencies | 42 | ## Run-time Dependencies |
| 24 | `libcuda.so`, which is automatically installed by the NVIDIA GPU driver. | 43 | `libcuda.so`, which is automatically installed by the NVIDIA GPU driver. |
| 25 | 44 | ||
| 45 | (Technically `libdl` is also required, but this should never need to be manually installed. This is a dependency of CUDA, and is also part of the GNU C Standard Library starting with version 2.34.) | ||
| 46 | |||
| 26 | ## Building | 47 | ## Building |
| 27 | To build, ensure that you have `gcc` installed and access to the CUDA SDK including `nvcc`. Then run: | 48 | To build, ensure that you have `gcc` installed and access to the CUDA SDK including `nvcc`. Then run: |
| 28 | ``` | 49 | ``` |
| @@ -66,8 +87,52 @@ nvcc benchmark.cu -o benchmark -I/playpen/libsmctl -lsmctrl -lcuda -L/playpen/li | |||
| 66 | ``` | 87 | ``` |
| 67 | The resultant `benchmark` binary should be portable to any system with an equivalent or newer version of the NVIDIA GPU driver installed. | 88 | The resultant `benchmark` binary should be portable to any system with an equivalent or newer version of the NVIDIA GPU driver installed. |
| 68 | 89 | ||
| 90 | ## Use Without Application Modification | ||
| 91 | As an alternative to modifying your application, `libsmctrl` can be installed system-wide, and partitions for each application can be set via the `nvtaskset` tool. | ||
| 92 | The `nvtaskset` tool works very similarly to the Linux CPU-affinity-setting tool `taskset`. | ||
| 93 | |||
| 94 | To install `libsmctrl` system-wide, such that all CUDA-using applications automatically load it, ensure that `patchelf` is installed (`sudo apt install patchelf`), and run: | ||
| 95 | ``` | ||
| 96 | make libcuda.so.1 install | ||
| 97 | ``` | ||
| 98 | Or, if you do not want to modify any system-wide state, and only want `libsmctrl` loaded as part of anything run from this console: | ||
| 99 | ``` | ||
| 100 | make libcuda.so.1 | ||
| 101 | export LD_LIBRARY_PATH=$(pwd) | ||
| 102 | ``` | ||
| 103 | (This works because CUDA is always dynamically loaded from `libcuda.so.1`, and `lbsmctrl` creates a "fake" `libcuda.so.1` in this directory that wraps CUDA. | ||
| 104 | Setting `LD_LIBRARY_PATH` ensures that the wrapped version is the first one loaded. | ||
| 105 | The only difference with running `make install` is that it copies our "fake" `libcuda.so.1` to a location where the loader will automatically find it.) | ||
| 106 | |||
| 107 | And then to start an application within a specific TPC partition, e.g., the first 10 TPCs: | ||
| 108 | ``` | ||
| 109 | ./nvtaskset -t 0-9 my_program my_args | ||
| 110 | ``` | ||
| 111 | Note that this will automatically start NVIDIA MPS, which is a prerequisite to co-run tasks on NVIDIA GPUs without timeslicing. | ||
| 112 | |||
| 113 | And to change the TPCs available for a process ID 1234 to to the first 10 TPCs: | ||
| 114 | ``` | ||
| 115 | ./nvtaskset -tp 0-9 1234 | ||
| 116 | ``` | ||
| 117 | |||
| 118 | Or, to change a process of ID 1234 to only run on GPC 3: | ||
| 119 | ``` | ||
| 120 | ./nvtaskset -gp 3 1234 | ||
| 121 | ``` | ||
| 122 | |||
| 123 | To remove the system-wide installation of `libsmctrl`, run: | ||
| 124 | ``` | ||
| 125 | make remove | ||
| 126 | ``` | ||
| 127 | |||
| 69 | ## Run Tests | 128 | ## Run Tests |
| 70 | To test partitioning: | 129 | |
| 130 | To run them all: | ||
| 131 | ``` | ||
| 132 | make run_tests | ||
| 133 | ``` | ||
| 134 | |||
| 135 | If you prefer to run them individually, to test partitioning: | ||
| 71 | ``` | 136 | ``` |
| 72 | make tests | 137 | make tests |
| 73 | ./libsmctrl_test_global_mask | 138 | ./libsmctrl_test_global_mask |
| @@ -82,18 +147,26 @@ make tests | |||
| 82 | ./libsmctrl_test_next_mask_override | 147 | ./libsmctrl_test_next_mask_override |
| 83 | ``` | 148 | ``` |
| 84 | 149 | ||
| 85 | And if `nvdebug` has been installed: | 150 | To test that `nvtaskset` can dynamically change the mask of a running program: |
| 86 | ``` | 151 | ``` |
| 87 | make tests | 152 | make libsmctrl_test_supreme_mask |
| 153 | ./libsmctrl_test_supreme_mask | ||
| 154 | ``` | ||
| 155 | |||
| 156 | To test that TPC to GPC mappings can be obtained (if `nvdebug` has been installed): | ||
| 157 | ``` | ||
| 158 | make libsmctrl_test_gpc_info | ||
| 88 | ./libsmctrl_test_gpc_info | 159 | ./libsmctrl_test_gpc_info |
| 89 | ``` | 160 | ``` |
| 90 | 161 | ||
| 162 | The `CUDA_VISIBLE_DEVICES` environment variable can be set to run any of the partitioning tests on a different GPU. | ||
| 163 | |||
| 91 | ## Supported GPUs | 164 | ## Supported GPUs |
| 92 | 165 | ||
| 93 | #### Known Working | 166 | #### Known Working |
| 94 | 167 | ||
| 95 | - NVIDIA GPUs from compute capability 3.5 through 8.9, including embedded "Jetson" GPUs | 168 | - NVIDIA GPUs from compute capability 3.5 through 8.9, including embedded "Jetson" GPUs |
| 96 | - CUDA 6.5 through 12.6 | 169 | - CUDA 6.5 through 12.8 |
| 97 | - `x86_64` and Jetson `aarch64` platforms | 170 | - `x86_64` and Jetson `aarch64` platforms |
| 98 | 171 | ||
| 99 | #### Known Issues | 172 | #### Known Issues |
