| Commit message (Collapse) | Author | Age |
|---|
| |
|
|
|
|
|
|
|
|
|
|
| |
- Use alternate path if nvidia-cuda-mps-control is not found.
- Work-around missing CUDA_MPS_PIPE_DIRECTORY default in L4T.
- Install to L4T-specific paths if they exist.
Note that CUDA_MPS_PIPE_DIRECTORY must be manually set by the user
on L4T systems before they can call nvidia-cuda-mps-control, or to
have to have non-nvtaskset-launched applications run with MPS.
Tested on Jetson AGX Orin with L4T 36.3.0 and cuda-compat-12-6.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The GPU needs to be on before the GPC-to-TPC mapping registers can
be read. The easiest way to power on the GPU is to create a CUDA
context, but this is fairly expensive.
In nvtaskset, some GPU-using task will likely already be running,
and so we can skip CUDA context creation in the common case.
Bug fixes:
- Delete the temporary CUDA context created by nvtaskset after
it is done with it. Fixes bug where nvtaskset would leak this
context into any program it launches.
Additional changes:
- Style fixes in libsmctrl.c
- Remove superfluous newlines from error() calls in nvtaskset.c
|
| |
|
|
|
|
|
|
|
| |
patchelf seems unreliable (--add-needed was not working in version
0.9), and it can be replaced with a simple sed command.
While on the original test system, /lib was a symlink to /usr/lib,
this does not seem to generally be the case, and CUDA is installed
in /usr/lib.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Rather than requiring libsmctrl.so to be preloaded, we now wrap
libcuda.so.1. All CUDA-using applications will load libcuda.so.1,
ensuring that our wrapper will always be dynamically loaded, no
matter if LD_PRELOAD is enabled, or if a program has been staticly
linked. All that needs to be done is that the location of our
"fake" libcuda.so.1 need to be put within the loader search path.
This can be done by setting LD_LIBRARY_PATH, or by installing
our wrapper into /lib/x86_64-linux-gnu.
The mask can still be set via the LIBSMCTRL_MASK environment
variable, but the easier-to-use nvtaskset tool is now the
recommended way to view or change the supreme TPC mask for any
CUDA-using application. This allows launching a program on the
first two GPCs via a command as simple as:
./nvtaskset -g 0-1 ./a_program a_program_args
(Note that use of the -g option requires the nvdebug kernel module
to first be loaded.)
These changes support the final version of the ECRTS'25 paper.
Note that nvtaskset does not yet fully support multi-GPU systems.
Bugfixes:
- Fix crash that would occur if both libsmctrl.so and libsmctrl.a
were built into an application.
- Correctly use GPU ID when initializing a context in
`libsmctrl_test_gpc_info`.
- Include `nvtaskset` as a prerequisite for
`libsmctrl_test_supreme_mask`.
- Fix malfunction of `libsmctrl_test_gpc_info` if
CUDA_VISIBLE_DEVICES is set.
Other minor changes:
- Adds make target to run all the tests.
- Fixes typos in comments.
- Enables -Wall build option.
- Upgrades supreme mask from 64 to 128 bits.
- Removes `detect_parker_soc()` from the global namespace.
- Adjusts test messages to be more succinct.
- Updates README with overview of how to partition unmodified
applications, more details on the tests, and information on the
new ECRTS'25 paper.
|
| |
|
|
|
| |
Indicate that the loader should search the standard library paths
by ommiting the "./" prefix.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
These updates are featured in the paper:
J. Bakita and J. H. Anderson, “Hardware Compute Partitioning on
NVIDIA GPUs for Composable Systems”, Proceedings of the 37th
Euromicro Conference on Real-Time Systems (ECRTS), to appear,
Jul 2025.
They:
1. Fix reported GPC to TPC mappings (requires nvdebug update).
2. Add support for a "supreme" mask, which overrides all others and
can be set on a per-process basis via an environment variable,
and optionally modified at runtime via the nvtaskset utility.
3. Add test for the supreme mask.
|
| |
|
|
|
|
|
|
|
| |
Code built with CUDA > 6.5 cannot run on CUDA 6.5 or older, so the
check added unecessary overhead.
Tested on CUDA 6.5 and CUDA 10.2 to generate the correct code, and
global and next tested to work on GTX 1060 3 GB with either build
while using CUDA 10.2 at runtime.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
- Add an "all" build target
- Fix build if libcuda.so is not on linker search path
- Do not assume that nvcc is available on $PATH
- Allow specifying CFLAGS and LDFLAGS when running make
- Allow passing non-standard CUDA build locations to make
Suggested usage if CUDA is installed in a non-standard location,
say, /playpen/jbakita/CUDA/cuda-archive/cuda-12.2:
make CUDA=/playpen/jbakita/CUDA/cuda-archive/cuda-12.2
|
| |
|
|
|
|
|
|
|
| |
The prefix "lib" should not be included, per the documentation.
This was not caught in local tests as the fallback was always used
locally.
Patch courtesy of Guanbin Xu <xugb@mail.ustc.edu.cn>.
|
| | |
|
| |
|
|
|
|
|
| |
Make automatically provides CXX and CC, and these manual
definitions were being ignored.
Also fix a missing space in one of the messages from the tests.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Use a different callback to intercept the TMD/QMD later in the
launch pipeline.
Major improvements:
- Fix bug with next mask not overriding stream mask on CUDA 11.0+
- Add CUDA 6.5-10.2 support for next- and global-granularity
partitioning masks on x86_64 and aarch64 Jetson
- Remove libdl dependency
- Partially support TMD/QMD Version 4 (Hopper)
Minor improvements:
- Check for sufficient CUDA version before before attempting to
apply a next-granularity partitioning mask
- Only check for sufficient CUDA version on the first call to
`libsmctrl_set_next_mask()` or `libsmctrl_set_global_mask()`,
rather than checking every time (lowers overheads)
- Check that TMD version is sufficient before modifying it
- Improve documentation
Issues:
- Partitioning mask bits have a different meaning in TMD/QMD
Version 4 and require floorsweeping and remapping information to
properly construct. This information will be forthcoming in
future releases of libsmctrl and nvdebug.
|
| |
|
|
| |
Also test and note that stream masking on CUDA 6.5 seems impossible.
|
| |
|
|
| |
Also update a comment
|
| |
|
|
|
|
|
|
|
|
| |
Commit 3f9bda39 made an error by using the pre-CUDA-12 mask
structure layout on CUDA 12.6 on aarch64 Jetson. Switch to the
CUDA 12+ layout (as used on x86_64).
Tests work either way on the Jetson Orin, so this change is not
strictly required, but seems advisable to support potenital large
(PCIe-attached?) GPUs on Jetson/DRIVE platforms.
|
| | |
|
| |
|
|
| |
Credit to Nordine Feddal for testing CUDA 12.4 on 550.544.14.
|
| | |
|
| |
|
|
|
| |
Also allow building with an alternate version of g++ for backwards
compatibility.
|
| |
|
|
|
|
|
|
|
|
|
| |
Stream-level masks should always override globally-set masks.
Next-kernel masks should always override both stream-level masks
and globally-set masks.
Tests reveal an issue with the next-kernel mask not overriding the
stream mask on CUDA 11.0+. CUDA appears to apply the per-stream
mask to the QMD/TMD after `launchCallback()` is triggered, making
it impossible to override as currently implemented.
|
| | |
|
| |
|
|
| |
Also rewrite the global masking test to be much more thorough.
|
| |
|
|
|
|
|
|
|
| |
Previously did not delineate between aarch64 and x86_64 stream
offsets, causing incorrect offsets to be used in many circumstances.
This has now been fixed.
A new function, libsmctrl_set_stream_mask_ext() has also been added
which supports masking up to 128 TPCs (rather than just 64).
|
| |
|
|
|
|
| |
nvcc links against a stub version of libcuda.so by default which is
missing a required symbol starting around CUDA 11.8. Use libdl to
resolve the symbol at runtime instead.
|
| |
|
|
|
| |
Also use static linking for tests, to avoid a need to set
LD_LIBRARY_PATH to include the libsmctrl directory.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Also improve documentation and abort with an error message if
attempting to set a global SM mask on an unsupported CUDA version.
(Would crash/corrupt state before.)
Also uncomment a line which errantly disabled global masking on
CUDA 10.2 on aarch64.
Tested with CUDA 10.2 on:
- x86_64 (GTX 1060 3GB, driver 440.100, jbakita-old.cs.unc.edu)
- aarch64 (Jetson TX2, driver r32.5, grizzly.cs.unc.edu)
|
| |
|
|
|
| |
Necessary for declarations of included functions. Absence would
result in a compilation error for programs omitting this include.
|
| |
|
|
|
|
|
|
| |
This function was previously unreliable when using CUDA 9.0 on the
Jetson TX2.
Also update some version comments and remove `set_sm_mask()`---a
legacy partitioning function that's no longer used.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Initially supports the GPU information functions via:
- pysmctrl.get_gpc_info(dev_id)
- pysmctrl.get_tpc_info(dev_id)
- pysmctrl.get_tpc_info_cuda(cuda_dev_id)
All functions are extensively documented. See pysmctrl/__init__.py
for details.
Device partitioning functions have yet to be mapped into Python, as
these will require more testing.
As part of this:
- libsmctrl_get_*_info() functions have been modified to consistently
return positive error codes.
- libsmctrl_get_tpc_info() now uses nvdebug-style device numbering and
uses libsmctrl_get_gpc_info() under the covers. This should be more
reliable.
- libsmctrl_get_tpc_info_cuda() has been introduced as an improved
version of the old libsmctrl_get_tpc_info() function. This continues
to use CUDA-style device numbering, but is now resiliant to CUDA
failures.
- Various minor style improvements in libsmctrl.c
|
| |
|
|
|
| |
This function would previously would yield invalid results for
GPUs with more than 31 TPCs.
|
|
|
- Tested working with cuda_scheduling_examiner
- Supports everything described in the accepted RTAS'23 paper
- Can be used as either a shared or staticly-linked library
- Documented in libsmctrl.h
|
