NVIDIA /apex
A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch
★ 9,007⑂ 1,533👁 95⑃ 798● 1,229▤ 0
+0
日增长
9,007
stars
1,533
forks
95
watchers
798
prs
1,229
issues
0
releases
README
Introduction This repository holds NVIDIA-maintained utilities to streamline mixed precision and distributed training in Pytorch. Some of the code here will be included in upstream Pytorch eventually. The intent of Apex is to make up-to-date utilities available to users as quickly as possible. Installation Each apex.contrib module requires one or more install options other than --cpp_ext and --cuda_ext. Note that contrib modules do not necessarily support stable PyTorch releases, some of them might only be compatible with nightlies. Containers NVIDIA PyTorch Containers are available on NGC: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch. The containers come with all the custom extensions available at the moment. See the NGC documentation for details such as: - how to pull a container - how to run a pulled container - release notes From Source To install Apex from source, we recommend using the nightly Pytorch obtainable from https://github.com/pytorch/pytorch. The latest stable release obtainable from https://pytorch.org should also work. We recommend installing Ninja to make compilation faster. Linux For performance and full functionality, we recommend installing Apex with CUDA and C++ extensions using environment variables: Using Environment Variables (Recommended) bash git clone https://github.com/NVIDIA/apex cd apex Build with core extensions (cpp and cuda) APEX_CPP_EXT=1 APEX_CUDA_EXT=1 pip install -v --no-build-isolation . To build with additional extensions, specify them with environment variables APEX_CPP_EXT=1 APEX_CUDA_EXT=1 APEX_FUSED_CONV_BIAS_RELU=1 pip install -v --no-build-isolation . To build all contrib extensions at once APEX_CPP_EXT=1 APEX_CUDA_EXT=1 APEX_ALL_CONTRIB_EXT=1 pip install -v --no-build-isolation . To reduce the build time, parallel building can be enabled: bash NVCC_APPEND_FLAGS="--threads 4" APEX_PARALLEL_BUILD=8 APEX_CPP_EXT=1 APEX_CUDA_EXT=1 pip install -v --no-build-isolation . When CPU cores or memory are limited, the --parallel option is generally preferred over --threads. See pull1882 for more details. Using Command-Line Flags (Legacy Method) The traditional command-line flags are still supported: bash Using pip config-settings (pip = 23.1) pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./ For older pip versions pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./ To build with additional extensions pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--group_norm" ./ Python-Only Build APEX also supports a Python-only build via: bash pip install -v --disable-pip-version-check --no-build-isolation --no-cache-dir ./ A Python-only build omits: - Fused
Top issues
#1679`pyproject.toml` missing `packaging` dependency46 comments · 2023-06-12#1594can't install apex45 comments · 2023-02-21#439GPU memory issues (leak?)43 comments · 2019-08-16#150Compilation fails with --cuda_ext43 comments · 2019-02-10#180amp + checkpoint loading = problems41 comments · 2019-03-06#475New FusedAdam (since #424 commit 8 Aug 2019) Issues35 comments · 2019-09-05#319RuntimeError: CUDA error: an illegal memory access was encountered (multi_tensor_apply at csrc/multi_tensor_apply.cuh:101)32 comments · 2019-05-20#1852关于解决ModuleNotFoundError: No module named 'torch'导致安装失败30 comments · 2024-10-24