NVIDIA /apex

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

★ 9,007⑂ 1,533👁 95⑃ 798● 1,229▤ 0
PythonBSD-3-ClauseCreated 2018-04-23
+0
日增长
9,007
stars
1,533
forks
95
watchers
798
prs
1,229
issues
0
releases

README

Introduction

This repository holds NVIDIA-maintained utilities to streamline mixed precision and distributed training in Pytorch.
Some of the code here will be included in upstream Pytorch eventually.
The intent of Apex is to make up-to-date utilities available to users as quickly as possible.

 Installation
Each apex.contrib module requires one or more install options other than --cpp_ext and --cuda_ext.
Note that contrib modules do not necessarily support stable PyTorch releases, some of them might only be compatible with nightlies.

 Containers
NVIDIA PyTorch Containers are available on NGC: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch.
The containers come with all the custom extensions available at the moment. 

See the NGC documentation for details such as:
- how to pull a container
- how to run a pulled container
- release notes

 From Source

To install Apex from source, we recommend using the nightly Pytorch obtainable from https://github.com/pytorch/pytorch.

The latest stable release obtainable from https://pytorch.org should also work.

We recommend installing Ninja to make compilation faster.

 Linux

For performance and full functionality, we recommend installing Apex with CUDA and C++ extensions using environment variables:

 Using Environment Variables (Recommended)

bash
git clone https://github.com/NVIDIA/apex
cd apex
 Build with core extensions (cpp and cuda)
APEX_CPP_EXT=1 APEX_CUDA_EXT=1 pip install -v --no-build-isolation .

 To build with additional extensions, specify them with environment variables
APEX_CPP_EXT=1 APEX_CUDA_EXT=1 APEX_FUSED_CONV_BIAS_RELU=1 pip install -v --no-build-isolation .

 To build all contrib extensions at once
APEX_CPP_EXT=1 APEX_CUDA_EXT=1 APEX_ALL_CONTRIB_EXT=1 pip install -v --no-build-isolation .


To reduce the build time, parallel building can be enabled:

bash
NVCC_APPEND_FLAGS="--threads 4" APEX_PARALLEL_BUILD=8 APEX_CPP_EXT=1 APEX_CUDA_EXT=1 pip install -v --no-build-isolation .


When CPU cores or memory are limited, the --parallel option is generally preferred over --threads. See pull1882 for more details.

 Using Command-Line Flags (Legacy Method)

The traditional command-line flags are still supported:

bash
 Using pip config-settings (pip = 23.1)
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./

 For older pip versions
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./

 To build with additional extensions
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--group_norm" ./


 Python-Only Build

APEX also supports a Python-only build via:
bash
pip install -v --disable-pip-version-check --no-build-isolation --no-cache-dir ./

A Python-only build omits:
- Fused 

Top issues

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON