End-to-End Object Detection with Transformers

★ 15,344⑂ 2,678👁 3⑃ 89● 526▤ 1
PythonApache-2.0Created 2020-05-26
下载:Detectron2 and torchscript support, attention and panoptic notebooks, code improvementsDetectron2-0.2 ↓0.22020-06-29
+0
日增长
15,344
stars
2,678
forks
3
watchers
89
prs
526
issues
1
releases

README

DE⫶TR: End-to-End Object Detection with Transformers
========



PyTorch training code and pretrained models for DETR (DEtection TRansformer).
We replace the full complex hand-crafted object detection pipeline with a Transformer, and match Faster R-CNN with a ResNet-50, obtaining 42 AP on COCO using half the computation power (FLOPs) and the same number of parameters. Inference in 50 lines of PyTorch.

!DETR

What it is. Unlike traditional computer vision techniques, DETR approaches object detection as a direct set prediction problem. It consists of a set-based global loss, which forces unique predictions via bipartite matching, and a Transformer encoder-decoder architecture. 
Given a fixed small set of learned object queries, DETR reasons about the relations of the objects and the global image context to directly output the final set of predictions in parallel. Due to this parallel nature, DETR is very fast and efficient.

About the code. We believe that object detection should not be more difficult than classification,
and should not require complex libraries for training and inference.
DETR is very simple to implement and experiment with, and we provide a
standalone Colab Notebook
showing how to do inference with DETR in only a few lines of PyTorch code.
Training code follows this idea - it is not a library,
but simply a main.py importing model and criterion
definitions with standard training loops.

Additionnally, we provide a Detectron2 wrapper in the d2/ folder. See the readme there for more information.

For details see End-to-End Object Detection with Transformers by Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.

See our blog post to learn more about end to end object detection with transformers.
 Model Zoo
We provide baseline DETR and DETR-DC5 models, and plan to include more in future.
AP is computed on COCO 2017 val5k, and inference time is over the first 100 val5k COCO images,
with torchscript transformer.


  
    
      
      name
      backbone
      schedule
      inf_time
      box AP
      url
      size
    
  
  
    
      0
      DETR
      R50
      500
      0.036
      42.0
      model | logs
      159Mb
    
    
      1
      DETR-DC5
      R50
      500
      0.083
      43.3
      model | logs
      159Mb
    
    
      2
      DETR
      R101
      500
      0.050
      43.5
      model | logs
      232Mb
    
    
      3
      DETR-DC5
      R101
      500
      0.097
      44.9
      model | logs
      232Mb
    
  


COCO val5k evaluation results can be found in this gist.

The models are also available via torch hub,
to load DETR R50 with pretrained weights simply do:
python
model = torch.hub.load('facebookresearch/detr:main', 'detr_resnet50', pretrained=True)



COCO panoptic val5k models:

  
    
      
      name
      backbone
      box AP
      segm AP
      PQ
      url
      size
    
  
  
    
    

Top issues

Latest release

Detectron2 and torchscript support, attention and panoptic notebooks, code improvements v0.2 2020-06-29
This release contains several improvements compared to the initial release of DETR.

 Highlights
 Detectron2 support
DETR now can be used with Detectron2 in a seamless way. It reuses the original abstractions of DETR, while making a compatibility layer so that it fits Detectron2's API. More information can be found in the README under the d2/ folder

 TorchScript support for all models
DETR models can now be natively exported to torchscript. This enables DETR to be run in C++ via libtorch. Converting a model to torchscript is easy:
python
model = torch.hub.load('facebookresearch/detr'

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON