Post

mm框架整理

Posted Updated
By qiuming
7 min read

写在前面

整理本文的主要目的是因为自己老是记不住，捋不清项目的主要结构，故记下以帮助自己理解和使用时查找。

mmdet总框架

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

├── apis  
├── core  
│   ├── anchor  
│   ├── bbox  
│   │   ├── assigners  
│   │   ├── coder  
│   │   ├── iou_calculators  
│   │   ├── match_costs  
│   │   └── samplers  
│   ├── evaluation  
│   ├── export  
│   ├── mask  
│   ├── post_processing  
│   ├── utils  
│   └── visualization  
├── datasets  
│   ├── pipelines  
│   └── samplers  
├── models  
│   ├── backbones  
│   ├── dense_heads  
│   ├── detectors  
│   ├── losses  
│   ├── necks  
│   ├── roi_heads  
│   │   ├── bbox_heads  
│   │   ├── mask_heads  
│   │   ├── roi_extractors  
│   │   └── shared_heads  
│   └── utils  
└── utils

主要包含被外部调用的api、dataset、model、utils和core。

其中api的主要目录结构如下：

1
2
3
4
5
6

├── __init__.py
├── inference.py
├── test.py
└── train.py

包含直接使用config.py配置训练，测试推理的代码。

core是整个mmlab的核心组件，包含anchor生成，bbox处理，NMS后处理等其他操作，也包括evaluation，export、visualization等核心功能。

core是整个项目实现的主要调用部分，基本上检测的框架一些专门的部件都抽象，然后封装在里面。这里面东西还挺多的，就分别整理一下。

检测中的一般流程是先抽取特征，然后再在特征上对生成的anchor bbox进行rpn得到第一次粗粒度生成的框proposal，然后对这些proposal进行assigner和sample，再对应到原图上进行bbox_head进行分类和回归，并计算loss。单阶段没有proposal操作直接一步到位对bbox进行操作。

core下的anchor的目录结构如下：

1
2
3
4
5
6
7

├── __init__.py
├── anchor_generator.py
├── builder.py
├── point_generator.py
└── utils.py

其中包括普通的anchor_generator和anchor-free的基于point的generator

core下的bbox目录结构如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

├── assigners
│   ├── __init__.py
│   ├── approx_max_iou_assigner.py
│   ├── assign_result.py
│   ├── atss_assigner.py
│   ├── base_assigner.py
│   ├── center_region_assigner.py
│   ├── grid_assigner.py
│   ├── hungarian_assigner.py
│   ├── max_iou_assigner.py
│   ├── point_assigner.py
│   └── region_assigner.py
├── coder
│   ├── __init__.py
│   ├── base_bbox_coder.py
│   ├── bucketing_bbox_coder.py
│   ├── delta_xywh_bbox_coder.py
│   ├── legacy_delta_xywh_bbox_coder.py
│   ├── pseudo_bbox_coder.py
│   ├── tblr_bbox_coder.py
│   └── yolo_bbox_coder.py
├── iou_calculators
│   ├── __init__.py
│   ├── builder.py
│   └── iou2d_calculator.py
├── match_costs
│   ├── __init__.py
│   ├── builder.py
│   └── match_cost.py
├── samplers
│   ├── __init__.py
│   ├── base_sampler.py
│   ├── combined_sampler.py
│   ├── instance_balanced_pos_sampler.py
│   ├── iou_balanced_neg_sampler.py
│   ├── ohem_sampler.py
│   ├── pseudo_sampler.py
│   ├── random_sampler.py
│   ├── sampling_result.py
│   └── score_hlr_sampler.py
├── __init__.py
├── builder.py
├── demodata.py
└── transforms.py

包括assigner，sampler，code，iou和match_cost计算模块。

Dataset目录结构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

├── pipelines
│   ├── __init__.py
│   ├── auto_augment.py
│   ├── compose.py
│   ├── formating.py
│   ├── instaboost.py
│   ├── loading.py
│   ├── test_time_aug.py
│   └── transforms.py
├── samplers
│   ├── __init__.py
│   ├── distributed_sampler.py
│   └── group_sampler.py
├── __init__.py
├── builder.py
├── cityscapes.py
├── coco.py
├── custom.py
├── dataset_wrappers.py
├── deepfashion.py
├── lvis.py
├── utils.py
├── voc.py
├── wider_face.py
└── xml_style.py

主要是pipeline，samplers和coco.py(custom.py)值得关注。

model是项目的重中之重，包含backbone, neck, roi_head, dense_head, detector, loss和util 整体结构如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14

├── models  
│   ├── backbones  
│   ├── dense_heads  
│   ├── detectors  
│   ├── losses  
│   ├── necks  
│   ├── roi_heads  
│   │   ├── bbox_heads  
│   │   ├── mask_heads  
│   │   ├── roi_extractors  
│   │   └── shared_heads  
│   └── utils
backbones主要目录结构如下 

├── __init__.py
├── darknet.py
├── detectors_resnet.py
├── detectors_resnext.py
├── hourglass.py
├── hrnet.py
├── regnet.py
├── res2net.py
├── resnest.py
├── resnet.py
├── resnext.py
├── ssd_vgg.py
└── trident_resnet.py
neck目录如下

├── __init__.py
├── bfp.py
├── channel_mapper.py
├── fpg.py
├── fpn.py
├── fpn_carafe.py
├── hrfpn.py
├── nas_fpn.py
├── nasfcos_fpn.py
├── pafpn.py
├── rfp.py
└── yolo_neck.py

utils目录如下

1
2
3
4
5
6
7
8
9

├── __init__.py
├── collect_env.py
├── contextmanagers.py
├── logger.py
├── profiling.py
├── util_mixins.py
└── util_random.py

Registry有哪些：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

RUNNERS
LOOPS
HOOKS
DATASETS
DATA_SAMOLERS
MODELS
MODEL_RAPPERS
WEIGHT_INITIALIZERS
OPTIMIZERS
OPTIMIZER_WRAPPERS
OPTIMEZER_WRAPPER_CONSTRUCTORS
PARAM_SCHEDULERS
METRICS
EVALUATOR
VISUALIZERS
VISBACKENS
LOG_PROCESSORS

调用流程有哪些？ 主要是调用forward函数, 然后是三种模式tensor, loss, predict. 分别调用方法: _forward(), loss(), predict().


classDiagram
  class BaseDetector {
    with_bbox
    with_mask
    with_neck
    with_shared_head
    _forward(batch_inputs: Tensor, batch_data_samples: OptSampleList)*
    add_pred_to_datasample(data_samples: SampleList, results_list: InstanceList) SampleList
    extract_feat(batch_inputs: Tensor)*
    forward(inputs: torch.Tensor, data_samples: OptSampleList, mode: str="tensor") ForwardResults
    loss(batch_inputs: Tensor, batch_data_samples: SampleList)* Union[dict, tuple]
    predict(batch_inputs: Tensor, batch_data_samples: SampleList)* SampleList
  }
  class BaseModel {
    data_preprocessor : NoneType, dict
    _run_forward(data: Union[dict, tuple, list], mode: str) Union[Dict[str, torch.Tensor], list]
    _set_device(device: torch.device) None
    cpu() nn.Module
    cuda(device: Optional[Union[int, str, torch.device]]) nn.Module
    forward(inputs: torch.Tensor, data_samples: Optional[list], mode: str)* Union[Dict[str, torch.Tensor], list]
    mlu(device: Union[int, str, torch.device, None]) nn.Module
    musa(device: Optional[Union[int, str, torch.device]]) nn.Module
    npu(device: Union[int, str, torch.device, None]) nn.Module
    parse_losses(losses: Dict[str, torch.Tensor]) Tuple[torch.Tensor, Dict[str, torch.Tensor]]
    test_step(data: Union[dict, tuple, list]) list
    to() nn.Module
    train_step(data: Union[dict, tuple, list], optim_wrapper: OptimWrapper) Dict[str, torch.Tensor]
    val_step(data: Union[tuple, dict, list]) list
  }
  class BaseModule {
    _is_init : bool
    _params_init_info
    init_cfg : NoneType
    is_init
    _dump_init_info()
    init_weights()
  }
  BaseDetector --|> BaseModel
  BaseModel --|> BaseModule
  BaseModule --|> Module
  defaultdict --* BaseModule : _params_init_info

关于二阶段检测器的类图 重写了三个方法


classDiagram
  class BaseDetector {
    with_bbox
    with_mask
    with_neck
    with_shared_head
    _forward(batch_inputs: Tensor, batch_data_samples: OptSampleList)*
    add_pred_to_datasample(data_samples: SampleList, results_list: InstanceList) SampleList
    extract_feat(batch_inputs: Tensor)*
    forward(inputs: torch.Tensor, data_samples: OptSampleList, mode: str) ForwardResults
    loss(batch_inputs: Tensor, batch_data_samples: SampleList)* Union[dict, tuple]
    predict(batch_inputs: Tensor, batch_data_samples: SampleList)* SampleList
  }
  class BaseModel {
  }
  class BaseModule {
  }
  class TwoStageDetector {
    backbone
    neck
    roi_head
    rpn_head
    test_cfg : Optional[OptConfigType]
    train_cfg : Optional[OptConfigType]
    with_roi_head
    with_rpn
    _forward(batch_inputs: Tensor, batch_data_samples: SampleList) tuple
    _load_from_state_dict(state_dict: dict, prefix: str, local_metadata: dict, strict: bool, missing_keys: Union[List[str], str], unexpected_keys: Union[List[str], str], error_msgs: Union[List[str], str]) None
    extract_feat(batch_inputs: Tensor) Tuple[Tensor]
    loss(batch_inputs: Tensor, batch_data_samples: SampleList) dict
    predict(batch_inputs: Tensor, batch_data_samples: SampleList, rescale: bool) SampleList
  }
  BaseDetector --|> BaseModel
  TwoStageDetector --|> BaseDetector
  BaseModel --|> BaseModule
  BaseModule --|> Module
det
mmlab detection
This post is licensed under CC BY 4.0 by the author.