Community effort to bring CPU and pure Caffe2 / C++ inference support

[gadcam]

It looks like many people are asking for CPU inference and it seems it needs much work to make it happen. What I offer is that we use this issue to publicly state what work is needed and so people eager to have this feature could easily help to implement it.

@daquexian, @orionr, @rbgirshick do you have time to share a list of features / ops needed to convert all the models with convert_pkl_to_pb.py ?

Feature/Operator	Where do we need it ?	State	Difficulty
CollectAndDistributeFpnRpnProposals	FPN	🕔 PR #372 submitted & Review needed	?
...	...	...	...

I would like to contribute to this effort but I do not know where to begin.
If you are willing to implement a feature do not hesitate to tell it in this issue.

Ps: To avoid any confusion I am only a random user of the Detectron & my initiative was not solicited by the maintainers

[daquexian]

Based on #372, models containing FPN can be correctly converted to caffe2's .pb files. (I will rebase the PR on master soon) However only detection net will be converted even in Mask R-CNN and Keypoint R-CNN which has mask net or keypoint net.

[gadcam]

@daquexian I am really sorry but I think I failed to understand properly what you mean as I do not have a deep understanding how the Detectron repo works.

Do you mean that, when #372 will be merged, if we try to convert for example e2e_keypoint_rcnn_R-50-FPN_1x only the proposal part would be converted and so we could not use it on CPU ?
If the answer to this question is yes, can you help us understand what steps we need to take to achieve a complete conversion ?

[daquexian]

@gadcam If we try to convert e2e_keypoint_rcnn_R-50-FPN_1x, we will only get bounding boxes but not keypoint. Because in here only model.net is used, but mask and keypoint are in model.mask_net and model.keypoint_net like it. The solution seems straightforward because there are only normal layers in these nets. But if you want to infer masks or keypoints after getting bounding boxes (in order to save inference time), it seems better to save these nets in different .pb files

[HappyKerry]

@daquexian would you like to write a detail guild on how to change pkl to pb? Thanks

[daquexian]

@HappyKerry Just fetch and checkout my branch

git remote add daquexian https://github.com/daquexian/Detectron
git fetch daquexian
git checkout daquexian/add-export-support-fpn

and run convert_pkl_to_pb.py with your configuration files and weights

[gadcam]

@daquexian I ran convert_pkl_to_pb.py (with your patch) successfully on e2e_keypoint_rcnn_R-50-FPN_s1x and on MSRA's original ResNet-50 model.

For e2e_keypoint_rcnn_R-50-FPN_s1x I have no warning.
For MSRA's original ResNet-50 model I have the following output

Blob fpn_inner_res5_2_sum_w with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_inner_res5_2_sum_b with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_inner_res4_5_sum_lateral_w with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_inner_res4_5_sum_lateral_b with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_inner_res3_3_sum_lateral_w with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_inner_res3_3_sum_lateral_b with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_inner_res2_2_sum_lateral_w with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_inner_res2_2_sum_lateral_b with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_res5_2_sum_w with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_res5_2_sum_b with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_res4_5_sum_w with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_res4_5_sum_b with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_res3_3_sum_w with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_res3_3_sum_b with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_res2_2_sum_w with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_res2_2_sum_b with type <class 'str'> is not supported in generating init net, skipped.
Blob conv_rpn_fpn2_w with type <class 'str'> is not supported in generating init net, skipped.
Blob conv_rpn_fpn2_b with type <class 'str'> is not supported in generating init net, skipped.
Blob rpn_cls_logits_fpn2_w with type <class 'str'> is not supported in generating init net, skipped.
Blob rpn_cls_logits_fpn2_b with type <class 'str'> is not supported in generating init net, skipped.
Blob rpn_bbox_pred_fpn2_w with type <class 'str'> is not supported in generating init net, skipped.
Blob rpn_bbox_pred_fpn2_b with type <class 'str'> is not supported in generating init net, skipped.
Blob fc6_w with type <class 'str'> is not supported in generating init net, skipped.
Blob fc6_b with type <class 'str'> is not supported in generating init net, skipped.
Blob fc7_w with type <class 'str'> is not supported in generating init net, skipped.
Blob fc7_b with type <class 'str'> is not supported in generating init net, skipped.
Blob cls_score_w with type <class 'str'> is not supported in generating init net, skipped.
Blob cls_score_b with type <class 'str'> is not supported in generating init net, skipped.
Blob bbox_pred_w with type <class 'str'> is not supported in generating init net, skipped.
Blob bbox_pred_b with type <class 'str'> is not supported in generating init net, skipped.

If I try to convert model.keypoint_net from e2e_keypoint_rcnn_R-50-FPN_s1x I get

Blob fpn_res2_2_sum with type <class 'str'> is not supported in generating init net, skipped.
Blob keypoint_rois_fpn2 with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_res3_3_sum with type <class 'str'> is not supported in generating init net, skipped.
Blob keypoint_rois_fpn3 with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_res4_5_sum with type <class 'str'> is not supported in generating init net, skipped.
Blob keypoint_rois_fpn4 with type <class 'str'> is not supported in generating init net, skipped.
Blob fpn_res5_2_sum with type <class 'str'> is not supported in generating init net, skipped.
Blob keypoint_rois_fpn5 with type <class 'str'> is not supported in generating init net, skipped.
Blob keypoint_rois_idx_restore_int32 with type <class 'str'> is not supported in generating init net, skipped.

So I have a few questions

• Why is there no Blob ____ is not supported for the keypoint model when we have some for the ResNet ?
• What does it means to have some Blob ____ is not supported in the ResNet ? Should we implement these operators ? (I thought the ResNet would be converted without trouble)
• Same question for the keypoint part of e2e_keypoint_rcnn_R-50-FPN_s1x

[daquexian]

@gadcam You should use the model in Model Zoo.

[gadcam]

@daquexian Then that is perfect : I did use the models in the Model Zoo.
To be accurate what I call

• MSRA's original ResNet-50 model is https://s3-us-west-2.amazonaws.com/detectron/ImageNetPretrained/MSRA/R-50.pkl
• e2e_keypoint_rcnn_R-50-FPN_s1x is https://s3-us-west-2.amazonaws.com/detectron/37697714/12_2017_baselines/e2e_keypoint_rcnn_R-50-FPN_s1x.yaml.08_44_03.qrQ0ph6M/output/train/keypoints_coco_2014_train%3Akeypoints_coco_2014_valminusminival/generalized_rcnn/model_final.pkl (so in section End-to-End Keypoint-Only Mask R-CNN Baselines)

Why did you suspect I tried to convert something else ? Because I have some Blob ____ is not supported when I should not ?

[daquexian]

@gadcam Yes. It is reasonable that Blob ____ is not supported appears when you use an ImageNet pretrained model, because fpn, rpn and some other layers are not in ImageNet pretrained models.

Could you please tell me what ops not supported output means?

[gadcam]

@daquexian

Could you please tell me what ops not supported output means?

I meant Blob ____ is not supported I am sorry for my inaccuracy. (I corrected it)

It is reasonable that Blob ____ is not supported appears when you use an ImageNet pretrained model, because fpn, rpn and some other layers are not in ImageNet pretrained models.

I am not sure I got this part : do you mean that when we see Blob ____ is not supported it means the Blob needs some code from the Detectron to be fully defined ?

So I think we are getting to the point of my issue : what should we implement to avoid it ? Or can you direct me where to dive to know what we need to implement ?

If we take an example (but we could say the same thing for keypoint_rois_idx_restore_int32)

Blob keypoint_rois_fpn2 with type <class 'str'> is not supported in generating init net, skipped.

The only mention I found of keypoint_rois_fpn in the code is here

Detectron/detectron/roi_data/fast_rcnn.py

Line 103 in b3c93df

blob_names += ['keypoint_rois_fpn' + str(lvl)]

So I do not really know where to investigate to avoid this Blob ____ is not supported error.

As a side question should we implement something like

Detectron/detectron/ops/generate_proposal_labels.py

Line 30 in e5bb3a8

class GenerateProposalLabelsOp(object):

in Caffe2/PyTorch repo and then add some conversion code here to get full CPU support ?

[daquexian]

@gadcam Blob ____ is not supported here just indicates that the blob doesn't have any value (I don't know why its type will be 'str' when it doesn't have any value, caffe2 is strange). There is no more layers needed to implement. You can add the name of these blobs into empty_blobs like

Detectron/tools/convert_pkl_to_pb.py

Line 558 in b3c93df

empty_blobs = ['data', 'im_info']

('data' and 'im_info' are the inputs of model.net, 'fpn_res2_2_sum', 'keypoint_rois_fpn2' and so on are the inputs of model.keypoint_net)

The converted model will crash when you try to verify it. Because its inputs are not legal. Maybe giving it some proper inputs ('fpn_res2_2_sum' and so on produced by bbox branch, and also "keypoint_rois_fpnX" below) will make it run.

Detectron/detectron/core/test.py

Lines 540 to 566 in b3c93df

	def im_detect_keypoints(model, im_scale, boxes):
	"""Infer instance keypoint poses. This function must be called after
	im_detect_bbox as it assumes that the Caffe2 workspace is already populated
	with the necessary blobs.
	Arguments:
	model (DetectionModelHelper): the detection model to use
	im_scales (list): image blob scales as returned by im_detect_bbox
	boxes (ndarray): R x 4 array of bounding box detections (e.g., as
	returned by im_detect_bbox)
	Returns:
	pred_heatmaps (ndarray): R x J x M x M array of keypoint location
	logits (softmax inputs) for each of the J keypoint types output
	by the network (must be processed by keypoint_results to convert
	into point predictions in the original image coordinate space)
	"""
	M = cfg.KRCNN.HEATMAP_SIZE
	if boxes.shape[0] == 0:
	pred_heatmaps = np.zeros((0, cfg.KRCNN.NUM_KEYPOINTS, M, M), np.float32)
	return pred_heatmaps
	inputs = {'keypoint_rois': _get_rois_blob(boxes, im_scale)}
	# Add multi-level rois for FPN
	if cfg.FPN.MULTILEVEL_ROIS:
	_add_multilevel_rois_for_test(inputs, 'keypoint_rois')

[gadcam]

@daquexian Thank you for your hints, with a bit of work I was able to run e2e_keypoint_rcnn_R-50-FPN_s1x on CPU !
I will tidy up my code before sharing it.
If I am able to write something clean enough I will do a PR to enable conversion of keypoints and mask-models with test to check the correctness of the conversion. (and so an example of how to run it)
For the moment the main problem is that I could not pick programmatically the input blobs.

[daquexian]

@gadcam Great! Looking forward to your PR

[dongmingsun]

@gadcam Hi, are we able to convert the Mask R-CNN model from .pkl to .pb now?

[gadcam]

@dongmingsun With @daquexian's #372 + my (future) PR you will be able to convert the models from the Zoo from .pkl to two .pb files, one for the bbox and one for the mask or keypoints, and you would need to use some helper function to run them.
What I achieved is to run it without the need of a GPU, not to have a pure Caffe2 model.
I think someone more experimented than me would be able to merge these two .pb files at least. I will investigate quickly this option.