Rockchip RK3588. I have problems with converting yolov5. I tried this:
But it is not working for me. In rknn-toolkit I can’t make an onnx file that will be the same format as as example In model-zoo I have different outputs in rknn model. What version of yolov5 I need to use to convert yolov5? I can’t find any working guide to convert it
Which RK3588 card are you using?
You may need to follow the link above and use RKNN-Toolkit2 to solve your problem.
RKNN-Toolkit2 project, path examples/onnx/yolov5 has a model for yolov5.
You can refer to Rockchip_Quick_Start_RKNN_SDK_V1.4.0_CN.pdf.
I am using ROC-RK3588S-PC Rockchip RK3588 8K AI. I tried RKNN-Toolkit2. examples/onnx/yolov5/test.py runs correct with a model from repository (yolov5s.onnx). But I want to use a custom model. For my start I tried to use yolov5s from version 6.2. I convert it to onnx with GitHub - airockchip/yolov5: YOLOv5 in PyTorch > ONNX > CoreML > TFLite. But when I changed yolov5s.onnx from examples/onnx/yolov5/ to my yolov5s.onnx it coverted wrong. Test drew so much rectangles
I tried another versions too. With 6.1 and 6.0 problem is same. With 5.0 it is not working.
In code I just added target_platform=“rk3588” to the function rknn.config
I also tried to set QUANTIZE_ON = False. It didn’t help me. Problem was the same
You can refer to here rknn_model_zoo/models/CV/object_detection/yolo/RKNN_model_convert at main · airockchip/rknn_model_zoo · GitHub
The conversion of each version of yolo, if there are still problems, you need to sort out the information (provide logs, conversion, reasoning py files, it is best to provide models)
I also tried to convert yolov5s.onnx with rknn_model_zoo/models/CV/object_detection/yolo/RKNN_model_convert at main · airockchip/rknn_model_zoo · GitHub.
My log with error (I have connect my Rockchip with usb):
root@bc58a61d4794:/examples/rknn_model_zoo/models/CV/object_detection/yolo/RKNN_model_convert# ./convert_yolo_ppyolo.sh
========== parser_config ==========
NPU_VERSION: 2
RK_device_platform: RK3588
input_example: [{'in_0': './bus.jpg'}] ...
pre_compile: online
core_mask: 1
config:
target_platform: RK3588
quantized_dtype: asymmetric_quantized-8
quantized_algorithm: normal
optimization_level: 3
mean_values: [[0, 0, 0]]
std_values: [[255, 255, 255]]
quantized_method: channel
quant_img_RGB2BGR: [False]
export_rknn:
export_path: ./model_cvt/RK3588/yolov5s_RK3588_i8.rknn
verbose: False
dataset: ./dataset.txt
quantize: True
build:
do_quantization: True
dataset: ./dataset.txt
model_framework: onnx
load:
model: ./yolov5s.onnx
inputs:
in_0:
shape: [3, 640, 640]
mean_values: [0, 0, 0]
std_values: [255, 255, 255]
img_type: RGB
outputs:
export_pre_compile_path: ./model_cvt/RK3588/yolov5s_RK3588_i8_precompile.rknn
===================================
./model_cvt/RK3588/yolov5s_RK3588_i8.rknn is already exists! Do you want to overwrite it? [Y/N]
Y or N:Y
---> Create RKNN object
W __init__: rknn-toolkit2 version: 1.4.0-22dcfef4
---> Seting RKNN config
---> Loading onnx model
---> Building
Analysing : 100%|███████████████████████████████████████████████| 145/145 [00:00<00:00, 2961.52it/s]
Quantizating : 100%|█████████████████████████████████████████████| 145/145 [00:00<00:00, 227.15it/s]
W build: The default input dtype of 'images' is changed from 'float32' to 'int8' in rknn model for performance!
Please take care of this change when deploy rknn model with Runtime API!
W build: The default output dtype of 'output' is changed from 'float32' to 'int8' in rknn model for performance!
Please take care of this change when deploy rknn model with Runtime API!
W build: The default output dtype of '329' is changed from 'float32' to 'int8' in rknn model for performance!
Please take care of this change when deploy rknn model with Runtime API!
W build: The default output dtype of '331' is changed from 'float32' to 'int8' in rknn model for performance!
Please take care of this change when deploy rknn model with Runtime API!
---> Export RKNN model
WARNING: RK3588 model needn't pre_compile. Ignore!
Convert Done!
adb: unable to connect for root: closed
Traceback (most recent call last):
File "../../../../../common/rknn_converter/rknn_convert.py", line 79, in <module>
convert(config_dict, args)
File "../../../../../common/rknn_converter/rknn_convert.py", line 24, in convert
vp = validate_phase(rknn, model_config_dict, args)
File "/examples/rknn_model_zoo/common/rknn_converter/phase.py", line 192, in __init__
self.check_board_info()
File "/examples/rknn_model_zoo/common/rknn_converter/phase.py", line 535, in check_board_info
self._smart_record('librknn_runtime_version', "{}".format(bc.get_librknn_version()))
File "/examples/rknn_model_zoo/common/utils/board_checker.py", line 81, in get_librknn_version
_result = result[0].rstrip('\n').split('version: ')[1]
IndexError: list index out of range
It finally converted my model. But this is not working. I get more then 8000 boxes in output (problem is the same with RKNN-Toolkit2 example). With onnx model from example in RKNN-Toolkit2 it is working well):
adb: unable to connect for root: closed
Your log shows that the debug environment for the PC and ROC-RK3588S-PC is not set up properly.
Please refer to this article:
ROC-RK3588S-PC/usage_npu
=============================================
https://github.com/airockchip/rknn_model_zoo/tree/main/models/CV/object_detection/yolo
Please check every step of this link.
I recommend using a trained model first to make finding the problem less difficult.
Step 1. Obtain the model
Use the yolov5/ yolov7/ YOLOX repositories under the airockchip organisation to train and export models according to the README_rkopt_manual.md hint in the repository. Get the model in torchscript / onnx format.
Or download the RKNN models already provided, get the web site, the exact location in the link is rknn_model_zoo/models/CV/object_detection/yolo/{YOLOX/yolov7/yolov5}/deploy_models/{ toolkit}/{platform}
If you can’t download the web site from github, you can download it from our google drive.
Step 2. model_transformation
Refer to the RKNN_model_convert instructions to convert the torchscript / onnx format model obtained in step 1 to an RKNN model.
The conversion script will call the rknn.eval_perf interface to obtain the inference performance of the RKNN model and calculate the similarity (based on cos distance) between the RKNN model and the original model, provided that the debugging function of the board is working.
Step 3. Test the RKNN/ torchscript/ onnx model using the Python demo
Refer to the RKNN_python_demo instructions to test the inference results of the yolo model in RKNN/ torchscript/ onnx format on images and plot them.
Support for folder testing, testing all images in the folder
Support for COCO benchmark testing
Step 4. Test the RKNN model using the C demo
RK1808/ RV1109/ RV1126 Please use RKNN_toolkit_1 version
RK3566/ RK3568/ RK3588 Please use the RKNN_toolkit_2 version
Support for single graph tests
Support for COCO benchmark testing
Support for post-processing of u8/fp output
I tried to use YOLOv5 from airrockchip. I started train.py without any corrections:
codeinside@CI-1442:~/yolov5_5.0/yolov5_airockchip/yolov5$ python3 train.py
train: weights=yolov5s.pt, cfg=, data=data/coco128.yaml, hyp=data/hyps/hyp.scratch-low.yaml, epochs=300, batch_size=16, imgsz=640, rect=False, resume=False, nosave=False, noval=False, noautoanchor=False, noplots=False, evolve=None, bucket=, cache=None, image_weights=False, device=, multi_scale=False, single_cls=False, optimizer=SGD, sync_bn=False, workers=8, project=runs/train, name=exp, exist_ok=False, quad=False, cos_lr=False, label_smoothing=0.0, patience=100, freeze=[0], save_period=-1, seed=0, local_rank=-1, entity=None, upload_dataset=False, bbox_interval=-1, artifact_alias=latest
github: ⚠️ YOLOv5 is out of date by 408 commits. Use `git pull ultralytics master` or `git clone https://github.com/ultralytics/yolov5` to update.
YOLOv5 🚀 v6.2-1-g4390b71 Python-3.8.10 torch-1.9.1+cu102 CUDA:0 (NVIDIA GeForce GTX 1660 Ti, 5937MiB)
hyperparameters: lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=0.05, cls=0.5, cls_pw=1.0, obj=1.0, obj_pw=1.0, iou_t=0.2, anchor_t=4.0, fl_gamma=0.0, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0
Weights & Biases: run 'pip install wandb' to automatically track and visualize YOLOv5 🚀 runs in Weights & Biases
TensorBoard: Start with 'tensorboard --logdir runs/train', view at http://localhost:6006/
ClearML Task: overwriting (reusing) task id=c9f723e1534d4927b79066fc7b67f3b9
ClearML results page: https://demoapp.demo.clear.ml/projects/774c1d35e7724c2c843e0d9ddff9853f/experiments/c9f723e1534d4927b79066fc7b67f3b9/output/log
Downloading https://github.com/ultralytics/yolov5/releases/download/v6.1/yolov5s.pt to yolov5s.pt...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14.1M/14.1M [00:10<00:00, 1.39MB/s]
from n params module arguments
0 -1 1 3520 models.common.Conv [3, 32, 6, 2, 2]
1 -1 1 18560 models.common.Conv [32, 64, 3, 2]
2 -1 1 18816 models.common.C3 [64, 64, 1]
3 -1 1 73984 models.common.Conv [64, 128, 3, 2]
4 -1 2 115712 models.common.C3 [128, 128, 2]
5 -1 1 295424 models.common.Conv [128, 256, 3, 2]
6 -1 3 625152 models.common.C3 [256, 256, 3]
7 -1 1 1180672 models.common.Conv [256, 512, 3, 2]
8 -1 1 1182720 models.common.C3 [512, 512, 1]
9 -1 1 656896 models.common.SPPF [512, 512, 5]
10 -1 1 131584 models.common.Conv [512, 256, 1, 1]
11 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
12 [-1, 6] 1 0 models.common.Concat [1]
13 -1 1 361984 models.common.C3 [512, 256, 1, False]
14 -1 1 33024 models.common.Conv [256, 128, 1, 1]
15 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
16 [-1, 4] 1 0 models.common.Concat [1]
17 -1 1 90880 models.common.C3 [256, 128, 1, False]
18 -1 1 147712 models.common.Conv [128, 128, 3, 2]
19 [-1, 14] 1 0 models.common.Concat [1]
20 -1 1 296448 models.common.C3 [256, 256, 1, False]
21 -1 1 590336 models.common.Conv [256, 256, 3, 2]
22 [-1, 10] 1 0 models.common.Concat [1]
23 -1 1 1182720 models.common.C3 [512, 512, 1, False]
24 [17, 20, 23] 1 229245 models.yolo.Detect [80, [[10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119], [116, 90, 156, 198, 373, 326]], [128, 256, 512]]
Model summary: 270 layers, 7235389 parameters, 7235389 gradients, 16.6 GFLOPs
Transferred 349/349 items from yolov5s.pt
AMP: checks passed ✅
optimizer: SGD(lr=0.01) with parameter groups 57 weight(decay=0.0), 60 weight(decay=0.0005), 60 bias
train: Scanning '/home/codeinside/yolov5_5.0/yolov5_airockchip/datasets/coco128/labels/train2017.cache' images and labels... 126 found, 2 missing, 0 empty, 0 corrupt: 100%|██████████| 128/128 [00:00<?, ?it/s]
val: Scanning '/home/codeinside/yolov5_5.0/yolov5_airockchip/datasets/coco128/labels/train2017.cache' images and labels... 126 found, 2 missing, 0 empty, 0 corrupt: 100%|██████████| 128/128 [00:00<?, ?it/s]
Plotting labels to runs/train/exp5/labels.jpg...
/home/codeinside/.local/lib/python3.8/site-packages/clearml/utilities/plotlympl/mpltools.py:371: MatplotlibDeprecationWarning:
The is_frame_like function was deprecated in Matplotlib 3.1 and will be removed in 3.3.
/home/codeinside/.local/lib/python3.8/site-packages/clearml/utilities/plotlympl/mpltools.py:371: MatplotlibDeprecationWarning:
The is_frame_like function was deprecated in Matplotlib 3.1 and will be removed in 3.3.
AutoAnchor: 4.27 anchors/target, 0.994 Best Possible Recall (BPR). Current anchors are a good fit to dataset ✅
Image sizes 640 train, 640 val
Using 8 dataloader workers
Logging results to runs/train/exp5
Starting training for 300 epochs...
Epoch gpu_mem box obj cls labels img_size
0/299 2.87G 0.05491 0.07362 0.02855 247 640: 100%|██████████| 8/8 [00:34<00:00, 4.31s/it]
Class Images Labels P R mAP@.5 mAP@.5:.95: 50%|█████ | 2/4 [00:30<00:25, 12.80s/it] WARNING: NMS time limit 1.260s exceeded
Class Images Labels P R mAP@.5 mAP@.5:.95: 75%|███████▌ | 3/4 [00:32<00:08, 8.15s/it] WARNING: NMS time limit 1.260s exceeded
Class Images Labels P R mAP@.5 mAP@.5:.95: 100%|██████████| 4/4 [00:35<00:00, 8.82s/it]
all 128 929 9.47e-06 0.00139 6.47e-06 1.88e-06
Epoch gpu_mem box obj cls labels img_size
1/299 4G 0.05262 0.07901 0.02556 207 640: 100%|██████████| 8/8 [00:31<00:00, 3.92s/it]
Class Images Labels P R mAP@.5 mAP@.5:.95: 100%|██████████| 4/4 [00:34<00:00, 8.57s/it]
all 128 929 1.43e-05 0.00216 1.01e-05 2.96e-06
After this I got trained model, I moved it from runs to folder of yolov5. I started export.py to get .onnx
codeinside@CI-1442:~/yolov5_5.0/yolov5_airockchip/yolov5$ python3 export.py --weights best.pt --include onnx --rknpu RK3588S
export: data=data/coco128.yaml, weights=['best.pt'], imgsz=[640, 640], batch_size=1, device=cpu, half=False, inplace=False, train=False, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=12, verbose=False, workspace=4, nms=False, agnostic_nms=False, topk_per_class=100, topk_all=100, iou_thres=0.45, conf_thres=0.25, include=['onnx'], rknpu=RK3588S
YOLOv5 🚀 v6.2-1-g4390b71 Python-3.8.10 torch-1.9.1+cu102 CPU
Fusing layers...
Model summary: 213 layers, 7225885 parameters, 0 gradients, 16.4 GFLOPs
---> save anchors for RKNN
[[10.0, 13.0], [16.0, 30.0], [33.0, 23.0], [30.0, 61.0], [62.0, 45.0], [59.0, 119.0], [116.0, 90.0], [156.0, 198.0], [373.0, 326.0]]
PyTorch: starting from best.pt with output shape (1, 255, 80, 80) (56.0 MB)
ONNX: starting export with onnx 1.8.1...
ONNX: export success, saved as best.onnx (27.6 MB)
Export complete (2.16s)
Results saved to /home/codeinside/yolov5_5.0/yolov5_airockchip/yolov5
Detect: python detect.py --weights best.onnx
Validate: python val.py --weights best.onnx
PyTorch Hub: model = torch.hub.load('ultralytics/yolov5', 'custom', 'best.onnx')
Visualize: https://netron.app
I used model best.onnx in modelzoo converter. I have next error:
(venv) codeinside@CI-1442:~/rknn_model_zoo/models/CV/object_detection/yolo/RKNN_model_convert$ ./convert_yolo_ppyolo.sh
========== parser_config ==========
NPU_VERSION: 2
RK_device_platform: RK3588
input_example: [{'in_0': './bus.jpg'}] ...
pre_compile: online
core_mask: 1
config:
target_platform: RK3588
quantized_dtype: asymmetric_quantized-8
quantized_algorithm: normal
optimization_level: 3
mean_values: [[0, 0, 0]]
std_values: [[255, 255, 255]]
quantized_method: channel
quant_img_RGB2BGR: [False]
export_rknn:
export_path: ./model_cvt/RK3588/yolov5s_RK3588_fp.rknn
verbose: False
dataset: ./dataset.txt
quantize: False
build:
do_quantization: False
dataset: ./dataset.txt
model_framework: onnx
load:
model: ./yolov5s.onnx
inputs:
in_0:
shape: [3, 640, 640]
mean_values: [0, 0, 0]
std_values: [255, 255, 255]
img_type: RGB
outputs:
export_pre_compile_path: ./model_cvt/RK3588/yolov5s_RK3588_fp_precompile.rknn
===================================
---> Create RKNN object
W __init__: rknn-toolkit2 version: 1.4.0-22dcfef4
---> Seting RKNN config
---> Loading onnx model
---> Building
---> Export RKNN model
WARNING: RK3588 model needn't pre_compile. Ignore!
Convert Done!
adb: unable to connect for root: closed
Traceback (most recent call last):
File "../../../../../common/rknn_converter/rknn_convert.py", line 79, in <module>
convert(config_dict, args)
File "../../../../../common/rknn_converter/rknn_convert.py", line 24, in convert
vp = validate_phase(rknn, model_config_dict, args)
File "/home/codeinside/rknn_model_zoo/common/rknn_converter/phase.py", line 192, in __init__
self.check_board_info()
File "/home/codeinside/rknn_model_zoo/common/rknn_converter/phase.py", line 535, in check_board_info
self._smart_record('librknn_runtime_version', "{}".format(bc.get_librknn_version()))
File "/home/codeinside/rknn_model_zoo/common/utils/board_checker.py", line 81, in get_librknn_version
_result = result[0].rstrip('\n').split('version: ')[1]
IndexError: list index out of range
I also tried to cover this. I have nex error:
codeinside@CI-1442:~/yolov5_5.0/yolov5_airockchip/yolov5$ python3 export.py --weights yolov5s_tk2.pt --include onnx --rknpu RK3588S
export: data=data/coco128.yaml, weights=['yolov5s_tk2.pt'], imgsz=[640, 640], batch_size=1, device=cpu, half=False, inplace=False, train=False, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=12, verbose=False, workspace=4, nms=False, agnostic_nms=False, topk_per_class=100, topk_all=100, iou_thres=0.45, conf_thres=0.25, include=['onnx'], rknpu=RK3588S
YOLOv5 🚀 v6.2-1-g4390b71 Python-3.8.10 torch-1.9.1+cu102 CPU
/home/codeinside/.local/lib/python3.8/site-packages/torch/serialization.py:602: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)
warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"
Traceback (most recent call last):
File "export.py", line 688, in <module>
main(opt)
File "export.py", line 682, in main
run(**vars(opt))
File "/home/codeinside/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "export.py", line 509, in run
model = attempt_load(weights, device=device, inplace=True, fuse=True) # load FP32 model
File "/home/codeinside/yolov5_5.0/yolov5_airockchip/yolov5/models/experimental.py", line 82, in attempt_load
ckpt = (ckpt.get('ema') or ckpt['model']).to(device).float() # FP32 model
File "/home/codeinside/.local/lib/python3.8/site-packages/torch/jit/_script.py", line 667, in __getattr__
return super(RecursiveScriptModule, self).__getattr__(attr)
File "/home/codeinside/.local/lib/python3.8/site-packages/torch/jit/_script.py", line 384, in __getattr__
return super(ScriptModule, self).__getattr__(attr)
File "/home/codeinside/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'RecursiveScriptModule' object has no attribute 'get'
I tried to convert this. I have a error:
codeinside@CI-1442:~/yolov5_5.0/yolov5_airockchip/yolov5$ python3 export.py --weights yolov5s_relu_tk2.pt --include onnx --rknpu RK3588S
export: data=data/coco128.yaml, weights=['yolov5s_relu_tk2.pt'], imgsz=[640, 640], batch_size=1, device=cpu, half=False, inplace=False, train=False, keras=False, optimize=False, int8=False, dynamic=False, simplify=False, opset=12, verbose=False, workspace=4, nms=False, agnostic_nms=False, topk_per_class=100, topk_all=100, iou_thres=0.45, conf_thres=0.25, include=['onnx'], rknpu=RK3588S
YOLOv5 🚀 v6.2-1-g4390b71 Python-3.8.10 torch-1.9.1+cu102 CPU
/home/codeinside/.local/lib/python3.8/site-packages/torch/serialization.py:602: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)
warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"
Traceback (most recent call last):
File "export.py", line 688, in <module>
main(opt)
File "export.py", line 682, in main
run(**vars(opt))
File "/home/codeinside/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "export.py", line 509, in run
model = attempt_load(weights, device=device, inplace=True, fuse=True) # load FP32 model
File "/home/codeinside/yolov5_5.0/yolov5_airockchip/yolov5/models/experimental.py", line 82, in attempt_load
ckpt = (ckpt.get('ema') or ckpt['model']).to(device).float() # FP32 model
File "/home/codeinside/.local/lib/python3.8/site-packages/torch/jit/_script.py", line 667, in __getattr__
return super(RecursiveScriptModule, self).__getattr__(attr)
File "/home/codeinside/.local/lib/python3.8/site-packages/torch/jit/_script.py", line 384, in __getattr__
return super(ScriptModule, self).__getattr__(attr)
File "/home/codeinside/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'RecursiveScriptModule' object has no attribute 'get'