基于深度学习的船舶驾驶员疲劳检测算法

王鹏; 神和龙; 尹勇; 吕红光

doi:10.3963/j.jssn.1674-4861.2022.01.008

基于深度学习的船舶驾驶员疲劳检测算法

doi: 10.3963/j.jssn.1674-4861.2022.01.008

1.
大连海事大学航海动态仿真和控制交通行业重点实验室辽宁大连 116026
2.
大连海事大学航海学院辽宁大连 116026

基金项目:

国家重点研发计划项目 2019YFB1600602

详细信息

作者简介:
王鹏（1996—），硕士研究生. 研究方向：计算机视觉航海应用. E-mail：Arieswp1996@163.com

通讯作者:
神和龙（1984—），博士，副教授. 研究方向：航海动态仿真、计算机视觉航海应用. E-mail：shenhelong@126.com

中图分类号: U676.1
计量
- 文章访问数: 1365
- HTML全文浏览量: 859
- PDF下载量: 130
- 被引次数: 0
出版历程
- 收稿日期: 2021-08-12
- 网络出版日期: 2022-03-31

A Detection Algorithm for the Fatigue of Ship Officers Based on Deep Learning Technique

1.
Key Laboratory of Marine Simulation & Control for Ministry of Transportation, Dalian Maritime Univers ity, Dalian 116026, Liaoning, China
2.
Navigation College, Dalian Maritime University, Dalian 116026, Liaoning, China

摘要

摘要: 针对日益凸显的船舶值班人员疲劳驾驶问题，为有效预警值班驾驶员的疲劳状态，保障船舶航行安全，研究了基于深度学习的疲劳检测算法。考虑到船舶驾驶台空间大、背景复杂等特点，使用深度可分离卷积改进RetinaFace人脸检测模型，优化模型的检测速度；基于Channel Split和Channel Shuffle思想，结合批量归一化、全局平均池化等技术搭建改进的ShuffleNetV2网络，自动提取图像特征，识别眼睛、嘴巴的开闭状态；根据PERCLOS准则融合眼睛、嘴巴2个特征参数综合判定驾驶员是否疲劳。实验结果表明：改进后RetinaFace模型的检测速度由9.33帧/s提升至22.60帧/s，人脸检测精度和速度均优于多任务卷积神经网络（MTCNN）；改进的ShuffleNetV2网络识别眼睛、嘴巴状态的准确率高达99.50%以上；算法在模拟驾驶台环境中识别疲劳状态的精确率达到95.70%，召回率达到96.73%，均高于目前常见的Haar-like+Adaboost以及MTCNN+CNN疲劳检测算法。算法检测每帧图片仅需0.083 s，基本满足实时检测的要求。
- 交通安全 /
- 疲劳检测 /
- RetinaFace /
- ShuffleNetV2 /
- PERCLOS准则
Abstract: Aiming at preventing Officers on Watch (OOW) from fatigue driving, a fatigue detection and alert algorithm based on deep learning technique is developed. Considering the large space and complex background of the ship bridge, the RetinaFace model is improved by using Depthwise Separable Convolution to optimize the detection speed. An upgraded ShuffleNetV2 network is then developed by adopting the concepts of Channel Split, Channel Shuffle, and other techniques such as batch normalization and global average pooling. The proposed algorithm can extract image features and automatically identify the opening and closing of the eyes and mouth of the OOW. According to the PERCLOS criteria, the two features of the eyes and mouth are integrated to determine whether the OOW is fatigued. Experimental results show that the detection speed of the improved RetinaFace model improves from 9.33 to 22.60 frames/s. The detection accuracy and speed for the face detection are superior to the multi-task convolutional neural network. The upgraded ShuffleNetV2 network achieves over 99.50% accuracy in recognizing the states of eyes and mouth. The algorithm has an accuracy of 95.70% and a recall rate of 96.73% in identifying the fatigue state in a simulated ship bridge scenario, which are higher than Haar-like+Adaboost and MTCNN+CNN fatigue detection algorithmsused in practice. It only takes 0.083 s for the algorithm to complete the process, which indicates that the algorithm is capable of carrying out real-time detection.
- transport safety /
- fatigue detection /
- RetinaFace /
- ShuffleNetV2 /
- PERCLOS

HTML全文

图 1 算法流程图

Figure 1. Overall flow chart of algorithm

下载: 全尺寸图片幻灯片

图 2 RetinaFace人脸检测模型示意图

Figure 2. Schematic diagram of RetinaFace model

下载: 全尺寸图片幻灯片

图 3 深度可分离卷积示意图

Figure 3. Schematic diagram of depthwise separable convolution

下载: 全尺寸图片幻灯片

图 4 RetinaFace改进效果示意图

Figure 4. Improvement effect of RetinaFace model

下载: 全尺寸图片幻灯片

图 5 眼睛、嘴巴裁剪区域示意图

Figure 5. Cropped area of eyes and mouth

下载: 全尺寸图片幻灯片

图 6 ShuffleNet V2常用卷积块

Figure 6. Convolution blocks of ShuffleNet V2

下载: 全尺寸图片幻灯片

图 7 部分眼睛、嘴巴样本

Figure 7. Partial samples of eyes and mouth

下载: 全尺寸图片幻灯片

图 8 眼睛分类网络的训练曲线

Figure 8. Training curve of eyes classification network

下载: 全尺寸图片幻灯片

图 9 嘴巴分类网络的训练曲线

Figure 9. Training curve of mouth classification network

下载: 全尺寸图片幻灯片

图 10 眼睛、嘴巴状态识别结果

Figure 10. Recognition results of eye and mouth status

下载: 全尺寸图片幻灯片

图 11 对比实验结果

Figure 11. Compare experimental results

下载: 全尺寸图片幻灯片

图 12 部分疲劳检测结果

Figure 12. Partial results of fatigue detection

下载: 全尺寸图片幻灯片

表 1 不同模型在自建眼睛数据集上的分类对比

Table 1. Classification comparison of different models on self-built eye dataset

算法	准确率	速度/(ms/帧）
LeNet	95.53	12
VGG16	97.88	16
MobileNetV2	97.43	9
本文模型	99.71	7

下载: 导出CSV

表 2 不同疲劳检测算法结果对比

Table 2. Comparison of different fatigue detection algorithms

算法	N_t	N_d	N_p	精确率p/%	召回率R/%
Haar-like+Adaboost	92	96	75	78.12	81.52
MTCNN+CNN	92	94	84	89.36	91.30
本文算法	92	93	89	95.70	96.73

下载: 导出CSV

参考文献(20)

[1]	郭其顺. 世界海难事故的现状与原因分析[J]. 中国船检, 2000(3): 44-46. https://www.cnki.com.cn/Article/CJFDTOTAL-ZGCJ200003030.htm GUO Q S. Current situation and cause analysis of maritime accidents in the world[J]. China Ship Survey, 2000(3): 44-46. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-ZGCJ200003030.htm
[2]	张堃. 船舶机舱值班人员眼部视觉识别及疲劳状态监测[D]. 大连: 大连海事大学, 2019. ZHANG K. Eye visual recognition and fatigue monitoring of engineering watch-keepers on board ship[D]. Dalian: Dalian Maritime University, 2019. (in Chinese)
[3]	刘建蓓, 马小龙, 张志伟, 等. 基于心电分析的青藏高原驾驶人疲劳特性[J]. 交通运输工程学报, 2016, 16(4): 151-158. doi: 10.3969/j.issn.1671-1637.2016.04.016 LIU J B, MA X L, ZHANG Z W, et al. Fatigue characteristics of driver in Qinghai-Tibet Plateau based on electrocardiogram analysis[J]. Journal of Traffic and Transportation Engineering, 2016, 16(4): 151-158. (in Chinese) doi: 10.3969/j.issn.1671-1637.2016.04.016
[4]	LAWOYIN S, FEI D Y, BAI O. Accelerometer-based steering-wheel movement monitoring for drowsy-driving detection[J]. Journal of automobile engineering, 2015, 229(2): 163-173. doi: 10.1177/0954407014536148
[5]	刘梦佳, 赵建国. 基于SVM的疲劳驾驶人脸图像特征检测算法[J]. 半导体光电, 2020, 41(2): 278-282. https://www.cnki.com.cn/Article/CJFDTOTAL-BDTG202002026.htm LIU M J, ZHAO J G. Research on feature detection algorithm of fatigue driving face image based on SVM[J]. Semiconductor Optoelectronics, 2020, 41(2): 278-282. (in Chinese) https://www.cnki.com.cn/Article/CJFDTOTAL-BDTG202002026.htm
[6]	王迪. 基于人眼状态的疲劳检测算法研究与应用[D]. 成都: 电子科技大学, 2020. WANG D. Research and application of fatigue detection algorithm based on human eye state[D]. Chengdu: University of Electronic Science and Technology of China, 2020. (in Chinese)
[7]	黄玲, 洪佩鑫, 吴泽荣, 等. 基于防疫常态化的驾驶员疲劳状态检测方法[J]. 交通信息与安全, 2021, 39(4): 26-34. doi: 10.3963/j.jssn.1674-4861.2021.04.004 HUANG L, HONG P X, WU Z R, et al. A detection method for drivers' fatigue states based on normalization of epidemic prevention[J]. Journal of Transport Information and Safety, 2021, 39(4): 26-34. (in Chinese) doi: 10.3963/j.jssn.1674-4861.2021.04.004
[8]	张博雅. 基于深度学习的驾驶安全评估方法研究[D]. 西安: 西安科技大学, 2019. ZHANG B Y. Research on driving safety assessment method based on deep learing[D]. Xi'an: Xi'an University Of Science And Technology, 2019. (in Chinese)
[9]	LIU Z M, PENG Y X, HU W J. Driver fatigue detection based on deeply-learned facial expression representation[J]. Journal of Visual Communication and Image Representation, 2020 (71): 102-107.
[10]	郭佳伟. 基于计算机视觉的驾驶员异常行为识别与预警[D]. 大连: 大连海事大学, 2019. GUO J W. Driver's abnormal behavior recognition and early-warning based on computation vision[D]. Dalian: Dalian Maritime University, 2019. (in Chinese)
[11]	LI K N, GONG Y B, REN Z L. A fatigue driving detection algorithm based on facial multi-feature fusion[J]. IEEE Access, 2020, 8(11): 244-259.
[12]	陈军, 张黎, 周博, 等. 基于级联卷积神经网络的驾驶员分心驾驶行为检测[J]. 科学技术与工程, 2020, 20(14): 5702-5708. doi: 10.3969/j.issn.1671-1815.2020.14.037 CHEN J, ZHANG L, ZHOU B, et al. Driver distracted driving behavior detection based on cascaded convolutional neural network[J]. Science Technology and Engineering, 2020, 20(14): 5702-5708. (in Chinese) doi: 10.3969/j.issn.1671-1815.2020.14.037
[13]	DENG J K, GUO J, VERVERAS E, et al. Retinaface: Single-shot multi-level face localisation in the wild[C]. The IEEE conference on computer vision and pattern recognition, Seattle: IEEE, 2020.
[14]	GAVAI N R, JAKHADE Y A, TRIBHUVAN S A, et al. MobileNets for flower classification using TensorFlow[C]. 2017 International Conference on Big Data, IoT and Data Science (BID), Pune: IEEE, 2017.
[15]	YANG S, LUO P, LOY C C, et al. Wider face: A face detection benchmark[C]. The IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas: IEEE, 2016.
[16]	ZHANG K P, ZHANG Z P, LI Z F, et al. Joint face detection and alignment using multitask cascaded convolutional networks[J]. IEEE Signal Processing Letters, 2016, 23(10): 1499-1503. doi: 10.1109/LSP.2016.2603342
[17]	MA N N, ZHANG X Y, ZHENG H T, et al. ShuffleNet V2: practical guidelines for efficient CNN architecture design[C]. European Conference on Computer Vision, Munich: ECCV, 2018.
[18]	IOFFE S, SZEGEDY C. Batch normalization: Accelerating deep network training by reducing internal covariate Shift[C]. International Conference on Machine Learning, Lille: PMLR, 2015.
[19]	PENG Y, DONG Y, CHENG D. Design and implementation of a driver's eye state recognition algorithm based on PERCLOS[J]. Chinese Journal of Electronics, 2014, 23(4): 669-672.
[20]	ABTAHI S, OMIDYEGANEH M, SHIRMOHAMMADI S, et al. YawDD: A yawning detection dataset[C]. The 5^th ACM multimedia systems conference, Singapore: ACM, 2014.