Improving the performance of license plate detection using deep neural networks on diverse datasets

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 09 Issue: 09 | Sep 2022 www.irjet.net p-ISSN: 2395-0072

Improving the performance of license plate detection using deep neural networks on diverse datasets

1 Faculty of Information Technology, Hanoi Architectural University, Hanoi, Vietnam 2 Vitech Institute, Hoang Mai, Hanoi, Vietnam ***

Abstract – The detection of license plates (LPs) is a crucial step to develop the intelligent traffic management systems. Several challenges exist for the detection of LPs such as the high variation of the geometry of LPs or the frequent variation in the conditions of LP image acquisition. The paper proposed improvements for the detection of LPs. Firstly, advanced deep neural networks are employed to detect LPs accurately. For training the deep neural networks efficiently, the data augmentation techniques are applied. Then, the strategies of the deployment and testing of the deep neural networks on various hardware plat- forms are proposed to improve the inference time of LP detection. We have performed the evaluation on two public datasets (Vietnamese license plate detection and Kaggle datasets).Theperformance comparison (the detection accuracy and execution time) with existed methods on various hardware platforms shows the effectiveness of the proposed method.

Key Words: Deep neural network, hardware platform, data augmentation, license plate detection.

1.INTRODUCTION

Thelicenseplatedetectionisoneofthemostimportant steptodeveloptheintelligenttrafficmanagementsystems. TheLPdetectionhasbeenwellresearchedinrecentyears [1]–[3],however,theaccuracyofthedetectionremainsstill lowandneedstobeimproved.Licenseplatedetectionaims toobtainpositioninformationofLPsinimages.Fig-1shows examplesofcapturedlicenseplateimages.AsshowninFig1,severalchallengesofthelicenseplate detectioncanbe describedas follows:



Duetoconditionsofimageacquisition,license plateimagesmaybeblurred,curvedandskew. Thecharacteristics cause many difficulties in the detection of LPs.

In recent years, embedded systems (e.g. Jetson Nano, Rasp- berry Pi3, NPU VIM3 toolkit) have significantly advanced

[4] sothattheDeep Neural Network (DNN) models can beimplementedonthesystems.Theimplementationand deploymentofDNNsonthesystemsallowtoobtain high accuracywithlowcomputationaltimeforobjectdetection task.Taking theadvantagesofthecomputationalresources oftheembeddedsystems,thepaperpresentsthedetection ofLPsapplyingtheYOLOv4[5]ontheNPUVIM3toolkit to achievethetrade-offbetweenthedetectionaccuracyand thecomputationaltime.Thecontributionsofthepaperare twofold:

(1) Variousdeepneuralnetworks[5]areemployedand optimizedtodetectLPsaccurately.Comparedtotraditional methods,theLPdetectionusingtheneuralnetworkmodel gainshigheraccuracy.Moreover,toimprovethedetection accuracy, the augmentation technique is applied in the trainingofthenetwork.

(2) Thetraineddeepneuralnetworksaredeployedon differenthardwareplatforms[6].Thedeploymentallowsto obtain more efficiency in term of inference time and financialcostoftheLP detection.

Therestof the paper is organized as follows. Section 2 reviews related studies of the work. In Section 3, the proposedmethodisdescribed.InSection4,wedescribeand analyzeexperimentalresults.Finally,Section5providesthe conclusion of and the future work.

2. RELATED WORKS



ThehighvariationofthetypesandsizesofLPs causemany errors in the detection of LPs.

This section reviews important related work of the licenseplatedetection.Bothtraditionalanddeeplearning approachesfortheLPdetectionareanalyzedinthesection. Theoverviewoftraditionalanddeeplearningapproaches fortheLPdetectionareshowninFig-2.



ThecomplexbackgroundsofcapturedLPsmay causetheambiguities in the detection of LPs.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 09 Issue: 09 | Sep 2022 www.irjet.net p-ISSN: 2395-0072

roughlyestimatedusingthehorizontalprojectionofvertical edge in the obtained images after the wavelet transform. Finally,thepreciselocationofLPsisobtainedusingseveral threshold values of the column search method. The experimentswereperformedonsmalldataset(315images) and the detection accuracy is 92.4%. The method is not robust for images with complex backgrounds and the determinationofthresholdsaredifficultforlargedatasets. Theworkin[1]appliedtheHistogramofOrientedGradient (HOG)andalinearSupportVectorMachine(SVM)todetect BrazilianLPs. Thedetectionaccuracyofthemethodislow (78%) on a small dataset (377 images). The work in [2] proposed a multi stages method to detect LPs. Firstly, the connectedcomponentsofLPimageswereextracted.Thenthe Haar-likefeatureandtheAdaboostalgorithmareappliedto detect LPs. The method was evaluated on the Korean LP datasets containing 1800 images. The obtained detection accuracyofthemethodis98.4%. Theworkin[8]proposed an approach based on the linear SVM and color saliency featurestodetectLPs.TheapproachescandetectLPswith high accuracy in some circumstances, however, these methodsconsistofmuti-stepsthatarecomplicatedforreal timeapplication.

2.2 LP detection using Deep Neural Networks (DNNs)

In recent years, the DNNs [3], [9] have shown the high performanceintheLPdetectionandrecognitiontasks.The work [10] proposed a method to solve the detection and recognitionof LPs in natural scene images. In the work, a unifieddeepneuralnetworkmodelisproposedtodetectand recognize LPs simultaneously. A large datasets containing 46000 Chinese car LPs are collected to train and test the network. The work in [3] applied the lightweight convolutional neural networks (CNNs) to detect and recognizeLPs.Theworkin[4]implementedtheTinyYOLOv3 [11]architecturetodetectLPs.ThenthedetectedLPswere recognizedapplyingasmallCNNnetwork.Thedetectionand recognition are performed on the Raspberry Pi3 with the PiNoIRV2 camera. The system was evaluated on small datasetscontainingabout610PLimages.Theaccuraciesof thedetectionandrecognitionofLPsare99.37%and97.00%, respectively.TheSSDnetwork[12]wasappliedtodetectLPs inpicturesandvideosonJetsonNanoplatform[13]withthe 39 Frame per second (FPS).Asanalyzedabove,thereare relativelyfewmethodshaveattemptedtodetectLPsusing DNNs deployed on various hardware platforms NPU hardwareplatforms(e.g.,JetsonNano,NPUVIM3andserver withGraphicProcessingUnit(GPU)).

2.1 Traditional methods for LP detection

Fordecades,severalmethodshavebeenproposedtodetect LPs.Earlyresearchesofthefieldappliedthetraditionalimage processingandhandcraftedfeatureextractiontechniquesto detectLPs Themethodin[7]appliedthetraditionalimage processing to detect LPs. Firstly, the wavelet transform is appliedforLPimages.Then,theLPpositioninformationis

3. PROPOSED SYSTEM

Theprinciplestepsofourproposedsystemaredescribedin Fig-3.Theframeworkconsistsofthefollowingsteps:

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 09 Issue: 09 | Sep 2022 www.irjet.net p-ISSN: 2395-0072

Fig- 2:Traditionalanddeeplearningapproachesfor theLPdetection.

Fig- 4:YOLOv4architectureforthedetectionofLPsusing YOLOv4.

Fig-4illustratestheYOLOv4architectureforthedetectionof LPs.Fig-5describesdetailstructureofDarknet-53

Fig- 3:Flowchartoftheproposedsystem.

(1) Step 1: InputLPimagesarenormalizedtothesizeof 416x416 for the training and testing DNNs (YOLOv4 network).

(2) Step 2: The YOLOv4 [5] is trained using training datasetsofimagescontainingLPs.Thenetworkconsistsof thefollowingsubnetworks:

• Cross Stage Partial Network (CSPDarknet-53) network

[14]playsanimportantroleforextractingrichfeaturesof inputimages.

• SpatialPyramidPoolingnetworks(SPPnet)consist ofdifferentmax-poolinglayerstoincreasethereceptivefield ofthenetwork.

• Path Aggregation Network (PANet) consists of Down-SamplingandUpSamplinglayerstoextractfeatures repeatedly.

• Three YOLO heads are employed to detect LPs of differentsizes.

Fig- 5:DetailstructureoftheDarknet-53network

Network that is used as the backbone of the YOLOv4. The networkistrainedontheserverequippedwithNvidiaTesla V100(16GBVRAM),CPU4cores,and8GBRAMmemory.The initiallearningrateissetas0.001.TheAdamsolvealgorithm [15]wasutilizedfortrainingthenetworkwith60epochs,in whichthemomentumissetat0.93.

Toimprovethedetectionaccuracy,thedata augmentation techniqueisapplied.Dataaugmentationencompassesasuite of techniquesthat enhance the size and quality oftraining datasetssuchthatbetterDNNmodelscanbebuiltusingthem [16].Then,theaugmenteddatasetsareusedtotrainDNNs. Thefollowingtechniquesareappliedtoaugmenttheoriginal imagedatasets:

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 09 Issue: 09 | Sep 2022 www.irjet.net p-ISSN: 2395-0072

• Create a translation transformation of images.

• Create a randomized rotation transformation of images.

• Random scaling images by 10 percent.

• CreateHueJitterof images.

• Addsyntheticnoisesto images.

Fig- 6 illustrates the data augmentation using image processingtechniques.

Fig- 7:ValueofthelossfunctionoftheYOLOv4duringthe trainingprocess.

ThelossfunctionoftheYOLOv4[17]networkisthesum of thelocalization,confidenceandtheCompleteIntersection overUnion(CIoU)lossfunctions.Thelocalizationlossfunctionmeasuresthedifferencesbetweenthepredictedandthe ground-truthboundingboxes.Theconfidencelossfunction is calculated based on the classification results of the network.CIOUlossfunctioniscomposedoftheoverlaparea, center point distance, and aspect ratio, and is used to fit positioninginformation.Valueofthelossfunctionduringthe trainingtheYOLOv4networkisillustratedinFig-7.Theloss function value sharply decreases from 0 to 200 iterations andthevaluealmostconvergesfrom500to600iterations.

Fig-6:Illustrationofthedataaugmentationusingimage processing

TrainingdatasetsconsistofLPimagesandboundingboxes oftheLPs.Thelossfunctionofthetrainingthenetworkis definedasfollows:

Loss= + + (1)

Fig-8:FlowchartofthequantizationprocessinNPUVIM3 hardwareplatform

Fig-9:HardwareinformationofNPUVIM3toolkit[6].

(3) Step 3: To deploy the trained DNNs on low computational hardware platforms, the quantization process is applied. The quantization process aims to convertthetrainedDNN models(describedinstep2)to intermediatefilesandexportthefilestocasecodessothat

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 09 Issue: 09 | Sep 2022 www.irjet.net p-ISSN: 2395-0072

the models can be executed on the NPU tool kit. The processisillustratedinFig-8.Fig-9illustratesarchitecture oftheNPUVIM3toolkit.Fig-10demonstratesanexample ofthequantizationfrom floating point to integer numbers.

Fig-10:IllustrationofthequantizationprocessinNPU VIM3hardwareplatform.

(4) Step 4: The LP detection is performed on the various platforms: server with graphical processing unit (GPU),JetsonNanoandNPUVIM3toolkit.Inthephase,the trained YOLOv4 is employed to detect LP images that are normalizedtothesizeof416x416.Detailinformationofthe platformsaredescribedasfollows:

The NPU VIM3 tool kit [6] consists of the following components:

• Processors of Cortex A73 performance-cores (2.2GHz),andx2CortexA53efficiency-cores(1.8GHz).

• Memoryof2GBRam16GBeMMC.

• The tool kit supports an onboard 5.0 Trillion OperationsperSecond(TOPS)computationalresource[6] fortestingthedetectionofLPsusingtrainedDNNs.

• TheoperatingsystemoftoolkitisUbuntu20.04.

TheserverusedfortrainingDNNsconsistsofthefollowing information:

• GPUsGeforceGTX1050.

• 8GBofRAM.

• Corei5-9300Hprocessor.

• Windows10operatingsystem.

HardwareinformationoftheNvidiaJetsonNanoisdescribed asfollows:

• CPUQuad-coreARMCortex-A571.43GHz.

• GPU128-coreNVIDIAMaxwell.

• 2GBRam64-bitLPDDR4.

4. EXPERIMENTAL RESULTS

4.1 Datasets and evaluation metric

4.1.1 Datasets

Weperformedtheframeworkontwopublicdatasets:the Vietnamese LP [18] and Kaggle car LP [19] datasets. The KagglecarLPdatasetcontains400imageswithboundingbox annotationsofthecarlicenseplateswithintheimage.Each image consists of minimum 1 and maximum of 4 LPs. Annotations are provided in the PASCAL VOC format. The VietnamLPdatasetconsistsof8000Vietnameseimages(e.g. car, motor plates). LPs in each image are annotated with bounding boxes information and classes of object. The trainingandtestingdatasetsareusedfortrainingandtest DNNs. To improve the training accuracy, we applied the augmentationtechniquesfortrainingdataset.Forfairtesting the DNNs, the testing dataset is not augmented. Statistic information of LP datasets before and after using the augmentationtechniquesisshowninTable-1.

4.1.1 Evaluation metric

TheIntersectionoverUnion(IoU)metric[20]wasapplied intheworkfortheperformanceevaluationofLPdetection. Themetricisdefinedasfollows: (2)

where is the intersection of the predicted and ground- truth bounding boxes of LPs. denotes the unionofthepredictedandground-truthboundingboxesof LPs. Adetectioniscorrectif .Toobtainthetrust performanceevaluation,themeanaverageprecision(mAP)is alsoappliedinthework.TheLPdetectionisperformedand evaluated ten times and the mAP value of the detection is obtained.

Table -1: StatisticinformationofLPdatasetsbeforeand afterapplyingtheaugmentationtechnique

Datasets

NumberofLPimages (Vietnamesedataset)before applyingaugmentation

NumberofLPimages (Vietnamesedataset)after applyingaugmentation

NumberofLPimages(Kaggle cardataset)beforeapplying augmentation

NumberofLPimages(Kaggle cardataset)afterapplying augmentation

2022, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal

Training

Testing

7000 1000

35000 1000

200 200

1000 200

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 09 Issue: 09 | Sep 2022 www.irjet.net p-ISSN: 2395-0072

4.2 Performance evaluation

To evaluate the performance of the proposed method, we compared the detection accuracy with other DNNs (SSD, YOLOv3networks)onvarious hardwareplatforms(Jetson Nano and GPUs). The comparison of the LP detection accuracy isshowninTable-2,3,4and5. Asshowninthe tables,theYOLOv4achievesthehighestdetectionaccuracy compared to the SSD and YOLOv3 networks. Table -6 indicates thatthedetectionaccuracyofLPsontheNPUVIM3 tool kit is competitive with that on the Jetson Nano. The detection ofLPsontheGPUsallowstoobtainthehighest score. The results come from the fact that the hardware resources oftheGPUsarebetterthanthoseofNPUVIM3tool kit.TheJetsonNanosystemssupportthe16floatingpoint formatprecision,whereastheNPUVIM3toolkitsupportsthe 8integerone.Thedifferences of the hardware resources and thenumberprecisionaffectthedetectionaccuracyon GPUs,JetsonNanoandNPUVIM3toolkit.

Table -2: LPdetectionresultsusingtheYOLOv4withand withoutusingthedataaugmentationtechniqueson VietnameseLPdatasets( ).

Methods mAP

YOLOv4withoutapplyingthedata augmentationtechnique 93.75% YOLOv4 with applying data augmentation technique 94.80%

Table -3: LPdetectionresultsusingtheYOLOv4withand withoutusingthedataaugmentationtechniquesonKaggle LPdatasets( ).

Method mAP

YOLOv4withoutapplyingthedata augmentationtechnique 90%

YOLOv4 with applying data augmentation technique 92%

Table -4: LPdetectioncomparisonusingvariousmethods withVietnameseLPdatasetsonserverwithGPU

Method mAP

SSDMobilenetV2[12] 89.30% YOLOv3 92.50%

Proposed method based on YOLOV4 94.80%

Table -5: LPdetectioncomparisonusingvariousmethods withVietnameseLPdatasetsonNPUVIM3toolkit

Method mAP

SSDMobilenetV2[12] 89.30% YOLOv3 92.50%

Proposed method based on YOLOV4 94.80%

Table -6: LPdetectioncomparisonusingvariousmethods withVietnameseLPdatasetsonvarioushardware platforms

Platform mAP

GPUGeforceGTX1050 96.11% JetsonNano[18] 94.39% NPU Khadas VIM3 94.80%

ExamplesoftheLPdetectionusingtheYOLOV4areshownin Fig- 11 and 12. The results illustrate that the proposed systemcandetectLPsaccuratelyindifferentcases(i.e.,skew and tiny LPs). An example of the detection using the SSD network is shown in Fig-13. The detection using the SSD networkisfasterthanthatoftheYOLOv4network.However, thedetectionaccuracyoftheSSDislowerthanthatofthe YOLOv4network.

Toevaluatetheexecutiontimeoftheproposedsystemand other hardware platforms, we compared the average inference time (in millisecond) of the YOLOv4 on various hardwareplatforms.Fig-14showstheinferencetimeofthe LPdetectiononusingtheYOLOv4testingdatasetson

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 09 Issue: 09 | Sep 2022 www.irjet.net p-ISSN: 2395-0072

Fig-12:ExamplesofthedetectionoftinyandblurLPs usingYOLOv4

(b) ExecutiontimeoftheLPdetectionusingthe YOLOv4

Fig-15:Comparisonofaverageexecutiontime(in millisecond)oftheLPdetectionontheVietnamesetesting datasetsonvarioushardwareplatforms.

Fig-13:ExamplesofthedetectionoftinyandskewLPs usingtheSSDnetwork

various hardware platforms. The Figure shows that the inference time on the NPU VIM3 tool kit is significantly improvedincomparisonwiththatontheJetsonNano.The betterresultscomefromthedeploymentprocessofDNNs andthesupportofthe8integerpointformatprecisionofthe NPUVIM3toolkit.ComparedtotheYOLOv4,theinference time of the SSD Mobilenet is much reduced because the architectureoftheSSDMobilenetissimpler.

4.3 Discussion and error analysis

(a) ExecutiontimeoftheLPdetectionusingtheSSD Mobilenet

As our results, the detection of LPs using the YOLOv4 on GPUsachievesthehighestaccuracythankstothepowerful hardware resources. The detection accuracy on the VietnamesedatasetishigherthanthatontheKaggledataset becauseVietnamesedatasetislargerthantheKaggle.The dataaugmentationallowstoimprovethedetectionaccuracy by 1.05% and 2% on Vietnamese and Kaggle datasets, respectively.ThedetectionaccuracyofLPsontheNPUVIM3 achievescompetitiveresultscomparedtothatontheJetson Nano.However,theexecutiontimeofthedetectiononthe NPUVIM3toolkitismuchreducedthanthatontheJetson Nano.

Some remained detection errors can be caused by the followingreasons:

1. The quality of some LP images is too low. Some captured images can be blurred. Some images containmanynoisesandshadow.

2. SeveralobjectsthataresimilartoLPsintheimages withcomplexbackgroundscausedtheambiguities ofthedetection.

5. CONCLUSIONS AND FUTURE WORKS

ThepaperhaspresentedtheimprovementofLPdetection usingdeepneuralnetworkstrainedondiversedatasets.The employmentoftheadvancedneuralnetworkallowstoobtain high accuracy of the LP detection. The data augmentation techniquesinthetraining of deep neural networks show theimprovementsofthedetectionaccuracy.Moreover,the strategy of deployment of the trained network on various hardwareplatformsachievesthesignificantimprovementof the execution time. The performance comparison with conventional methods on the two public datasets (Vietnamese and Kaggle LP detection datasets) and on various hardware platforms illustrates the effectivenessof theproposedmethod.Inthefuture,thedetectionresultscan be integrated with the recognition of LPs in several real

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 09 Issue: 09 | Sep 2022 www.irjet.net p-ISSN: 2395-0072

applications such as intelligent transportation systems. Moreover,otheradvancedCNNscanbecombinedtoimprove thedetectionaccuracyoftheLPdetection.

REFERENCES

[1]R. Prates etal., “Brazilianlicense plate detection using histogram of oriented gradients and sliding windows,” in International Journal of Computer Science Information Technology(IJCSIT),vol.5,no.6,2013.

[2]M.Sarkeretal.,“Afastandrobustlicense platedetection algorithm based on two-stage cascade adaboost,” in KSII TransactionsonInternet andInformationSystems,vol.8,no.10,2014.

[3]P.Anh,“Effectivedeepneuralnetworksforlicenseplate detection and recognition,” in Vis Comput. https://doi.org/10.1007/s00371-021-02375-0,2022.

[4] D. M. F. Izidio et al., “An embedded automatic license platerecognition system using deep learning,” in 2018 VIII Brazil- ian Symposium on Computing Systems Engineering (SBESC), DOI: 10.1109/SBESC.2018.00015, 2018.

[5] A. Bochkovskiy et al., “YOLOv4: Optimal speed and accuracyofobjectdetection,”inarXiv:2004.10934,2020.

[6] “Npu kit vim3,” in https://docs.khadas.com/linux/vim3/Hardware.html,2022.

[7] C.-T. Hsieh et al., “Multiple license plate detection for complexback-ground,”in19thInternationalConferenceon AdvancedInformationNetworkingandApplications,2005.

[8]Y.Yuanetal.,“Arobustandefficientapproachtolicense platedetection,”inIEEETransactionsonImageProcessing, vol.26,no.3,2017.

[9]S. Z. Masood et al., “License plate detection and recogni- tion using deeply learned convolutional neural networks,”inarxiv.org/abs/1703.07330,2017.

[10] H. Li et al., “Toward end-to-end car license plate detectionandrecog-nitionwithdeepneuralnetworks,”in IEEE Transactions on Intelligent Transportation Systems, vol.20,no.3,2019.

[11] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,”inarXiv:1804.02767,2018.

[12] W. Liu et al., “Ssd: Single shot multibox detector,” in arXiv:1512.02325,2016.

[13]“Jetsonnano,”inhttps://github.com/dusty-nv/jetsoninference,2022.

[14] C.-Y. Wang et al., “Cspnet: A new backbone that can enhance learning capability of cnn,” in https://arxiv.org/abs/1911.11929,2019.

[15]D.Kingma andJ.Ba,“Adam: Amethodforstochastic optimization,” inarXivpreprintarXiv:1412.6980,2014.

[16]C.ShortenandT.Khoshgoftaar,“Asurveyonimagedata augmentationfordeeplearning,”inJournalofBigData6(1), 2019.

[17]C.Guoetal.,“ImprovedYOLOv4-tinynetworkforrealtimeelectroniccomponentdetection,doi:10.1038/s41598021-02225-y,”inScientificReports,2021.

[18] “License plate dataset,” in https://github.com/winter2897/Real-time- Auto-LicensePlate-Recognition-with-Jetson-Nano,2022.

[19] “Car license plate dataset,” in https://www.kaggle.com/datasets/andrewmvd/car-platedetection,2022.

[20]M.Everinghametal.,“Thepascalvisualobjectclasses (voc)challenge,”inIntJComputVis,vol.88,2010.

2022, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal | Page1309