1. INTRODUCTION
Due to the rising number of cars, the management of intelligent traffic has become essential to identify and monitor vehicles that violates the law guiding queue management systems. Intelligent traffic gathers information about the vehicle so that it can be used in monitoring systems and security. For the identification of the license plate, there are two approaches: manual recognition by humans and automatic recognition by computer programs. As mentioned by Abod (2015), manual plate identification is useful, however automatic recognition are more advantageous because under different situations, it would be easier to recognize a license plate.
Automatic number plate recognition (ANPR) has been the subject of interest for decades, especially those who work in managing queue systems using image processing and artificial intelligence. Previous studies about license plate recognition and tracking can be broadly considered in two parts: license plate localization and license plate classification (Abbass & Marhoon, 2021; Altyar, Hussein, & Tawfeeq, 2023). Studies that have conducted research on the former have focused on identifying numerous attempts to solve the problem of possible license plate area detection from an image or video (Hashem, Abbas, & Mohamad, 2023). Different modern methods have applied various approaches, techniques, and algorithms for image processing to design their license plate detection systems. Kaur and Kaur (2014) proposed a method for localization of Indian license plates. They used bilateral filtering for noise removal, adaptive histogram equalization for contrast enhancement, and morphological operations and Sobel edge detection were used for plate area detection. In related studies, Chowdhury, Khan, and Uddin (2017) proposed a method for plate localization, where they used a sliding concentric window as a segmentation technique to separate the region of interest, which is the plate, and then further utilized morphological image processing to localize the plate inside the image. However, these approaches do not ensure region of interest segmentation for vehicles with a dark background license plate, and white colour of characters.
To develop an approach that would be usable regardless of the colour or size differences, Asif et al. (2017) used the YUV colour space to identify the car's backlight region. In addition to the use of distinctive histogram method to determine a map of heuristic energy in the relevant area, including the dense edge section. Other studies on license plate localization are Rabbani et al. (2018), Omran and Jarallah (2018), and Farag, El Din, and El Shenbary (2019). In the separate works of these authors, improved visuals were achieved by using a haze removal approach with application of the Wiener filter to remove the noise, combination of morphological techniques and vertical Sobel edge detection, and discrete wavelet transform. However, the limitations of the studies in this paragraph include some methods being limited to viewing the vehicle from behind and inconsistency in results when other regions have the same license plate aspect ratio as the detected zone.
On the other hand, studies on the second broad part of license plate recognition and tracking, which is the license plate classification, have been made to the identification scheme for license plates. Hidayah, Akhlis, and Sugiharti (2017) determined the significance of each letter and number in the plate by first utilizing Otsus' Method to extract plate image and then converted it to binary. Whereas Wang, Bacic, and Yan (2018) developed a template matching method for determining plate characters by employing secondary positioning method, while the phase correlation and cross-correlation structured methods were reported to be effective by Sharma (2018). Although, there were success recorded in these studies, some of the shortcomings were that some studies considered all of their dataset images being captured from the rear, they didn't address how plate rotation plays a role in the recognition process, and their approach could only distinguish a limited number of characters.
ANPR systems are critical for various applications, including traffic monitoring, law enforcement, and access control. Although, it is worth noting that in each country, vehicles have unique plate number properties written on their license plates from the front or rear side with different types, colour, scale, font styles, aspect ratios (which is a ratio of length to width or width to height), and areas. Specifically, in Iraq, the effectiveness of standard ANPR systems is significantly hindered by the wide variation in license plate designs. The plates vary in font styles, scales, and even the same plate for the same vehicle can have multiple aspect ratios (the front plate aspect’s ratio differs from the back plate), which are not standardized across the region. This diversity makes it difficult for conventional ANPR technologies, which typically rely on predefined templates or simpler image processing techniques, to accurately detect and interpret the plates.
To overcome the limitations of existing ANPR systems in Iraq, this article develops a robust solution using advanced artificial intelligence techniques. The process consists of three steps: detection, segmentation, and recognition. In detection, it detects the license plate location. In segmentation, it depends on what is segmented from the car plate from the original image and segmenting the plate into characters. After that is the cropped image recognition where systems that recognize the license plate have various applications like monitoring traffic systems, tool ticketing, and parking fees.
The rest of the article is organised thus: the next section discusses the theoretical background and development of the vehicle tracking system consisting of the localization and classification. Thereafter, the results of the system and tests random vehicle images are discussed and conclusions and suggestions for future work are given.
Theoretical Background
Distinctive features of Iraqi license plates, such as their numbering system, background noise, plate size, camera's field of view, scene complexity, unwelcome features, plate color, plate position, and plate tilt, varies. Classification presents a number of difficulties, including plates of varying sizes, short datasets that cause overfitting, and the time required to analyze the model of a convolutional neural network (CNN) during training. This section will introduce CNNs, transfer learning, object detection and its uses.
A description of a CNN is that it is a regularized type of feed-forward neural network with the independent ability to learn feature engineering on its own using filters or kernel optimization. CNNs are also known as Shift Invariant or Space Invariant Artificial Neural Networks (SIANN) and this is founded on the shared-weight architecture of the convolution filters or kernels. These filters are known to slide along input features, while providing feature maps, which are basically translation-equivariant responses (Zhang et al., 1990).
The layers of a CNN include an input layer, hidden layers (which include one or more layers that perform convolutions) and an output layer. Using the layer's input matrix, the hidden layer specifically includes a layer that computes a dot product of the convolution kernel. This product is usually the Frobenius inner product, and its activation function is commonly the rectified linear activation function. Notably, a feature map is generated by the convolution operation as the convolution kernel slides along the input matrix for the layer, which in turn contributes to the input of the next layer. Thereafter, other layers such as the pooling layers, fully connected layers, and normalization layers, follow suit. A typical CNN architecture is shown in Figure 1 below.
Figure 1: ypical CNN Architecture
Another important theoretical concept in machine learning is the transfer learning concept. In transfer learning, there is a reuse of knowledge learned from a task for boosting performance on a related task (West et al., 2007). An example of transfer learning follows from an image classification scenario, where knowledge gained in the learning process of car recognition could be similarly adapted in the recognition of trucks. The reuse and transfer of information from tasks that have been previously learned to new tasks has the prospective to provide significant improvement in learning efficiency. This process is related to multi-objective optimization cost-sensitive machine learning as a result of transfer learning’s use of training with various objective functions. An illustration is given in Figure 2 below.
Figure 2: Illustration of Transfer Learning
Furthermore, the idea of object detection is a computer technology related to computer vision and image processing. Object detection could be applied in various fields of computer vision, such as video and retrieval surveillance. In addition, pedestrian detection and face detection are well-researched domains of object detection. This involves considering digital images and videos, and then detecting instances of semantic objects of a certain class such as cars, buildings, or humans. buildings, or cars. Object detected domains a wide application in computer vision tasks such as vehicle counting (Ala et al., 2020), image annotation (Guan, 2017), activity recognition (Wu et al., 2007), face detection, video object co-segmentation, and face recognition. It is also adopted in the tracking of objects, such as ball tracking in a football match, cricket bat movement tracking, or human tracking in a video. In most cases, a different data distribution is used to sample the test images thereby increasing the difficulty of the object detection task significantly (Oza et al., 2021). This translates to a domain gap between test and training data and studies have addressed this challenge through many unsupervised domain adaptation approaches as seen in studies by Oza et al. (2021), Mehran et al. (2019), Petru et al. (2021), and Maximilian and Andreas (2022). Although, Jun-Yan et al. (2020) proposed a simple and direct solution to reduce the domain gap by applying cycle-GAN which is an image-to-image translation approach.
Concept and Methods
Object class detection uses special features in accordance with the unique features possessed by every object class that helps in class classification. For example, when looking for circles, we seek objects that are at a particular distance from the center. In like manner, when the shape under consideration is a square, what is sought are objects that have equal side lengths and are perpendicular at corners. Whereas features like skin color and distance between eyes can be considered for face identification where eyes, nose, and lips can be found.
In general, the approaches adopted in object detection can either be classified as non-neural network-based or neural network-based approaches. For the former, it becomes essential to begin with a definition of the features and then do the classification using a suitable technique such as support vector machine (SVM). On the other hand, the latter (neural network-based techniques) have the ability to conduct end-to-end object detection without encountering the rigour of specific definition of features. This technique is typically based on CNNs which is the category of the YOLOv8 technique applied in this article.
YOLOv8 is the newest state-of-the-art YOLO model that can be used for object detection, image classification, and instance segmentation tasks. YOLOv8 was developed by Ultralytics, who also created the influential and industry-defining YOLOv5 model. YOLOv8 includes numerous architectural and developer experience changes and improvements over YOLOv5. YOLOv8 is an anchor-free model which means that it predicts directly the center of an object instead of the offset from a known anchor box. Anchor boxes were a notoriously tricky part of earlier YOLO models, since they may represent the distribution of the target benchmark's boxes but not the distribution of the custom dataset. Anchor free detection reduces the number of box predictions, which speeds up Non-Maximum Suppression (NMS), a complicated post processing step that sifts through candidate detections after inference. Figures 3 and 4 give a visualization of an anchor Box in YOLO and the detection head for YOLOv8.
Figure 3: Visualization of an anchor Box in YOLO
Figure 4: The detection head for YOLOv8
In terms of accuracy, YOLOv8 research was primarily motivated by empirical evaluation on the Common Objects in Context (COCO) benchmark. As each piece of the network and training routine are tweaked, new experiments are run to validate the changes effect on COCO modeling. COCO is the industry standard benchmark for evaluating object detection models. When comparing models on COCO, we look at the mAP value and FPS measurement for inference speed. Models should be compared at similar inference speeds. The image in Figure 5 below shows the accuracy of YOLOv8 on COCO, using data collected by the Ultralytics team and published in their YOLOv8 README.
Figure 5: YOLOv8 COCO evaluation accuracy of the art for models at comparable inference latencies
The difficulties discussed in many literature reviews on license plate localization and classification have an effect on the design system for doing so. The proposed international license plate classification system is up to the task of overcoming the aforementioned obstacles, notably those related to maintaining high image quality and avoiding de-skewing license plates, both of which have an impact on how those plates are categorized. To address these issues, the methodology begins with dataset creation by assembling and annotating a comprehensive dataset that represents the wide variety of license plate designs found in Iraq. The dataset images are collected under different conditions (day and night images, different weather conditions, …etc). Then, the next phase implements a model adaptation by adapting the YOLOv8 convolutional neural network architecture to handle the high variability in Iraqi license plate designs. This involves training the model to recognize a wide range of non-standard fonts, scales, and aspect ratios effectively. Thereafter, a two-stage recognition process was implemented, where the first model focuses solely on detecting the precise location and boundary of the license plates irrespective of their design variability. Once the plates are detected and isolated, the second model classifies the textual and numerical information on the plate. Furthermore, transfer learning is utilized to accelerate the training process and improve the accuracy of the models by leveraging pre-trained neural networks that have already learned significant features from extensive, diverse datasets. The aim is to achieve high accuracy and confidence levels in both detection and classification stages of the project, ensuring the system is practical and reliable for real-world applications.
Methodology
This section showcases the methodology employed in the development of automatic number plate recognition (ANPR) system, designed specifically to address the unique challenges posed by the diverse and non-standard license plates in Iraq. The methodology is crucial in understanding how the theoretical aspects aforementioned are practically applied to achieve the desired outcomes.
The process was separated into phases:
- Data collection.
- Labeling images for the localization model.
- Training the localization model.
- Implementing and using of the localization model.
- Labeling images for the classification model.
- Training the classification model.
- Creating a pipeline of models.
Data Collection
A dataset of about 2000 images of Iraqi cars from an Iraqi private company called Ardh Al-Yamama were obtained, but most of the cars in this dataset was of a single type of plates (cargo plates). Also, about 200 image of Iraqi cars that were offered for sale from the internet specifically from opensooq.com, and Facebook’s marketplace were downloaded. A sample of the dataset is given in Figure 6.
Figure 6: A sample of dataset
Labeling Images For The Localization Model
An open-source tool provided by a third party that is called Yolo-Label was used for labeling and normalizing the labels of plates by creating a single detection class and calling it “plate” (See Figure 7).
Figure 7: Labeling plates using Yolo-Label tool
Training The Localization Model
The model was trained using the colab service provided by google for researching AI and ML, alongside a newly developed python script. The detection head of the YOLOv8n model was removed by utilizing transfer learning and a new head was trained to detect Iraqi plates’ locations in an image. The dataset structure is assumed to be organized in this way for the training script to use.
Dataset
│
└───images
│ └───train
│ │
│ image_1.jpg
│ │
│ image_2.jpg
│ │
│ ...
│ │
│ └───val
│ │
│ val_image_1.jpg
│ │
│ val_image_2.jpg
│ │
│ ...
│ │
└───labels
│ └───train
│ │
│ image_1.txt
│ │
│ image_2.txt
│ │
│ ...
│ │
│ └───val
│
│ val_image_1.txt
│
│ val_image_2.txt
│
│ ...
└───test
│ test_image_1.jpg
│ test_image_2.jpg
│ ...
Implementing And Using The Localization Model
The model is exported from a closed format that can only work with pytorch and python to an open format that can work with any ML framework. That format is called Open Neural Network Exchange (ONNX), to build a C# program that can loop over the dataset and create a new dataset of only Iraqi plates images by extracting the plates locations from the original dataset as shown in Figure 8.
Figure 8: Using the localization model in C# Windows forms application
The line in the training script used for exporting is:
!yolo mode=export model=/content/drive/MyDrive/yolov8/training_results/plate/weights/best.pt format=onnx
Labeling Images For The Classification Model
Similar tool as that used in labeling images for the localization model was also used for the classification model. In reference to Figure 9, the new labels’ classes comprise all the characters and colors used for the Iraqi plates (plate types, Arabic numeric, Arabic alphabets, Iraqi provinces, and vehicle types).
Figure 9. The new dataset extracted from the old one
Training The Classification Model
In this part, the same services and python script used in training the localization model was adopted to utilize transfer learning in training another YOLOv8n model on the newly extracted dataset mentioned (See Figure 10). This second model is the classification model.
Figure 10:Labeling plate’s information
Creating A Pipeline Of Models
After the training of the second model, the new model was then exported to the onnx format to use it with ML.NET framework to use it in C#. Then a pipeline is created by modifying the detect function above to pass the outputted image of the localization model as an input image for the classification model.
|
Localization Model |
|
|
Input |
2-D Image |
|
Output |
Four outputs that are the coordinates and size of the detected plate in the image which are (x, y, width, height) |
|
Classification Model |
|
|
Input |
Cropped 2-D Image of the plate |
|
Output |
The Plate’s info as a text |
Then by merging the above functionality we get the complete pipeline:
|
Pipeline |
|
|
Input |
2-D Image |
|
Output |
The plate’s info as a text |
Figure 10: The complete process
2.RESULTS AND DISCUSSIONS
This section presents the findings of the study, which aimed to develop an advanced automatic number plate recognition (ANPR) system tailored to address the specific challenges posed by the diverse and non-standard license plates found in Iraq. Thus, this section details the outcomes through a series of tests, providing a critical evaluation of the system's performance. The graphs and tables for the training process for both the detection model and the classification model are given henceforth.
Detection Results
The performance of the developed ANPR system was evaluated across several epochs, focusing on key metrics such as box loss, class loss, direction-focused loss, precision, recall, and mean Average Precision (mAP). These metrics were calculated for both training and validation phases to assess the model’s accuracy and generalization capabilities.
The improvement in both loss metrics and precision/recall rates throughout the training epochs indicates effective learning and adaptation by the model to the complexities of the dataset. The high values in recall and mAP suggest that the model is highly capable of detecting and recognizing license plates even under varied conditions, which is critical for practical applications in environments with non-standardized plate designs as seen in Iraq.
While the model has shown strong performance metrics, there were challenges, particularly in maintaining a balance between precision and recall in certain epochs. Minor fluctuations in direction-focused loss also suggest areas for potential improvement in model training strategies, possibly by fine-tuning the model further or exploring alternative data augmentation techniques.
The results as shown in Table 1 and Figures 11-19 support the use of CNNs, specifically YOLOv8, in settings where traditional ANPR systems struggle due to non-standard license plate characteristics. The findings could guide future developments in ANPR technologies, particularly in adapting detection systems to regional variations in vehicle identification standards.
