2.1 The research of object detection

In the realm of multi-object tracking, target detection techniques play a pivotal role. These techniques serve as the foundation for identifying and localizing objects within a scene, a prerequisite for subsequent tracking processes [28, 31]. Notably, the emergence of deep learning architectures has significantly advanced the state-of-the-art in target detection. Among these architectures, YOLO (You Only Look Once) models [29, 30] have gained considerable attention for their real-time performance and accuracy. Noteworthy variants such as YOLOv5, YOLOv8, and YOLOv9 have been introduced, each refining the original architecture to achieve better performance in terms of speed, accuracy, or both.

Beyond YOLO models, recent advancements have been made in transformer-based object detection and tracking methods. One notable approach is DETR (DEtection TRansformer) [32], which leverages transformer architectures to directly predict object bounding boxes and categories in an end-to-end manner. Variants such as Deformable DETR, PnP-DETR, and RT-DETR have further extended the capabilities of transformer-based approaches, addressing shortcomings such as scalability, robustness to occlusions, and handling deformable objects [33].

Each of these algorithms has its strengths and limitations. YOLO models excel in real-time performance and simplicity but may sacrifice fine-grained localization accuracy compared to slower, two-stage detectors [8]. Transformer-based approaches offer end-to-end learning capabilities and global context modeling, but they may suffer from increased computational complexity and training data requirements. Understanding these trade-offs is crucial for selecting the most suitable algorithm for specific tracking tasks.

2.2 Research on multi-object tracking

In the current landscape of multi-object tracking algorithms, several notable approaches have emerged, each with its strengths and limitations. SORT (Simple Online and Realtime Tracking) is a popular method known for its simplicity and efficiency [26, 10]. It operates on a tracking-by-detection framework, associating object detections across frames based on motion models and appearance features. SORT is renowned for its real-time performance and effectiveness in tracking multiple objects simultaneously. However, its performance may degrade in scenarios with heavy occlusions or complex interactions between objects due to its reliance on simple motion and appearance models. DeepSORT [27] (Deep Simple Online and Realtime Tracking) builds upon SORT by integrating deep learning techniques for improved object appearance modeling and feature extraction. By leveraging deep neural networks, DeepSORT achieves better robustness to occlusions and appearance variations, resulting in more accurate and reliable tracking performance compared to traditional SORT. However, DeepSORT may suffer from higher computational costs and resource requirements due to the complexity of deep learning models. MOTDT (Multiple Object Tracking with Deep Learning and Tracklet) is another notable algorithm that combines deep learning methods with tracklets, short track segments, to improve tracking performance. MOTDT excels in handling complex scenarios with occlusions and crowded environments by leveraging deep feature representations and tracklet association techniques. However, its performance may be affected by issues such as track fragmentation and identity switches in highly dynamic scenes [14].

JDE (Joint Detection and Embedding) integrates object detection and feature embedding into a unified framework for multi-object tracking. By jointly optimizing detection and embedding tasks, JDE achieves superior performance in associating object detections and maintaining object identities over time. This approach benefits from end-to-end learning and feature embedding, resulting in robust tracking performance across various challenging scenarios. However, JDE may suffer from increased computational complexity and training data requirements compared to simpler tracking methods.

Figure 1 Overall structure diagram of the proposed model.

Bytetrack is a recent addition to the multi-object tracking landscape, leveraging advanced techniques such as deep learning, attention mechanisms, and graph neural networks. Bytetrack aims to address the limitations of existing methods by incorporating contextual information and global dependencies into the tracking process, leading to improved robustness and accuracy. However, its effectiveness in real-world scenarios and computational efficiency relative to traditional methods remain areas of ongoing research and evaluation [15].

Overall, while each of these algorithms offers distinct advantages in multi-object tracking, they also exhibit specific limitations that need to be addressed for more comprehensive and effective tracking solutions [20, 21, 22]. The progression from simpler methods like SORT to more advanced approaches like Bytetrack reflects the continuous evolution and refinement of multi-object tracking techniques in response to the challenges posed by complex real-world scenarios [7, 9, 23].

Figure 2 Schematic diagram of the CT-DETR model architecture.

3.1 CT-DETR mode

The complete architecture of the CT-DETR model comprises a backbone network, a hybrid encoder, and a transformer decoder with auxiliary prediction heads. The backbone network serves as the foundational component, responsible for extracting feature representations of input images, utilizing a pre-trained EfficientNet. The hybrid encoder, on the other hand, undertakes the task of feature encoding, converting the feature maps extracted by the backbone network into more expressive and semantically understandable feature representations. It consists of a series of Transformer encoders, encoding feature vectors into high-dimensional representations through self-attention mechanisms and fully connected layers. The transformer decoder with auxiliary prediction heads is a crucial component responsible for translating the encoded feature maps into the results of target detection. It consists of multiple layers of Transformer decoders, each layer containing self-attention mechanisms and cross-attention mechanisms. The auxiliary prediction heads provide additional supervisory signals, contributing to the convergence and stability of model training. By integrating these components, the CT-DETR model achieves efficient detection and re-identification of targets in complex scenes. The network structure is illustrated in Figure 2, where initially, features from the last three stages of the backbone network S3, S4, S5 are utilized as the input to the encoder. The efficient hybrid encoder transforms multi-scale features into a sequence of image features through Scale-Invariant Feature Interaction (AIFI) and Cross-Attention Feature Fusion Module (CAFF).

3.1.2 IoU-aware Query Selection

We propose the method of IoU-aware Query Selection, wherein during training, the model is constrained to generate high classification scores for features with high IoU scores, and low classification scores for features with low IoU scores. This approach helps the model to more effectively utilize IoU information in the object detection task, thereby improving the accuracy of object localization and classification. Specifically, through IoU-aware Query Selection, the model can focus more on features closely related to the target, while appropriately downweighting features with lower relevance to the target, thus effectively enhancing the performance of object detection. This mechanism introduces a finer IoU-aware adjustment during training, enabling the model to intelligently handle object detection tasks in various scenarios, thereby improving its robustness and generalization capability. We redefine the optimization objective of the detector as follows:

$ℒ(\hat{y},y)={ℒ}_{box}(\hat{b},b)+{ℒ}_{cls}(\hat{c},\widehat{𝒃},y,𝒃)$

$={ℒ}_{box}(\hat{b},b)+{ℒ}_{cls}(\hat{c},c,IoU)$

Figure 3 Schematic diagram of the Transformer model structure.

where $\hat{y}=\{\hat{c},\hat{b}\}$ and $y=\{c,b\}$ represent predicted and true values, $c$ stands for class and $b$ for bounding box. We introduce the IoU score into the classification branch's objective function, akin to VFL, to ensure consistency in positive sample classification and localization.

3.2 Self-Attention

In our CT-DETR model, the Transformer plays a crucial role. This architecture enables the model to effectively capture long-distance dependencies within the image [18, 19, 24], whether they are spatial or temporal. By simultaneously calculating attention scores for each position in the image relative to every other position, the Transformer endows the backbone network of CT-DETR with a powerful global understanding capability. This not only increases the model's flexibility in processing input sequences of varying lengths but also significantly boosts the efficiency of parallel computations, thereby accelerating training speed. Most importantly, the introduction of the Transformer significantly enhances the model's ability to recognize key features within images, providing strong support for achieving high-precision object detection and identification. The network architecture diagram of the Transformer is shown in Figure 3.

where $Q$ is the matrix of queries, $X$ is the input representation (e.g., word embeddings), and $W^Q$ is the weight matrix for queries.

where $K$ is the matrix of keys, $X$ is the input representation, and $W^K$ is the weight matrix for keys.

where $V$ is the matrix of values, $X$ is the input representation, and $W^V$ is the weight matrix for values.

where $A$ represents the attention scores matrix, $Q{K}^T$ is the dot product of queries and keys transpose, and $\sqrt{d_k}$ is the scaling factor, with $d_k$ being the dimension of the key vectors.

where $A^{\prime }$ is the softmax-normalized attention scores, ensuring all the attention scores are positive and sum up to 1.

where $O$ is the output matrix of the self-attention mechanism, and $V$ is the matrix of values.

where Output is the final output of the self-attention layer, $O$ is the output matrix from the self-attention mechanism, and $W^O$ is the output weight matrix.

3.3 Loss Function Design

When constructing the loss function, we first calculate the loss between the predicted heatmap and the actual heatmap. This loss calculation is based on the discrepancy between the two, aiming to measure the deviation between the predicted results and the ground truth. Then, we use a Gaussian distribution to map the heatmap onto the heatmap, which helps to more accurately localize the keypoints in the heatmap. Finally, based on the difference between the mapped heatmap and the actual heatmap, we construct the loss function to guide the optimization of model parameters, thereby improving the model's prediction accuracy and precision regarding the target.We use the following equation to describe the content above.

Figure 4 Representative samples from the dataset.

where $Y_{xy}$ denotes the value of the actual heatmap at pixel location $(x,y)$, generated by applying a Gaussian distribution centered on the keypoint. This reflects the probability of the pixel being the keypoint location. ${\hat{Y}}_{xy}$ represents the predicted heatmap value at pixel location $(x,y)$, indicating the model's estimation of the keypoint likelihood at that pixel. $P_{d\left(x\right)}$ and $P_{d\left(y\right)}$ are the predicted coordinates of the keypoint, serving as the center of the Gaussian distribution applied to generate the actual heatmap. $\sigma$ is the standard deviation of the Gaussian distribution, controlling the spread of the heatmap around the keypoint and hence influencing the localization sensitivity. $N$ is the total number of pixels in the heatmap, used to normalize the loss over all pixels. $\partial$ and $\beta$ are hyperparameters that adjust the emphasis on different aspects of the loss, specifically focusing on penalizing false negatives and false positives respectively.

The final loss function is shown as follows:

3.4 Pedestrian reidentification

Pedestrian re-identification (ReID, Reidentification) is a specific computer vision task aimed at recognizing and tracking the appearance of the same pedestrian across a network of cameras [20, 21]. This capability is crucial for fields such as video surveillance, human traffic analysis, and social security. Pedestrian Re-ID systems work by extracting and comparing the appearance features of pedestrians from different video frames or different camera angles. These appearance features include, but are not limited to, clothing color, style, body posture, and gait. The detailed information in this study is shown in the equations.

where $L\left(k\right)$ is the expected probability distribution for the identity of the $k^{th}$ target, where the number of categories is denoted by $k$. $p\left(k\right)$ represents the actual probability distribution for the identity of the $k^{th}$ object.

4.1 Dataset

Our experiment makes use of two prominent datasets for evaluating object tracking algorithms: 2DMOT2015 and OTB-100. The 2DMOT2015 dataset serves as a comprehensive benchmark for multiple object tracking, offering a diverse array of challenging sequences derived from real-world scenarios [34, 35]. These sequences are annotated with ground truth data, including object positions and identities, enabling rigorous evaluation of tracking algorithms' performance under various conditions such as occlusions, scale variations, and object interactions. On the other hand, the OTB-100 dataset, also known as Object Tracking Benchmark-100, is specifically designed for single object tracking tasks [36]. It encompasses a collection of 100 video sequences, each presenting unique challenges like appearance changes, background clutter, and motion blur. Similar to the 2DMOT2015 dataset, the OTB-100 dataset provides ground truth bounding box annotations for each sequence, facilitating a thorough assessment of tracking algorithms' accuracy and robustness. These datasets collectively serve as invaluable resources for assessing the effectiveness and robustness of our proposed tracking algorithm across a wide range of tracking scenarios and challenges. Figure 4 displays examples from two datasets, and we extracted four frames for illustration. These examples highlight the diversity and complexity of the tracking scenarios considered in our evaluation.

4.2 Experimental Environment

In this experiment, we have meticulously documented the configuration of the experimental environment to ensure the reproducibility of the results. The specific parameters of the experimental environment are shown in Table 1.

4.2.1 Model training

Table 2 shows the parameter settings for CT-DETR, where each row lists a specific parameter used during the model's training and evaluation along with its corresponding value. The detailed parameters include a learning rate set to 0.01, indicating the step size at each iteration while moving toward the minimum of the loss function. The optimizer used is SGD (Stochastic Gradient Descent), a popular optimization method for training deep learning models. The batch size is set to 16, referring to the number of training examples used in one iteration. Weight decay is set to none, training epochs to 300, model parameters to 4,385,523, and the number of layers to 252. The image size is set to 640, seed values to 0, and early stopping is enabled (True).

Table 1 Experimental environment demonstrated.

Parameter	Configuration
CPU	Intel Core i7-12700KF
GPU	NVIDIA GeForce RTX 4090 (24 GB)
CUDA version	CUDA 11.8
Python version	Python 3.8.16
Deep learning framework	Pytorch 1.8.1
Operating system	Ubuntu 22.04.2

Table 2 Model parameter settings.

Parameter	Value
Learning Rate	0.01
Optimizer	SGD
Batch Size	16
Weight Decay	None
Training Epochs	300
Model Parameters	4,385,523
Number of Layers	252
Image Size	640
Seeds	0
Early Stop	True

Figure 5 Model performance on the OTB-100 dataset.

4.2.2 Evaluation Metrics

In this experiment, we employed numerous evaluation metrics to thoroughly analyze the model's performance, as shown in the following formulas.

$$Precision=\frac{TP}{TP+FP}$$

where $Precision$ is the proportion of true positive predictions in all positive predictions, $TP$ is the number of true positives, and $FP$ is the number of false positives.

$$Recall=\frac{TP}{TP+FN}$$

where $Recall$ measures the proportion of actual positives correctly identified, $FN$ is the number of false negatives.

$$F1\text{-}Score=2\times \frac{Precision\times Recall}{Precision+Recall}$$

where $F1\text{-}Score$ is the harmonic mean of precision and recall, offering a balance between them.

$$TP=\text{Number of true positive predictions}$$

where $TP$ directly counts instances correctly identified as positive.

$$mAP=\frac{1}{N}\sum _{q=1}^Q\sum _{k=1}^{n_q}P\left(k\right)\times rel\left(k\right)$$

where $mAP$ (mean Average Precision) averages the precision scores at different recall levels, $N$ is the number of queries, $Q$ is the total number of classes, $n_q$ is the number of relevant documents for the $q^{th}$ query, $P\left(k\right)$ is the precision at cutoff $k$, and $rel\left(k\right)$ is an indicator function equaling 1 if the item at rank $k$ is a relevant document, 0 otherwise.

Table 3 Performance evaluation metrics.

Measure	Better	Description
MOTA	higher	Multiple Object Tracking Accuracy.
MOTP	higher	Multiple Object Tracking Precision.
IDF1	higher	The ratio of correctly identified detections.
MT	higher	Mostly tracked targets.
ML	lower	Mostly lost targets.
FP	lower	The total number of false positives.
FN	lower	The total number of false negatives.
ID Sw.	lower	The total number of identity switches.

Figure 6 Visualization of ReID Results. Five frames extracted from Scene 258 are presented for demonstration.

4.3 Experimental Results and Analysis

In the evaluation of our pedestrian counting algorithm, we utilized the 2DMOT2015 benchmark for quantitative analysis of the algorithm's performance. The 2DMOT2015 is a recognized evaluation platform specifically designed for the fair comparison and assessment of multiple pedestrian tracking algorithms. This platform offers a series of detailed test video data encompassing various dynamic and complex urban environments. These settings include diverse lighting conditions, levels of crowd density, and camera movements, thereby providing rich testing scenarios for pedestrian detection and tracking algorithms. Each video contains precise ground truth data, which serves to verify the accuracy and consistency of each pedestrian target identified by the algorithm. The specific description is shown in Table 3.

The model uses the OTB-100 (Object Tracking Benchmark-100) dataset to test accuracy. Since it is designed for pedestrian tracking, the model selected multiple videos containing pedestrians such as "Human 6 Video", "Woman Video", and "Girl 2 Video" for evaluation. As shown in Table 4.3, the success rates and frames per second (FPS) of different tracking algorithms on various types of videos, including Human 6 Video, Woman Video, and Girl 2 Video. These algorithms have undergone testing on the OTB-100 dataset, which is a commonly used benchmark for evaluating tracking algorithm performance. In the Human 6 Video, our algorithm outperforms others with a success rate of 97%, significantly higher than KCF (20%), ASLA (38%), TLD (30%), DASiamRPN (91%), and SiamRPN (75%). Similarly, on Woman Video and Girl 2 Video, our algorithm also achieves excellent results with success rates of 94% each, comparable to DASiamRPN and SiamRPN but notably superior to KCF, ASLA, and TLD. In terms of performance, although our algorithm's FPS is 80, lower than KCF (245), DASiamRPN (160), and SiamRPN (200), it still maintains a relatively stable speed compared to these algorithms while achieving significant advantages in accuracy. Therefore, our algorithm demonstrates outstanding comprehensive performance in pedestrian tracking tasks. Figure 5 visualizes the table, providing a more intuitive representation of the performance of the model.

Figure 7 Visualization of pedestrian flow tracking and counting results by the model.

Figure 8 Visualization of pedestrian flow trajectory paths from tracking results.