2.1 Region-based Approach

R-CNN [11] extracts deep feature representations by applying high-capacity convolutional neural networks to each independent candidate region. Although this method is effective, it is computationally expensive and slow. Fast R-CNN [10] improves this process and introduces a RoI pooling layer, which can quickly extract the features of each region from a unified feature map of the entire image, significantly improving the speed and efficiency while passing Using softmax instead of SVM simplifies the training process. Faster R-CNN [30] further optimizes the generation process of candidate regions by integrating a Region Proposal Network (RPN), which can be jointly trained with the detection network end-to-end, further improving speed and accuracy. Mask R-CNN [13]: Based on Faster R-CNN, a branch is added to generate a pixel-level mask of the target, which enables the model to not only perform target detection but also perform more refined instance segmentation. This method is particularly suitable for application scenarios that require precise target contour information, such as medical image analysis and video editing.

2.2 Regression-based Approach

The regression-based method handles target detection in a more intuitive and fast way, directly predicting the location and category of the target on the entire image, greatly improving the processing speed.

YOLO [29] series: Especially YOLOv1 to YOLOv4, each generation has made significant improvements based on the previous generation, such as deeper networks, better feature utilization strategies, and more efficient framework design. YOLO transforms the target detection problem into a single regression problem, achieving very fast detection speed, and is very suitable for occasions that require real-time processing. SSD [24]: This algorithm effectively handles the multi-scale target detection problem by predicting the location and category of the target on multiple feature maps of different scales. The design of the SSD allows it to achieve very good speed performance while maintaining high accuracy. EfficientDet [32]: This algorithm further balances the depth, width, and input image resolution of the network through compound scaling technology, improving efficiency and accuracy. EfficientDet shows excellent performance on multiple standard datasets, especially on resource-constrained devices.

2.3 Structural Enhancement Approach

In order to adapt to changes in multi-scale and target morphology, some methods improve the adaptability and accuracy of the model through structural optimization.

Feature Pyramid Network (FPN): By establishing multi-level feature maps, each layer can extract information at different scales. FPN can effectively enhance the model's detection capabilities for targets of various sizes. This top-down structure provides rich semantic information and detailed information for target detection. Deformable Convolutional Networks (DCN): By allowing the shape of the convolution kernel to dynamically adapt to the characteristics of the input data, DCN provides the possibility to handle irregularly shaped targets, which is difficult to achieve in traditional convolutional networks. RetinaNet [21]: By introducing Focal Loss, it solves the imbalance problem of positive and negative samples often encountered in target detection, especially when there are a large number of easy-to-classify background samples, and improves the detection performance of small targets.

Each of the above methods has its own merits, but what they have in common is that they have greatly promoted the development of target detection technology, improved the accuracy, speed, and adaptability of detection, and met the needs of different platforms from mobile devices to high-performance servers.

Figure 1 A diagram of our overall network architecture.

3.1 Improved Feature Pyramid Network

In order to further improve the effect of multi-scale target detection, we developed the improved feature pyramid network(IFPN), which is a network architecture specially designed to improve the efficiency and quality of information fusion between feature layers. IFPN not only comprehensively utilizes features from different layers, but also introduces a novel adaptive weight adjustment mechanism, allowing the network to automatically adjust its contribution to the final detection task based on the importance of each layer's features.

Semantic Score Assessment: The core of IFPN lies in the semantic evaluation of each feature layer, which determines the weight of this layer in feature fusion. The semantic evaluation is based on the assumption that deeper networks usually capture higher-level semantic information, which is especially important for recognizing small-sized objects. We analyze the output of each feature layer through a deep convolutional network and calculate its semantic score. The process can be expressed as:

where $F_i$ is the feature map Conv of the $i$-th layer, represents the convolution operation $W_s$ and $b_s$ are the weight and bias of the convolution layer respectively, and $\sigma$ is usually a nonlinear activation function, such as ReLU [12] or Sigmoid, to ensure the output The score is within a reasonable range.

Dynamic Weight Adjustment: Based on the semantic score of each layer, we calculate dynamic weights to adjust the influence of each layer's features in the fusion process. This dynamic adjustment mechanism allows the network to automatically optimize the feature fusion strategy based on different inputs, thereby better-handling targets of various sizes. This process can be expressed as:

where $\gamma$ is a trainable scaling parameter used to adjust the sensitivity of the softmax function, thereby affecting the weight distribution of different feature layers.

Feature Fusion Strategy: Using the calculated weights, we perform a weighted fusion of features from all layers to generate the final feature map for object detection. This step is accomplished through upsampling and weighted summation, ensuring that all feature maps contribute to the final detection result. The fusion process is:

where $U(F_i,\text{size})$ is an upsampling operation, which upsamples each feature map to a uniform size for effective fusion.

Through this improved feature pyramid network design, iFPN can more effectively utilize features from different network depths and optimize the detection capabilities of multi-scale targets. This not only enhances the model's sensitivity to small-sized targets but also improves its ability to adapt to changes in background complexity. In addition, the adaptive characteristics of IFPN enable it to automatically adjust the feature fusion strategy according to specific tasks and data sets, further improving the versatility and flexibility of the model. These properties make IFPN a powerful tool for multi-scale object detection scenarios ranging from simple to extremely complex.

3.2 Dynamically Deformable Convolutional Network

In object detection, a dynamically deformable convolutional network (D-DCN) is a novel technology designed to solve the problem of irregular shape and pose changes of objects. Traditional fixed convolution kernels perform poorly in handling such changes because they cannot adapt to the local structure and shape changes of the target. In order to solve this problem, D-DCN introduces a dynamic deformation mechanism that enables the convolution kernel to adjust dynamically according to the specific shape of the target, thereby more accurately capturing the detailed information of the target.

Dynamic Offset Calculation: In dynamically deformable convolutional networks (D-DCN), the calculation of dynamic offsets is one of the key steps. These offsets allow the convolution kernel to adjust dynamically according to the local structure and shape changes of the target to more accurately capture the detailed information of the target. We will introduce the calculation process of position offset and shape offset respectively.

Position Offset Calculation: The position offset is used to adjust the position of the convolution kernel to adapt to the position change of the target. This process utilizes a convolutional layer and a nonlinear function to generate position offsets. Expressed in the following form:

where $\Delta p$ represents the position offset, $\theta$ is the learnable parameter Conv represents the convolution operation, and $x$ is the input feature map $W_p$ is the weight of the convolution layer. Through this formula, the network can learn the position offsets at different positions based on the input feature map $x$.

Shape Offset Calculation: Shape offset is used to adjust the shape of the convolution kernel to adapt to changes in the shape of the target. This process also uses a convolutional layer and a nonlinear function to generate shape offsets. The process can be expressed as follows:

where $\Delta s$ represents the shape offset, $\varphi$ is the learnable parameter, $\sigma$ is a nonlinear activation function, Conv represents the convolution operation, $x$ is the input feature map, and $W_s$ is the weight of the convolution layer. Through this formula, the network can learn the shape offsets at different positions based on the input feature map $x$.

Through the calculated position offset and shape offset, D-DCN can dynamically adjust the position and shape of the convolution kernel during the convolution operation, thereby better capturing the local structure and shape changes of the target.

3.3 Dynamic Convolution Operation

The dynamic convolution operation in the dynamically deformable convolutional network (D-DCN) is one of its core components. It enables the network to dynamically adjust the position and shape of the convolution kernel according to the local structure and shape changes of the target, thereby making it more precise. Capture the characteristics of the target well.

Generation of Dynamic Convolution Kernel: In dynamically deformable convolutional networks (D-DCN), the generation of dynamic convolutional kernels is a key task. In traditional convolution operations, the weight of the convolution kernel is fixed, but in D-DCN, the weight of the convolution kernel is adjusted according to the dynamic offset to adapt to the irregular shape and attitude changes of the target.

Offset-based Dynamic Convolution Kernel:The generation of dynamic convolution kernel is based on the adjustment of offset. During the convolution process, for each convolution kernel position p, it is adjusted according to the learned offset to adapt to the local structure and shape changes of the target. Specifically, the weight $W(p_n,\theta ,\varphi )$ of the dynamic convolution kernel can be calculated by the following formula:

where $p_n$ is the position index of the convolution kernel, and $\theta$ and $\varphi$ are learnable parameters, which are used to control the dynamic adjustment of the convolution kernel weight. Conv represents a convolution operation. Through the learned parameters and input feature maps, dynamically adjusted convolution kernel weights can be generated.

Dynamic convolution kernel has the following advantages: 1) Strong adaptability: The weight of the convolution kernel is adjusted according to the learned offset so that the network can better adapt to the irregular shape and posture changes of the target. 2) High flexibility: The weight of the convolution kernel can be dynamically adjusted according to the specific conditions of the target, to better capture the characteristic information of the target.

Through the generation of dynamic convolution kernels, the D-DCN network can better adapt to targets of various shapes and postures, thereby improving the accuracy and robustness of target detection. The calculation process of dynamic convolution operation includes the following steps: 1) Position adjustment of the convolution kernel: For each convolution kernel position $p$, the position of the convolution kernel is dynamically adjusted according to the learned offset $\Delta p_n(\theta ,\varphi )$. 2) Convolution kernel weight adjustment: Calculate the dynamically adjusted convolution kernel weight $W(p_n,\theta ,\varphi )$ based on the adjusted convolution kernel position and shape. 3) Calculation of convolution output: For each convolution kernel position $p$, use the dynamically adjusted convolution kernel weight to perform a weighted sum to obtain the convolution output value $y\left(p\right)$. The process can be described as:

Through this operation, the D-DCN network can dynamically adjust the position and weight of the convolution kernel according to the local structure and shape changes of the target, thereby better adapting to the characteristics of the target.

3.4 Loss

When designing the loss function, we hope that it can accurately reflect the difference between the model predictions and the real labels and that it can effectively guide the training process of the model. In target detection tasks, we usually need to consider two aspects of error: positioning error and category classification error.

Position Positioning Error loss function: The location error loss function measures the difference between the model's prediction of the target location and the true location. The commonly used positioning error loss function is Smooth L1 Loss, which is a variant of Huber Loss. It uses square loss for smaller errors and absolute value loss for larger errors to reduce the impact of outliers on training. This loss is expressed as:

where $l$ is the target position parameter predicted by the model, $l^{\ast }$ is the real target position parameter, and SmoothL1(x) is the Smooth L1 Loss function.

Classification Error loss function. The classification error loss function measures the difference between the model's prediction of the target class and the true class. A commonly used classification error loss function is the cross-entropy loss function, which imposes a greater penalty on incorrect classification predictions, thereby prompting the model to learn more accurate classifications. The loss function can be expressed as:

where $c$ is the target category probability distribution predicted by the model, $c^{\ast }$ is the true target category label, and $c^{\prime \ast }$ is the predicted probability corresponding to the true category label.

Comprehensive loss function design: To comprehensively consider positioning error and classification error, we can adopt the form of a joint loss function when designing the loss function, taking into account the errors in these two aspects at the same time. A common approach is to perform a weighted sum of the positioning error loss function and the classification error loss function to obtain a comprehensive loss function:

where $\lambda$ is the trade-off parameter between positioning error and classification error, which is used to balance the importance of the two in the training process. By designing an appropriate loss function, we can effectively guide the training of the model and improve the performance of the target detection task.

4.1 Experimental Setting

For a fair comparison, all models were trained on a server equipped with four NVIDIA A30 GPUs. We apply the Adam [18] optimizer with a learning rate of 0.001. Batch Normalization [16] is used following each parameter layer. A weight decay of 0.0001 is used in both networks. Refer to Table 1 and Table 2 for a comprehensive list of the parameters.

Figure 2 Visualization results on KITTI dataset.

Figure 3 Comparison with state-of-the-art methods on the KITTI dataset.

Figure 4 Results on NEXET dataset. a: SubCat. b: 3DVP. c: AOG. d: DeepParts. e: Regionlets. f: FilteredICF. g: Ours.

Figure 5 Visualization results on NEXET dataset.

Figure 6 Visualization results on Caltech dataset.

Figure 7 Comparison with state-of-the-art methods on the Caltech dataset.

4.2 Datasets

KITTI dataset: The KITTI dataset is one of the most popular datasets for use in mobile robotics and autonomous driving. It consists of hours of traffic scenarios recorded with a variety of sensor modalities, including high-resolution RGB, grayscale stereo cameras, and a 3D laser scanner. Despite its popularity, the dataset itself does not contain ground truth for semantic segmentation. However, various researchers have manually annotated parts of the dataset to fit their necessities.

NEXET dataset: The NEXET dataset, a widely utilized video dataset in autonomous driving research, encompasses data from over 77 countries and more than 1,400 cities. It includes footage captured under three different lighting conditions (day, night, and twilight) and across all four seasons. The dataset also features diverse road types, such as urban, rural, highway, residential, and desert roads, as well as various weather conditions, including clear skies, fog, rain, and snow. With its high-quality video clips and detailed annotations, NEXET serves as an excellent resource for evaluating our method.

Caltech dataset: The Caltech dataset is a widely used dataset for object recognition tasks and contains approximately 9000 images from 101 object categories. The categories were chosen to reflect the variety of objects in the real world, and the images themselves were carefully selected and annotated to provide a challenging benchmark for object recognition algorithms.

4.3 Evaluation Methods

We start with an evaluation of the proposal network. Following literature [15], prediction callbacks are used as a performance metric. To be consistent with KITTI's setting, the ground truth is considered recalled if the intersection over union (IoU) ratio of the best matching proposal is higher than 70% for cars and higher than 50% for pedestrians and cyclists. These benchmarks were chosen because, unlike VOC [7] and ImageNet [31], they contain many small objects. Typical image sizes for KITTI and Caltech are 1250x375 and 640x480 respectively. KITTI contains three categories of objects: cars, pedestrians, and cyclists, and three assessment levels: easy, medium, and hard. The "medium" level is the most commonly used. A total of 7,481 images are available for training/validation, and 7,518 for testing. Since no ground truth is available for the test set, we follow the recommendation of [22] and split the training validation set for training and validation. In all reduction experiments, the training set is used for learning and the validation set is used for evaluation. Next, models were trained separately for car detection and pedestrian/bicycle detection. A model for pedestrians was learned on Caltech.

4.4 Results

The results on the KITTI: Comparisons with previous methods are shown in Table 3 and Figure 3. We set a new record for detecting pedestrians and cyclists. Columns "Pedestrians-Mod" and "Cyclists-Mod" are 6 and 7 points higher than 3DOP respectively, and perform better than Faster-RCNN, Regionlets, etc. We also achieved considerable lead on the very new SDP+RPN using scale-dependent pooling. In terms of speed, this network is quite fast. For the largest input size, our detector is approximately 8 times faster than 3DOP. On the original image (1250x375), the detection speed reaches 10fps. Figure 2 shows our visualization results.

The results on the NEXET: Table 5 offers a thorough comparison of the performance of various methods on the NEXET dataset. Our method stands out with impressive metrics, achieving 75.07% in Precision, 72.37% in F1 Score, 73.67% in [email protected], and operating at 127 FPS. These results underscore the reliability and effectiveness of our approach in different scenarios. Additionally, qualitative outcomes presented in Figures 4 and 5 illustrate that our method performs exceptionally well in real-world applications. In summary, our approach has led to significant advancements in both speed and accuracy.

Detection on Caltech: Our detector is also evaluated on the Caltech pedestrian benchmark. The model is compared with methods such as DeepParts, CompACT-Deep, CheckerBoard, LDCF, ACF [25], and SpatialPooling, covering three tasks: Reasonable, medium, and partially obscured. As shown in Figures 6 and 7, our method demonstrates state-of-the-art performance. It performs exceptionally well on small and moving targets objects, surpassing DeepParts, which specifically addresses the occlusion problem.

4.5 Ablation Study

Effect of input upsampling: Table 4 shows that input upsampling can be a key factor in detection. Significant improvements can be achieved by upsampling the input by a factor of 1.5 to 2, but we found that gains beyond 2x are small. This is smaller than the 3.5x factor required by literature [39]. Larger factors cause detection to be significantly slower and require more memory.

Sampling strategies: Table 4 compares the sampling strategies: random and mixture. For cars, the three strategies are similar; for pedestrians, the bootstrap and hybrid strategies are similar, but the random strategy is significantly worse. Note that random sampling has more false positives.

CNN feature approximation: We tried three methods to learn deconvolution layers for feature map approximation: 1) weights with bilinear interpolation; 2) weights initialized by bilinear interpolation and learned by backpropagation; 3) weights initialized by Gaussian noise and learned by backpropagation. We found that the first method works best, which confirms the research results of [17]. As shown in Table 4, deconvolution layers help in most cases. The gains are greater for smaller input images, which tend to contain smaller objects. It is worth noting that the computational complexity of feature map approximation is very small and does not increase parameters.

Object Detection via Proposal Networks: Proposal networks can act as detectors by converting class-agnostic classification into class-specific classification. Table 4 shows that although not as powerful as the unified network, it achieves quite good results, which are better than some detectors on the KITTI rankings.