In recent years, Unmanned Aerial Vehicle (UAV) networks are widely used in both military and civilian scenarios. However, due to the distributed nature, they are also vulnerable to threats from adversaries. Time delay attack is a type of internal attack which maliciously delays the transmission of data packets and further causes great damage to UAV networks. Furthermore, it is easy to implement and difficult to detect due to the avoidance of packet modification and the unique characteristics of UAV networks. However, to the best of our knowledge, there is no research on time delay attack detection in UAV networks. In this paper, we propose an Efficient Time Delay Attack Detection Framework (ETD). First, we collect and select delay-related features from four different dimensions, namely delay, node, message and connection. Meanwhile, we utilize the pre-planned trajectory information to accurately calculate the real forwarding delay of nodes. Then, one-class classification is used to train the detection model, and the forwarding behaviors of all nodes can be evaluated, based on which their trust values can be obtained. Finally, the K-Means clustering method is used to distinguish malicious nodes from benign ones according to their trust values. Through extensive simulation, we demonstrate that ETD can achieve higher than 80% detection accuracy with less than 2.5% extra overhead in various settings of UAV networks and different routing protocols.