Institute of Information Technology (ITEC) Institute of Information Technology (ITEC) Alpen-Adria-Universität Klagenfurt Alpen-Adria-Universität Klagenfurt Austria Austria severin.kacianka@aau.at hermann.hellwagner@aau.at
ABSTRACT mobility of UAVs leads to ever-changing antenna orienta-
The core problem for any adaptive video streaming solution, tions and network topologies. The induced problems can be particularly over wireless networks, is the detection (or even so severe that some authors [2, 14] suggest that a new wire- prediction) of congestion. IEEE 802.11 is especially vulner- less LAN standard might be needed. Asadpour et al. [2] able to fast movement and change of antenna orientation. write that the “special ecosystem [of UAV networking] un- When used in UAV networks (Unmanned Aerial Vehicles), doubtedly requires a rethinking of wireless communications the network throughput can vary widely and is almost im- and calls for novel networking approaches.” possible to predict. This paper evaluates an approach orig- Streaming a video over such an unreliable channel causes inally developed by Kofler for home networks, in a single- additional problems: hop UAV wireless network setting: the delay between the • Video streaming requires a lot of bandwidth, sending of an IEEE 802.11 packet and the receipt of its cor- responding acknowledgment is used as an early indicator of • reacts strongly to link degradation, and the link quality and as a trigger to adapt (reduce or increase) the video stream’s bitrate. Our real-world flight-tests indi- • a single missing packet can cause the loss of several cate, that this avoids congestion and can frequently avoid seconds of video data. the complete loss of video pictures which happens without Yet the quality of video streams can be adapted to fit (al- adaptation. most) any available bandwidth. If a link degradation can be detected (or even predicted) in time, it is possible to reduce Categories and Subject Descriptors the stream’s bitrate and avoid congestion and a complete loss of the video picture. C.2.1 [Computer-Communication Networks]: Network The goal of our research was to address these problems Architecture and Design—Wireless communication and prevent the video signal’s complete loss by using a method described by Kofler [8, 9] to indicate near-term congestion Keywords and to adapt the UAV’s video stream bitrate to the current Video Streaming, Adaptive Streaming, UAVs, UAV Com- network conditions. munication 2. RELATED WORK 1. INTRODUCTION 2.1 Wireless Networks In most application scenarios, UAVs will require a perma- In a survey paper, Bekmezci et al. [4] identify communica- nent link to some base or control station. Especially in civil- tion as the biggest challenge for multi-UAV systems and coin ian and scientific systems, this link will often use the IEEE the term “Flying Ad-Hoc Networks (FANETs).” This term 802.11 wireless LAN technology. 802.11’s greatest advan- is inspired by previous work on “Mobile Ad-Hoc Networks” tages are that it is cheap, readily available, well understood (MANETs) and “Vehicle Ad-Hoc Networks” (VANETs) and by developers, and widely supported by software and op- can be considered a special form thereof. MANETs gener- erating systems. However, 802.11 was never intended to be ally consider people moving in an environment using mobile used for fast moving and direction changing devices. The in- devices and VANETs generally consider cars moving on a tended use cases were static devices like laptops. The high road. Bekmezci et al. differentiate FANETs from MANETs Permission to make digital or hard copies of all or part of this work for and VANETs for several reasons: personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear • UAVs move much faster and have more degrees of free- this notice and the full citation on the first page. Copyrights for components dom than people or cars. of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to • Because of UAVs’ higher mobility the network topol- redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org. ogy will change much more frequently. MoVid ’15, March 18-20, 2015, Portland, OR, USA Copyright 2015 ACM 978-1-4503-3353-5/15/03 ...$15.00 • FANETs deliver control commands as well as sensor http://dx.doi.org/10.1145/2727040.2727043 data. • The distance in FANETs is usually much greater than tion (like sync or probe messages) needs to be trans- in MANETs or VANETs. Thus communication hard- mitted. ware needs to be different. 2. Being a local approach, it does not need any sup- Andre et al. [1] focus on the design choices of aerial com- port from clients. Any client that supports RTP and munication networks and cover all common wireless com- H.264/SVC can be used. munication technologies, namely IEEE 802.15.4 (XBee Pro), 3. An increase in queueing delays precedes packet loss, IEEE 802.11 (WiFi), 3G/LTE (mobile phone network), and thus allowing fast reaction to upcoming link degrada- infrared. For video streaming from UAVs, Andre et al. rec- tion. ommend IEEE 802.11 technology. They find that WLAN can deliver the required data rates even over longer dis- The adaptation logic then works as follows: If the delay tances and that this distance can easily be extended by using exceeds a certain threshold, the proxy will reduce the video’s multi-hop networks. They conclude that “(...) the mea- quality. If, however, the proxy measures a delay which is sured throughput confirms the practicality of WLAN. Nev- lower than the threshold, the proxy will evaluate past delays ertheless, achievable throughput heavily depends on antenna and if it finds them low as well, it will try to increase the placement and the mobility of the MAV [micro aerial vehi- stream’s quality. Previous packets serve as “packet pairs” cle], and technological improvements are required to guar- and their delay is used to estimate available link capacity. antee necessary throughput.” [1] This works because video stream packets are usually the same size and sent back-to-back. 2.2 Video Streaming Despite the prominence of the use case, there is scant lit- 3. HYPOTHESES erature that focuses on UAV networks (or FANETs) and Kofler [8, 9] could link the delay between the sending of an video transmission. Most papers seem to focus either on the IEEE 802.11 packet and the receipt of its acknowledgment task of recording a video (quality, coding, object tracking, to the link quality and adapt a video stream accordingly. etc.) or on the UAV network and its properties. The au- This paper tries to apply this idea to UAV video streaming thors are not aware of any paper describing a system that and to evaluate three hypotheses: combines networking and adaptive video streaming or other video transmission techniques. For MANETs, however, Lin- 1. The delay is related to video quality. deberg et al.’s [10] survey paper is a good resource to get an overview of the “challenges and techniques” for video stream- 2. The delay is a good indicator of impending problems ing in MANETs. They stress the fact that meeting band- with the video stream. width, delay and jitter constraints in wireless networks is far 3. The delay can be the input for an adaptation proxy more challenging than in wired networks. They state that that improves the overall video quality. “(...) a significant amount of overall packet drops in wireless networks are perceived as being random, rather than caused Kofler could show that this delay is a good indicator for by congestion (...).” impending link degradation in a home-use setup. This setup Although technologies like Scalable Video Coding (SVC) consisted of a common-off-the-shelf router and a laptop. [13] or lightweight encoders [5] will become interesting in the Because the scenario was home use, the movement evalu- future, there are currently no common off-the-shelf solutions ated was only at walking speed (a person walking through that make them readily available (especially for real-time his/her flat). For this slow movement, the approach worked applications). So in real-world settings, one is reduced to remarkably well. However, as touched upon in the introduc- tweaking the H.264 encoder’s parameters. tion, UAV networks change much faster and more violently. This paper provides the first evaluation of an adaptive video 2.3 Congestion Detection streaming approach in such an environment. Kofler [8, 9] combines the idea of packet pairing and pas- sive sensing from the field of bandwidth estimation (e.g., 4. SYSTEM DESIGN [6, 12]) with the fact that video streams (using H.264 and RTP) will have many packets of equal size. He presents a 4.1 Overview proxy server that can adapt SVC video streams according to current network conditions. However, “instead of reacting The experiments used a Logitech C920 HD webcam [11] on packet loss, [the] approach uses an increase in queueing mounted on an Ascending Technologies (AscTec) Firefly [3] delay at the router to detect phases of throughput degrada- UAV (shown in Figure 1). It carried a “Mastermind board” tion.” [9] Kofler measures the queueing delay indirectly. He that was outfitted with a 1.86GHz processor and 4GB of noticed that the Linux kernel’s wireless monitoring interface RAM. The wireless card was a common off-the-shelf Atheros will only show frames after they were acknowledged by the AR93xx chipset with three external antennas and IEEE 802.11 recipient. So the worse a link is, the longer the time between a/b/g/n support. The system used AscTec’s Ubuntu 12.04 sending of the frame and the receipt of the corresponding (LTS) image with Linux kernel 3.5. acknowledgment will be (because, for example, a frame has Figure 2 shows an overview of the system architecture. to be retransmitted). It could be shown that this queue- The camera delivers an H.264 video stream and is controlled ing delay corresponds with the link quality and the packet by gstreamer [7], an open-source multimedia framework. loss. Increasing delay usually precedes actual packet loss by gstreamer can request (almost) any bit rate from the cam- almost two seconds. This approach has three advantages: era and passes the H.264 encoded video stream on to the proxy as a normal RTP stream. gstreamer (or more ex- 1. The information is available locally – no extra informa- actly, the shell script controlling it) then offers the proxy 3. When the connection is good (for example the UAV is next to the laptop), the delay does not increase and the video stream stays stable.
The proxy consists of two threads (Figure 3): The thread
proxy just accepts RTP packets from gstreamer and for- wards them to the base station while noting the time of sending. The sniffer thread listens on the Linux kernel’s monitoring interface (in our case always called mon0) and whenever it sees an RTP packet it will log the time (which is the time of the reception of the MAC layer acknowledg- ment) and compare it to the send time. It matches the packets by their RTP sequence number.
Figure 1: The UAV in flight
an interface to request a rate change. In one thread, the
proxy forwards the RTP stream unaltered to the base sta- tion (that is running the video streaming client software), but keeps track of when each RTP packet is sent. At the same time, another thread listens on the wireless monitoring interface of the Linux kernel for the MAC layer ACK of the sent frame. Depending on the delay between the sending of an RTP packet and its acknowledgment, the proxy will adapt the stream. Figure 3: Overview of the proxy architecture
The decision when to adapt is directly influenced by the
three observations mentioned above. The algorithm tries to avoid delays higher than 1.0 seconds by reducing the video stream’s bit rate as soon as the delay crosses the threshold of 0.3 seconds. As it usually takes some time for the delay to react to the lower bitrate, i.e., to decrease, the algorithm will wait for a period of 25 consecutive packets (“cooldown period”) before reducing the quality further. Without this cooldown period, the proxy would reduce the quality down to the lowest level most of the time. Figure 2: Overview of the system architecture When there is an extended period (250 packets) of packets with a low delay (<0.1sec), the proxy will increase the bit 4.2 Adaptation Proxy rate by one level. Please note that all these values were gathered empirically. After the initial experiments, these The Adaptation Proxy forwards the RTP stream sent from values were determined to give the best results. However, gstreamer, measures the delay between the sent packets and tuning these values was not a priority – so there are probably the reception of their ACKs and makes adaptation decisions. combinations that would provide better results. From Kofler’s work the following three assumptions were drawn: 5. RESULTS 1. A frame only shows up on the monitoring interface The final data set consists of 12 evaluation flights (9 with after it was transmitted and the ACK received. adaptation, 3 without adaptation) taken on two consecutive 2. The lower the delay, the better the connection. days (May 5–6, 2014)1 . The flights were conducted in an area close to the University and the UAV was controlled by 3. The delay is mainly affected by the connection quality one of the authors. During the flights, the UAV was flown (no random changes). towards a small wood, where three white hay bales were “hidden.” The goal of the flights was to find the “hidden” By means of extensive testing, the following observations hay bales. Around 50 flights were conducted, however, only to confirm these assumptions were made: 12 used the same software version and settings and are thus comparable. Each test flight produced two artifacts: (1) a 1. When the delay is high (above ∼1.0 sec), there will be log file that contains information about the packets and (2) strong artifacts in the video stream. 1 Example videos can be found on http://kacianka.at/ma/ 2. When the delay climbs above ∼0.3 sec, it usually con- demo_no_adapt_2014_05_06.mp4 and http://kacianka. tinues to climb further. at/ma/demo_adapt_2014_05_06.mp4. the recording of the video stream as it was received at the base station. The log file was stored on the UAV and con- tains the following variables2 : distance in meters between the UAV and its take off position; UNIX timestamp in sec- onds; time delay between the sending of a packet and the re- ceipt of the ACK in seconds; RTP sequence number. Flights that performed adaptation logged an integer representation of the quality level that was being used and the number of packets with a “low” delay (less than 0.1 seconds). In contrast to the log files, the videos are impossible to an- alyze objectively as there is no “perfect” video, i.e., a refer- ence, to compare them to. There are no clear-cut indicators when a video signal is “good” and at what point it becomes Figure 6: “Bad” frame, with hay bales lost due to “bad.” In the course of this work, “good” means that sig- artifacts nificant features on the ground can still be made out (even though the resolution is low) and “bad” means that artifacts blur the picture beyond recognition. Figure 5, for example, shows a low resolution image of the hay bales. Although this image is of significantly lower resolution than the image in Figure 4, the hay bales can still be made out and the image is thus considered “good.” Figure 6 on the other hand is blurred beyond recognition and is thus considered “bad.”
Figure 7: Development of delay without adaptation
as the delay crosses the threshold of 0.3 seconds. It does
not fall immediately, because the cooldown period keeps the bit rate too high. As soon as quality level 3 is reached, the delay immediately falls.
Figure 4: High quality frame showing the hay bales
Figure 8: Development of delay with adaptation
The following figures were created by aggregating the data
from all 12 flights. The first three bars (in blue) in each fig- Figure 5: Low quality frame showing the hay bales ure are the aggregate value of all flights (both with and with- out adaptation). The next nine bars (in red) show only the flights with adaptation and the final three bars (in yellow) 5.1 Characteristics of the Delay show only the flights without adaptation. Figure 7 shows how quickly the delay can rise from low levels to high peaks. Without a mechanism to counter the 5.2 Delay effect, the delay will rise until the UAV stops moving and/or Figures 9 and 10 show the maximum and mean delay, the antenna orientation improves. respectively. It is interesting to see that the flights with When active bit rate adaptation takes place, it can counter adaptation had a significantly lower mean delay than the the rise of the delay and keep the periods of high delay flights without adaptation. While the data set is small, these shorter. Figure 8 shows how the quality level is reduced results confirm Kofler’s findings and suggest that the delay 2 is indeed a very good indicator of the link quality. The raw data files can be found on http://kacianka.at/ma/raw_flight_data.zip. Figure 9: Maximum delay between sending a packet Figure 12: Mean number of packets between the and receiving the ACK packet first packet with a long delay (>0.3s) until the delay is below 0.3s again
Kofler [8] could confirm these hypotheses in the context of
(indoor) wireless home networks. The work at hand is based on these previous results and evaluated them in a single-hop UAV network. Kofler’s idea was implemented with different tools, but the basic results can be confirmed. This paper found a clear correlation between high delays (higher than 1 second) and bad video quality (up to the complete loss of the video stream). All experiments in the lab as well as around 50 real-world flight tests exhibited this behavior. A Figure 10: Mean delay between sending a packet connection breakdown without a period of high delays was and receiving the ACK packet never observed. However, sometimes artifacts that do not seem to be re- lated to any measured factors still arise. They might be in 5.3 Number of Packets to Low Delay the camera’s H.264 encoder or in the decoder. So the delay Figures 11 and 12 show the number of consecutive packets will not account for all possible problems in a video stream. that have a delay greater than 0.3s. 0.3s is the threshold used The results strictly assume that a good connection is char- to reduce the bit rate of the video. The fewer packets have a acterized by a low delay (less than 0.1 seconds) and that a high delay, the better the video quality is. The flights with bad connection is indicated by an increase of the delay. adaptation have far shorter periods of “bad connection” than The results of the real-world flight tests indicate that the the flights without adaptation. As longer periods of lower described solution works indeed better than just using a delays mean a better video quality, these results suggest that static video bit rate. The authors’ (biased) observation is applying Kofler’s congestion detection method and adaptive that flying with video adaptation leads to shorter “break video streaming to UAV networks will indeed lead to a better downs” of the video pictures and that one can still get a video picture quality. good overview of the area of flight when streaming in low quality. This is backed up by the data displayed in Figure 12 that shows that flights with adaptation have far fewer de- layed packets than flights without adaptation. In summary, this paper cautiously confirms all three hy- potheses in the context of UAV networks. However, the number of test flights was low and thus the results should be considered preliminary. They are encouraging and beckon further research.
Figure 11: Maximum number of packets between
7. LIMITATIONS OF THE EXPERIMENTAL the first packet with a long delay (>0.3s) until the DESIGN delay is below 0.3s again The experiments’ greatest weakness is the low number of flights. Due to new laws and problems with Austria’s flight control agency Austro Control, the authors were only able to 6. DISCUSSION conduct twelve comparable flights of which three were done without video adaptation. “Comparable,” however, does not In this paper three hypotheses were presented: mean that all influencing factors were actively controlled. On the contrary, the experimental design did not control the 1. The delay is related to video quality. UAV’s flight path and the antenna orientation. As laid out 2. The delay is a good indicator of impending problems in the introduction, Wi-Fi connectivity is highly impacted with the video stream. by antenna orientation, movement speed, and height. All these factors were not logged and not fixed. It is possi- 3. The delay can be the input for an adaptation proxy ble that the flights without adaptation were (unconsciously) that improves the overall video quality. conducted with a worse UAV positioning. As with the ori- entation of the UAV, the orientation of the base station was 11. REFERENCES also not controlled. [1] T. Andre, K. A. Hummel, A. P. Schoellig, E. Yanmaz, Another significant weakness is that the Wi-Fi equipment M. Asadpour, C. Bettstetter, P. Grippa, was used with default settings. It is likely that the settings H. Hellwagner, S. Sand, and S. Zhang. can be tweaked to improve performance and reliability of the Application-driven design of aerial communication connection. Fixing the channel’s bit rate or disabling block networks. IEEE Communications Magazine, acknowledgments could change the video stream’s behavior. 52(5):129–137, 2014. Another weak point is that the described solution requires [2] M. Asadpour, B. Van den Bergh, D. Giustiniano, acknowledgment packets, which are not sent for multicast K. Hummel, S. Pollin, and B. Plattner. Micro aerial transmissions. vehicle networks: An experimental analysis of challenges and opportunities. IEEE Communications 8. REPRODUCIBILITY Magazine, 52(7):141–149, 2014. These shortcomings in the experimental design violate a [3] Ascending Technologies. http://www.asctec.de/uav- cornerstone of the scientific method: reproducibility. While applications/research/products/asctec-firefly/. Online, the configuration and setup of the UAVs is easy to repro- no date. Accessed 2014/05/17. duce, the evaluation of Kofler’s congestion detection algo- [4] İ. Bekmezci, O. K. Sahingoz, and Ş. Temel. Flying rithm relies on preliminary “ad-hoc” tests. The aforemen- ad-hoc networks (FANETs): A survey. Ad Hoc tioned problems with Austro Control prevented the authors Networks, 11(3):1254–1270, 2013. from designing the flight tests more carefully and forced [5] M. Bhaskaranand and J. Gibson. Low-complexity them to rely on data gathered during “proof-of-concept” test video encoding for UAV reconnaissance and flights. surveillance. In Military Communications Conference, Future work should therefore be mindful of these influenc- 2011 - MILCOM 2011, pages 1633–1638, 2011. ing factors: [6] A. Cabellos-Aparicio, F. Garcia, and • The position of the base station and the UAV take-off J. Domingo-Pascual. A novel available bandwidth spot should be kept stable. estimation and tracking algorithm. In IEEE Network Operations and Management Symposium Workshops, • The UAV’s flight path should not be manually con- pages 87–94, 2008. trolled. Flying to a predefined set of GPS coordinates [7] GStreamer. http://gstreamer.freedesktop.org. Online, should be preferred. no date. Accessed 2014/06/11. [8] I. Kofler. In-Network Adaptation of Scalable Video • The UAV’s altitude and cruise speed should be prede- Content. PhD thesis, Alpen-Adria-Univerität fined. Klagenfurt, 2010. • The UAV’s yaw (and thus the antenna’s orientation) [9] I. Kofler, R. Kuschnig, and H. Hellwagner. In-network should be predefined. adaptation of H.264/SVC for HD video streaming over 802.11 g networks. In Proceedings of the 21st • The interferences of the fuselage and the mainboard International Workshop on Network and Operating on the antennas should be reduced by placing them on Systems Support for Digital Audio and Video, pages a base that extends below the UAV. 9–14. ACM, 2011. [10] M. Lindeberg, S. Kristiansen, T. Plagemann, and • All wireless traffic could be recorded by a third lis- V. Goebel. Challenges and techniques for video tening station that is positioned along the flight path. streaming over mobile ad hoc networks. Multimedia Later analysis of this data might yield useful informa- Systems, 17(1):51–82, 2011. tion. [11] Logitech. http://www.logitech.com/en- However, as touched upon by [1, 2, 14] reproducibility for us/product/hd-pro-webcam-c920. Online, no date. UAV networks and the design of testbeds are still open and Accessed 2014/05/17. difficult problems. [12] R. Prasad, C. Dovrolis, M. Murray, and K. Claffy. Bandwidth estimation: metrics, measurement 9. FUTURE WORK techniques, and tools. IEEE Network, 17(6):27–35, 2003. The results of these experiments suggest that the delay [13] T. Schierl, T. Stockhammer, and T. Wiegand. Mobile is a good indicator for the link quality. It can be used to video transmission using scalable video coding. IEEE adapt the quality of the video stream to fit the available Transactions on Circuits and Systems for Video bandwidth. Further research should first establish whether Technology, 17(9):1204–1217, 2007. the effect indeed exists and how (statistically) significant the effect really is. A follow-up study could repeat the experi- [14] E. Yanmaz, R. Kuschnig, and C. Bettstetter. Channel ment described in this paper, but take great care to conduct measurements over 802.11 a-based uav-to-ground the experiments in a reproducible way. links. In IEEE GLOBECOM Workshops (GC Wkshps), 2011, pages 1280–1284. IEEE, 2011. 10. ACKNOWLEDGMENTS The work was supported by the ERDF, KWF, and BABEG under grant KWF-20214/24272/36084 (SINUS). It has been performed in the research cluster Lakeside Labs.
A Lightweight and Wide-Swath UAV Camera For High-Resolution Surveillance MissionsA Lightweight and Wide-Swath UAV Camera For High-Resolution Surveillance Missions