The problem SAR for Natural Disasters using Drones and CV solves

Natural calamities affect life, economy, and infrastructure. Through this project, we try to develop methods to reduce harm to the human population during such disasters. When natural disasters like floods affect areas covering entire cities the traditional methods of search and rescue of life fail as there are never enough resources and manpower with the rescue teams to cover such areas and carry out rescue operations.
We are devising a Computer Vision and Drone based technique for identifying victims during such disaster and alert the rescue teams with such locations so that concentrated rescue operations can be carried out and more lives can be saved in the least possible time.
Drones can be deployed in such areas. When a victim sees a drone flying the person would try to capture the attention of the drone by waving hands or respond in a way that can be generalized as a set of standard responses in such situations. The live video feed would be processed by the Deep Learning-based system on-board the drone that is trained on simulations of these situations. The system would look for humans in the video and if found would do further processing to determine whether the person needs help or not. If it is determined that the person needs help then the location of the place would be sent to the rescue teams.

Challenges I ran into

The first challenge we faced was the dataset to train our deep learning models on. For the task, we needed datasets of videos and images captured from a height at which a drone would fly. At the time of starting very few such datasets were available. We recorded and annotated our own dataset for testing and used the Okutama-Action dataset for training models.
The second challenge we faced was that a lot of times Deep Learning models learn background features of datasets they are trained on in addition to the desired features. In such cases, the accuracy of the model decreases dramatically when there are significant variations in the data the model is being used on and what the model was trained on. The approach for a background-invariant object detection system was devised and implemented to overcome this issue.
Another major issue was the detection of actions of humans for the determination of whether the person need help or not. For this, a variety of Deep Learning models were trained with significant variations in training parameters and tested for accuracy. The results were very unsatisfactory and completely useless for deploying in real-world scenarios. A multi-layered system consisting of Object Detection and Human Pose Estimation was created which gave good results and is currently being worked upon for increasing accuracy and making it faster for real-time usage.