Learn How Alibaba Engineers Accommodated for Face Masks in their Face Recognition Algorithms
In order to win this inevitable battle and fight against COVID-19, we must work together and share our experiences around the world. Join us in the fight against the outbreak through the Global MediXchange for Combating COVID-19 (GMCC) program. Apply now at https://covid-19.alibabacloud.com/
By ApsaraDB
With the increasing threat of the novel coronavirus, social distancing is an important means of preventing the further spread of the virus. In China, soon after the virus started to spread, many local governments imposed strict regulations on the movements of residents. In particular, on February 3, the Yuhang district of Hangzhou city issued the “Ten Strict” control measures:
These restrictions brought new challenges and problems to community management. For example, most community complex entry and exit records were done manually, which would often result in errors and omissions, making it difficult to enforce this social distancing measure effectively.
To solve these access management problems, the Alibaba Cloud Intelligent database vector search team provided a set of facial recognition models for free. And, importantly, given the times, these models could work well even when people are wearing face masks. The vector search capabilities helped to set up a community resident management solution, which has been made open source.
The solution that Alibaba Cloud implemented effectively improved the efficiency of community access management during the epidemic in China, and it can be used to do the same elsewhere. Our team at Alibaba we were able to provide these systems and the AnalyticDB database engine free of charge for all the access management applications that are connected or related to the coronavirus epidemic.
Next, we will introduce our solution and give a detailed description of the technologies involved in facial recognition and the AnalyticDB vector search to help developers perform secondary development. Last, we will provide the address of the open-source product.
Community Resident Management Solution
What the Community Resident Management Solution Does
1. Automatically register the population, basic information, and facial features of people who live in the community, as shown below:
2. Use cameras to automatically recognize faces and report all entry and exit records of the family of the detected person. This makes it easier for community administrators to efficiently manage entry and exit. During this time, for obvious reasons, people often wore face masks, and keeping them on was important, as having to remove face masks would increase the risk of contracting the disease. Therefore, our solution provides a set of algorithms to enable facial recognition when people are wearing face masks, as shown below:
3. Retrieve access records of residents by using any combination of face photos and structured information. The solution also features statistical analysis capabilities, which provide global metrics for community administrators.
Overall Application Architecture Design
The figure to follow shows the overall architecture of the access management system. The frontend interface is implemented through HTML and JavaScript. Its functions support face access control even when a face is covered by a face mask. You can query entry and exit records for all family members over the past two days through facial recognition. In addition, you can register residents and search for entry and exit records in the background based on the combinations of face photos and structured information.
The facial recognition module converts videos that contain faces into adult facial feature vectors. This module mainly uses the facial detection and tracking module of the Seetafce engine as well as the facial recognition, eye recognition, and mask detection models developed by the AnalyticDB team. AnalyticDB stores and queries all structured data in the application and facial feature vectors generated by the facial recognition module.
Key Technologies
Facial Recognition Algorithm for Epidemics
The following figure shows how the algorithm works. During the registration process, the facial recognition and eye recognition models are used to extract the whole face features and eye features of the registrant. Then, the extracted feature vectors are written to AnalyticDB. During the query process, the mask detection model is first used to check whether the detected person is wearing a mask. If not, the features of the whole face are used to search for similar features in AnalyticDB. If the similarity between stored features and the facial features of the detected person meets the set threshold, the corresponding result is returned.
When a person is wearing a mask, the nose, mouth, and other features cannot be detected. Therefore, the extraction model for whole face features cannot accurately retrieve a correct match. In this case, the eye recognition model is used to extract the features of the visitor’s eyes, forehead, and other parts of the face that are not blocked by the mask. Then, the model searches for a match among the eye features stored in AnalyticDB.
The facial recognition model, eye recognition model, and mask detection model used in the system are all open-source. Testing showed that the accuracy of the mask detection model is greater than 99.5%. The following table lists the accuracy of the fa recognition and eye recognition models on common academic datasets.
As you can see, even when only eye features are used, the AnalyticDB model still has a recognition accuracy of over 99% on the LFW dataset.
Features of the AnalyticDB Vector Edition
AnalyticDB is a high-concurrency, low-latency, and real-time data warehouse on Alibaba Cloud, which supports the petabytes of data. It supports instant multi-dimensional analysis and service exploration for trillions of data entries within milliseconds.
AnalyticDB for MySQL is fully compatible with the MySQL protocol and the SQL:2003 standard. AnalyticDB for PostgreSQL supports SQL:2003 and is highly compatible with the Oracle syntax ecosystem. Currently, both products support vector search and similarity query and recommendation systems for faces, human bodies, and vehicles.
In actual application scenarios, AnalyticDB can query billions of vector data entries and respond within 100 milliseconds. AnalyticDB has been widely used in security projects across multiple cities.
In general application systems that involve vector search, developers usually use a vector search engine, such as Faiss, to store vector data and then use relational databases to store structured data. This means you have to alternate between both systems during queries. Moreover, this solution requires extra development work and does not provide optimal performance.
AnalyticDB supports the retrieval of structured and non-structured data, or vectors. This means you can simply use an SQL interface to quickly implement image search and hybrid image + structured data search. In hybrid search scenarios, the optimizer of AnalyticDB selects the optimal execution plan based on data distribution and query conditions in order to achieve optimal performance while ensuring retrievability.
In the access management system, AnalyticDB allows you to query access records by combinations of the photo, gender, age, start time, and end time conditions.
With the image + structure data search function, you can run an SQL statement to perform the following operations:
The demo.person the table stores the basic information of each person, demo.face_feature stores facial feature vectors, demo.access_record stores all access records, and pid indicates the unique ID of each user.
Hybrid structured + non-structured information retrieval is widely used in practical applications. For example, when the facial access control system is deployed in multiple communities, we can use a single table to store the facial features of the residents of all communities. Then, when performing a face search, we only need to search the facial features of the current community. In this case, you only need to add the condition where community name = ‘xxx’ to the SQL statement when using AnalyticDB.
While continuing to wage war against the worldwide outbreak, Alibaba Cloud will play its part and will do all it can to help others in their battles with the coronavirus. Learn how we can support your business continuity at https://www.alibabacloud.com/campaign/supports-your-business-anytime
