An AI solution that seamlessly swaps a character’s face with the user’s

This case study showcases our contribution to creating a solution that replaces a character's face with the user's face in a short, comical video.

Entertainment
Industry
25 specialists
Team size
Australia
Location
2 years
Project duration

[ Business context ]

How the client came to us with the concept for face animation software

The client approached us with a specific and entertaining requirement: they needed a software solution to replace a character's face in a short humorous video with the user's face.

Additionally, they desired basic animation to enhance the comedic effect.

[ Product scope ]

The Onix team was responsible for:

Gathering requirements to define objectives and project scope

Ensuring an effective and transparent development process

Creating an efficient and accurate face detection and replacement system within video streaming contexts

Developing scripts to determine the coordinates of the face oval

Creating an algorithm for determining characteristic points on a cropped face (conditional geometric center and points of the mouth corners)

Adding the ability to work with various accessories (hats, glasses, wigs, etc.)

Implementing reliable and advanced technologies to ensure high software operation and proper functioning

Have a project in mind?
Let's talk!

The Onix team developed a dependable solution that serves the interests of six distinct user groups:

General consumers

Any individual who wishes to create a personalized video for entertainment or to share with friends and family.

Marketers and advertisers

Those who wish to create engaging ad campaigns where the potential customer sees themselves in the ad, thus increasing the likelihood of the ad's message being remembered.

Social media influencers

Individuals looking to create engaging content for their followers to increase their reach and engagement.

Event organizers

Those organizing events (like birthdays, weddings, or corporate events) might use this software to create customized invitation videos or memorable highlights.

Video game development

Developers might use face replacement to allow players to insert their own faces onto game characters, enhancing the personalization and immersion of the gaming experience.

Educational institutions

Teachers might use face replacement to customize educational videos, making them more relatable to a specific audience or culture.

Solutions we provide

Inanimate object face replacement

Unlike human faces, animals or inanimate objects won't have the same geometric structure or landmarks. Placing a human face on them requires a dynamic approach that can adjust to various shapes and structures. Moreover some objects or animals move unpredictably, which complicates the tracking and anchoring of the human face onto them.

The Onix team implemented a mapping algorithm that doesn't solely rely on traditional facial landmarks but also uses the broader contours and shape of the target entity, whether it's an animal's face or an object. Moreover, we used advanced tracking algorithms in OpenCV to keep the human face consistently placed on the moving object.

Accessory augmentation

As users move or change expressions, accessories should adjust accordingly. A hat, for instance, should remain on the head, not float above it if a user raises their eyebrows. Also, different users have different face sizes and shapes. The accessories must fit proportionally, looking both too large and too small.

We used facial landmarks to create "anchor points" for accessories. For instance, glasses anchor to the bridge of the nose and ears, ensuring they move with the face realistically. Also, we developed an algorithm that adjusts accessory size based on detected face dimensions. For instance, the width between the eyes can determine the size of glasses.

Performance efficiency

Regardless of where you make changes - on the mobile version or web, all modifications are automatically reflected across all platforms.

For example, if you change the color of a button, this change will be applied both on mobile devices and on the website. This ensures design consistency and ease of management, as you don't need to make separate changes on each platform.

All changes are automatically synchronized, allowing you to efficiently control the appearance and functionality of your product on different devices.

Business goals we helped our client to achieve:

Personalization

The solution empowers businesses to craft tailored content for their users, enhancing engagement.

Personalized content often results in better retention and conversion rates.

Engagement and interaction

Software fosters active participation and interaction by enabling users to feature in videos, resulting in a more immersive user experience.

Viral marketing potential

Incorporating personalized, humorous videos enhances the likelihood of them being shared on social media platforms, amplifying the reach of marketing campaigns.

Monetization

Businesses can introduce this as a premium feature or utilize it for advertising campaigns, potentially generating revenue.

Core technology stack we used

OpenCV,

Medipipe,

RetinaFace,

PyTorch,

Python,

FastAPI,

Elasticsearch,

React

Results

Thanks to the results of our collaboration, our client now has a powerful solution for replacing faces in video streams with high efficiency and accuracy.

The potential of using this AI-based solution spans various sectors, including entertainment, education, research, and forensics.