The problem Digital Verse solves

Problem/Annotation

Deep Fake is realistic transformation of audio and video using artificial intelligence. Currently, there are several applications of this technology.

1. In negative context.
   I can temporarily become someone else and say or do things that this person will never do. For example, Trump in youtube video can declare war on North Korea or Obama can say inappropriate things about his opponents. Spreading false information, invading privacy, and destroying your reputation are just a small part of what Deep Fakes can do.
   Technology is developing fast and it is only a matter of time before fake videos will be no longer distinguishable from the original.

2. In positive context.
   The use of technology for entertainment or to optimize the production of video content. As an example - editing, internationalization, and personalization of advertising, marketing, and electronic educational videos in different cultures.

You can shoot only the original video and change faces and localization in it depending on the audience. A potential customer can also become the face of advertising. Or Keanu Reeves will be able to freely speak Russian, Japanese and any other language.

Solution

Our solution is software with which you can create Deep Fake videos for good purposes and recognize them to prevent using in negative scenarios. With the help of integration with the blockchain and NFT, the user can digitalize his face and sell the rights to use it in any video, confirm the originality of other videos with his participation.

Use case:
Celebrity/user login in our platform, digitalizing his face/voice (via machine learning algorithm) and sells the rights to use it in video-advertising (b2b) or in funtech (b2c). And all this without the need for real filming and with digital confirmation on the blockchain.

Challenges I ran into

We have developed a scalable architecture that consists of several modules:

Module for replacing faces on video

It consists of machine learning algorithms written in Python. There are 3 main parts:

Data preparation

Each uploaded video is divided into frames, on each frame we detect the faces, process them in a certain way (align, improve the resolution) and use in next modules.

Model training

For each celebrity/user we train a model on the GPU using it's facet, which later allows you to replace faces almost instantly without any additional training.

Video merging

Using the trained models, we replace faces on frames from uploaded video and generate a new video using various additional functions for color correction, quality improvement and etc.

Smart contracts that allows to mint and sell NFT tokens

Our contracts has all needed basic functions to work with NFT tokens:

• Mint a NFT token.
• Transfer any NFT token on other address.
• NFT marketplace

Backend server written in golang for interacting with machine learning algorithms, implementing business logic and interacting with the ethereum blockchain.

Current implemented workflow in the demo:

1. The user uploads the video to the platform;
2. With the help of the face replacement module, the video is divided into frames, a faces is detected on each frame. Using a machine learning model, this faces are replaced by another and then a new video is generated;
3. Next step is to upload the generated video to IPFS storage;
4. Then the link to the video in IPFS storage and additional information (name, description) is sent to the smart contract to create an NTF token. The token can be transferred, sold. The token structure can be checked by Unmarshal API.