The problem AvatarPose solves

Overview

This project uses Machine Learning + zero-knowledge proofs to create NFTs for users. It uses face landmark detection to detect facial features in the uploaded video. These poses can then be used to animate an SVG avatar.

This approach to NFT generation aims to create NFTs from “lived experiences.” We choose to call these experiential NFTs.

The problem

1. Current NFTs often lack a meaningful association to their owner or buyer, making them absent of any sentimental value. Without a personal experience tied to them, they feel more like digital assets than cherished memories.

2. Current NFTs lack a meaningful association with the creator. There is no trace of the creator's hand in the work.

The solution (Benefits of using this approach to NFT generation)

• Value derivation. Our NFT objects have a legitimate claim from where they derive value (unlike most of the stuff on the market now). This is the key value proposition for this project.

If you know that an animation uses poses objects based on a recording of Vitalik, then you'll surely be interested.

“Using poses of a certain individual for your avatar allows you to share in their experience.”

• Market differentiation. This approach is unique and brings much-needed innovation to the NFT space.

User Interaction and Data Flow

1. A user logs into the platform using a social login - we use a wallet abstraction service.

The user uploads a video (< 5s) to the DApp frontend.

2. The ezkl backend receives the video frames and processes them to detect human facial landmarks. We use ezkl to run our face landmark detection model network.onnx on the image input.json. Ezkl produces a proof.json file that contains the detected facial features and a zk proof.

3. Our dapp backend then stores the following NFT metadata on IPFS:

• facial landmarks
• svg image
• original video (optionally encrypted before storage)
• Some extra info e.g. name of NFT

Now, we mint the experiential NFT and then send our data to NFT.Storage for enduring preservation. NFT.Storage is a Filecoin service specifically designed for the long-term preservation of NFT data with a one-time payment ($2.99 per GB).

This service requires us to upload a CSV that contains a list of token IDs and CIDs for our NFTs.

4. At the frontend, a user queries the detected facial landmarks plus svg avatar and uses these to generate an animation (gif or video). They can then list (or show off!) this animation on an NFT marketplace.

NOTE on PRIVACY:

a) The main intended use case is for famous individuals to create NFT poses and list them. Users can then buy the facial landmarks and choose an SVG avatar for them. This means that the videos are already public (no privacy concerns).

b) In case a user wants to have their data private, we can also do that. Ezkl provides for inference using a private ML model and private data. This is different from a) where we use a private model + public data.

The project architecture and development process

The main components are:

1. frontend (browser)
2. ezkl backend (via REST API)
3. blockchain (AttestSingleProof.sol and ERC721 contracts)

Key differentiators and uniqueness of the project

This project showcases how NFTs can be meaningfully associated with the creator or buyer.

Trade-offs and shortcuts while building

• Model quantization. The inference model uses half-precision and its current size of 360kB. We can quantize it to about 200 - 250kB. We didn't do this for the hackathon because the onnx quantization feature does not out-of-the-box produce a model compatible with tract. Side note: Ezkl uses the tract onnx rust library
  Reducing the model size would reduce the pk size produced by ezkl by an estimated >30%. This means faster proof generation and less memory usage.

• Model splitting. We can also split the model into sub-models for faster processing. At the moment, it takes ~7 minutes to generate a proof. This isn't bad, but if we need faster speeds we could use the option mentioned here.

• Single proof attestation. In this demo, we are using Single proof attestation AttestSingleProof.sol (i.e the contract deployment can only attest for a single proof). This is because our Halo2Verifier.sol which would have verified for any number of proofs that use our vk and pk, is big (1MB) and can't be deployed at the moment. We still need to make modifications (mentioned above) to reduce its size.