AI dapp to transform any 2D PFP NFT into a 3D Decentraland Wearable/Avatar

by 0x51159c749702295e745319935e4eddde238f3a7f

Should the following $108,000 grant in the Platform category be approved?

Abstract

We are excited to present an AI-based solution to an unsolved metaverse problem and lowers barriers to onboard NFT collectors to decentraland:

Problem:
Thousands of NFT collectors want to use the PFP NFTs in their wallets as avatars across platforms. Especially in 3D metaverse environments, this is difficult to pull off: The costs of rigging and format conversions of 2D PFPs to 3D Avatars are high.

Solution:
Beamit.space combines several cutting edge AI models to transform 2D PFPs into 3D Metaverse Avatars for Decentraland. Any PFP you want - you can soon use it in the metaverse. And this can be done by anyone:

1000x faster than ever before.
100x cheaper than ever possible.

Grant size

108,000 USD in DAI

Project duration

8 months

Beneficiary address

0x51159c749702295E745319935E4EdddE238F3A7F

Email address

decentraland@beamit.space

Description

We believe Beamit.space is a game changer in 3D Avatar generation and will change the game for Metaverse onboarding. Our early prototype proves that we can make this a reality within 8 months.

The app solves this in 3 stages:

Beamit.space is using cutting edge technology. We are merging AI with 3D rendering and web3 technology in a unique way.

1. Generative AI

Our Generative AI (gAI) module enables users to complete PFPs to create a full body image. Our prototype (using Dall-E) shows, that decent results are achievable without even explicitly training the model. So we anticipate substantial improvement potential from training our own model (we are currently favoring training the open source model Stable Diffusion). The goal is to generate a highly accurate 360° full body image of the PFP (frontside and backside, eventually also side views).

2. Depth Prediction model We use a cutting edge Depth Prediction model to transform the frontside and backside 2D full body images into 3D models. Even though the model was trained on photography, the results for illustrations and even pixels design are stunning. We anticipate to be drastically improving the prototype results by training the model on appropriate datasets.

3. Blender Cloud (a. Background removal, b. Merge & c. Rigging)

The third step serves to set to finetune the raw renderings, merge them to a full 3D Avatar and rig them with the Decentraland skeleton rig. For this step we are using the open source 3D creation suite Blender.

a. Python scripts first will detect and remove the background from both the frontside and backside gAI results.

b. Next the backside and frontside view 3D models are combined to create the initial stage of the full avatar 3D model.

c. The automated rigging is the most challenging part of the project. We are working on our own AI model based on existing 3D Shape Recognition models, as well as prediction models on how to position the skeleton in the avatar mesh. However, as perfect results are not to be expected in the ealry stage of this project, we will provide a blender cloud browser-application the user can use to adjust the rigging manually, to get a functioning avatar ready for publishing/submission.

Results: All of the blender add-on scripts as well as the custom trained AI models will be open sourced.

You can now check our prototype (with Steps 1. and 2. in place) and test it with your NFTs - please request via website form beamit.space.

Roadmap and milestones

We anticipate the project to be ready within 8 months from now - with the result of an open sourced alpha release as result. We are working according to this roadmap, and will update here as well as in detail on our medium:

8 - 2023 Proof of Concept 3D frontside body, Ethereum
Status of the project to date.

9 - 2023 Frond side + added backside Render generative AI model (gAI )

• gAI model producing Both front & back side results.
• blender cloud set up and script testing.
• Project presentation & start of social media activities.

10 – 2023 Full Body 3D Render AI model (zoeAI)

• custom 3D rendering model based on ZoeDepth depth prediction model (for front and back).
• Community building activities.
• Unisat Wallet Ordinals integration

11 – 2023 Refined gAI model

• Refined custom gAI model to achieve optimized results by training the model on web3 PFP data.

12 – 2023 – Blender Cloud: remove background

• Remove the backgrounds of all 3D body perspectives via python script.

01 - 2023 – Blender Cloud full body merge

• Creating full body 3D Avatar.

02 – 2024 refined zoeAI + rAI model

• Trained models to further increase render quality
• Community building: private Alpha whitelist.

03 - 2024 – Blender Cloud: Rigging

• Rigging the 3D Avatar in the cloud, with basic script support.
• Community building
• private Alpha

This proposal is summarized due to technical limitations. To view it complete and vote on it, visit the DCL DAO Governance dApp