AI Integration into Decentraland

by 0xedf458ce355c755e1973778b51b917066d4c0bad (TonyMo)

Should the following $24,000 grant in the Platform category be approved?

Abstract

This is a resubmission of the previous grant for LLMs integration. The reasons for resubmission are:

• Several key VP holders supported the grant idea and were inclined to vote 'YES.' However, due to a limited budget and competing funds in the same category with DCL Metrics, they opted to vote for them instead.
• The previous grant was overly technical. This time, I'll highlight practical use-cases for better understanding.
• Additionally, we've made a progress that we're eager to share.

Grant size

24,000 USD in DAI

Project duration

3 months

Beneficiary address

0x64ac332de46656CEf7733375252CaC593997CE57

Email address

digalgfund@pm.me

Description

This grant is aimed to bring AI technology to Decentraland through Open Source tools.

Obviously, AI & LLMs (Large Language Models) have changed the world in the last year. We hear from all that we will not interact with the world in the old manner. That's why we need to stay updated in this field in Decentraland.

This proposal outlines an advanced AI integration system with applications in virtual environments and language processing. It includes a customizable AI character, multilingual interaction capabilities, and diverse backend setups compatible with various LLMs like OpenAI and Open Source Models. Key tools such as MemGPT and AutoGen enhance AI character detailing and interaction.

The system's use cases are versatile, covering virtual guides in Decentraland for navigation and information, interactive storytelling and roleplay, AI companions for social interaction, innovative educational modules, and AI-driven customer service representatives, decorations, quests, audio & speech features.

Progress from the last proposal:

1. Chat-bot for Golf Craft Game: Deployed in Genesis City
2. Chat-bot for Decentral Game ICE Poker
3. Multilanguage Q&A
4. Image generation integration
5. Retweet by LangChain(a widely used library for building AI/LLM applications) - They highlighted that it is a completely new interface for AI.

These tweets received 55K impressions! Many likes, views, etc. Several developers (non-DCL related) asked me in DMs how to build on Decentraland + AI. This brings in completely new people to the board.

Planned features (all open source):

• Frontend library for AI in SDK7 (includes NPC Characters, Shapes for Image Generations, prompts for music, voice, text generation).
• Backend server with abilities to use proprietary API (OpenAI, Claude2) and Local Open Sourced models.
• Backend for running local models (ollama) and tools (like MemGPT, LangChain, etc).
• Comprehensive documentation on how to run it.

Features available:

• Chat Bot NPC knowledgeable about specific projects, by uploading PDFs, links, websites.
• Multilanguage Q&A Characters.
• Image Generation.
• Voice Generation (new level of interactivity).
• Music Generation.
• AI role play NPC - predefined stories and behavior by creators.
• First experiments of integrating Web3 functionality to AI, including NPCs with their own Ethereum addresses for transactions.

Roadmap and milestones

1st Month:

• Backend development for a proprietary API (including OpenAI and Claude 2) and local open-source models.
• Backend setup for running local models (such as Ollama) and tools (like MemGPT, LangChain, etc.).
• Development of a frontend library.

2nd Month:

• Creation of a Chatbot NPC (data from PDF, website links).
• Development of multilingual Q&A characters.
• Implementation of image generation.
• Integration of voice generation capabilities.
• Integration of music generation capabilities.

3rd Month:

• Development of AI role-play NPCs, predefined by the creator with specific stories and desired behaviors.
• Initial experiments in integrating Web3 functionality into AI. This includes NPCs having their own Ethereum addresses, enabling them to receive or make payments.

This proposal is summarized due to technical limitations. To view it complete and vote on it, visit the DCL DAO Governance dApp