LLMs Integration | Phase 1

by 0xedf458ce355c755e1973778b51b917066d4c0bad (8MetaGames)

Should the following $24,000 grant in the Platform category be approved?

Abstract

Open Source tools for integrating LLMs (AI) into Decentraland.

This grant proposes integrating Large Language Models (LLMs) into Decentraland, offering creators and developers out-of-the-box access to AI technology. Aimed at enhancing user experience and fostering innovation, it will provide open-source tools for building advanced, AI-powered virtual environments, marking a significant step in leveraging AI breakthroughs within Decentraland.

Grant size

24,000 USD in DAI

Project duration

3 months

Beneficiary address

0xedF458ce355C755e1973778b51b917066d4c0bAd

Email address

digalgfund@pm.me

Description

Large Language Models Integration in Decentraland

1. AI-Character Integration

• Customizable AI Characters: Front-end interface for creating AI characters.
• Multilingual Interaction: Characters respond in users' native languages.
• Backend Setup: Connection to various LLMs including OpenAI, Authrophic, Azure, and local installations.

2. Frameworks and Tools for LLMs

• MemGPT: Framework for crafting detailed AI characters.
• AutoGen: Tool for generating interactive NPC groups.
• Ollama: Facilitates local LLM integration.
• Prompt Journey: System for mapping NPC interactions.
• Integration Instructions: Detailed guidelines for integrating OpenAI and local LLMs, such as Llama, Mistral, and SDXL models.

3. Stable Diffusion in Decentraland

• Text-Prompt-Based Image Generation: Backend integration for dynamic visual displays.

Use-Cases

• Virtual Guides and Assistants: AI characters helping users navigate Decentraland, providing event info, or assisting in tasks.
• Interactive Storytelling and Roleplay: Customize characters for interactive storytelling and roleplay with users.
• Virtual Companions: AI characters as companions for conversation and social interaction.
• Educational Tools: AI characters in interactive learning modules or virtual classrooms for various subjects.
• Customer Service Representatives: AI characters in virtual stores or service centers for user assistance.

Existing OpenAI Integration

We've initiated AI technology integration in Decentraland with OpenAI for image generation, showcasing our commitment to advanced AI in virtual environments.

• GitHub Repository: Decentraland-LLMs-Image-Generation
• Demonstration Video: Twitter Video

Roadmap and milestones

Month 1: Integration of Chatbot NPCs and AI Platforms

• Chatbot NPCs: Develop and integrate Non-Playable Characters (NPCs) with chatbot capabilities into Decentraland.
• OpenAI and Claude Integration: Establish connections with AI platforms like OpenAI and Claude.

Month 2: Integration of Local LLMs and Diverse AI Capabilities

• Local LLM Integration: Integrate local Large Language Models for versatile AI interactions.
• Runpod Integration: Implement Runpod for dynamic AI functionalities.
• Text and Image Generation: Develop AI-driven text and image generation capabilities.

Month 3: Integration of LLM-Based Characters

• LLM-Based Characters: Finalize and integrate LLM-powered characters into Decentraland.
• Final Testing and Optimization: Conduct extensive testing and optimization for seamless functionality.

This proposal is summarized due to technical limitations. To view it complete and vote on it, visit the DCL DAO Governance dApp