Develop new Low Latency Video Streaming components for SDK and builder.

by 0xb7c0dc404c2ce1b2226c709eca6d8479b1fb80dc (test#80dc)

Should the following Tier 4: up to $60,000 USD, 6 months vesting (1 month cliff) grant in the Platform Contributor category be approved?

Abstract

Video streaming in Decentraland has a 10-20 sec delay causing interactive video apps to be infeasible. Sub 1 sec streaming is set to grow significantly over 5 years for apps in education, media and entertainment.

Facebook's metaverse is built with Ultra Low Latency at its core. Decentraland is MISSING this feature. Let's fix this!!

Video of working prototype

https://www.youtube.com/watch?v=ml7nlTEk3p8

Grant is to complete development to Dcl's dev standards.

About me

• Expert in low latency remote control systems, IoT and WebRtc
• 4x CTO. Delivered over 500 software projects.
• 857 Github commits in last year

Grant size

48,000 USD

Beneficiary address

0xC8Fede4B21b462b1b5fEB0AcFC0513012B1795F8

Email address

lowlatencydcl@gmail.com

Description

Did you know, communication with astronauts on Earth’s moon has a latency (delay) of 1.25 seconds?

Due to the protocols used, live video in the Decentraland world is delayed at around 10-20 seconds. This makes live events infeasible. For example, live video calls or interactive events take 20 seconds to get a response.

This project introduces new ultra low latency streaming components to the decentraland SDK and builder to support ultra low latency video. Latency will reduce to ~1 second.

These components will allow scene builders to easily incorporate ultra low latency video in their scenes.

Background

In recent years, global investment in technology and infrastructure underlying ultra low latency video streaming has been significant. Live streaming is set to explode in the coming years with low latency video being a big driver behind new applications in entertainment, gaming, media, education, acedemia.

Ultra low latency video streaming typically operates on a sub 1 second delay. This project will integrate this hot, deep technology into the Decentraland platform.

Competitive Advantage

Metaverses such as Facebook's Meta already have native low-latency video and are building a strong portfolio of apps based on this technology. Other Metaverses do not yet have low latency video streaming technology.

Experimental Prototype

A video demonstration of the components I have developed is here:

https://youtu.be/ml7nlTEk3p8

This video shows a working prototype of an experimental Decentraland SDK component featuring a low latency video stream.

The video is streaming from North America to a video streaming server in Hong Kong (on Microsoft Azure) and then back to North America and into a Decentraland scene running in the browser.

The clock times on the right side are the current time whereas the time on the left, in the Decentraland scene, are latent due to the transit time around the planet and the subsequent decoding and rendering of the video.

Total time is demonstrated to be around 1 second.

A 5 year old laptop with no GPU was used for the test. The video resolution is set to 640x360.

The experimental prototype includes modifications to the Decentraland Unity Renderer and Kernel and adds a new video component to the SDK.

Example Use Cases

It is difficult to imagine a virtual future that runs on a 20 second delay. Use cases that need lower latency might include:

• Any live interaction such as video calls.
• Live events where people communicate from the virtual world to the real world.
• Any use cases that include augmentation of the virtual world with a real object (augmented virtuality).
• Virtual workspaces
• Virtual learning spaces that requires interaction
• Remote control of real world devices such as drones. (yikes!)
• Immersive commerce
• Gaming applications

Existing DAO pre-Proposal

An existing pre-proposal that covers this feature is here, from Tobik#7989. Note however this is a request for a protocol change, the proposed feature would likely be an additional component so that backwards compatibility is maintained:

https://governance.decentraland.org/proposal/?id=0627a110-2fd6-11ec-ac84-77607720a240

Specification

High Level Summary

The project objective is to create a feature to enable scene builders to easily incorporate low latency video into their scenes. To be successful the feature needs to be well engineered, tested, documented, incorporate ideas from dev and user communities and comply with Decentraland standards.

Deliverables

• Deliver WebRTC ultra low video and audio streaming components to the Decentraland SDK.
• Support for Decentraland web client and feasibility of development on new Desktop client on supported platforms.
• Complete all relevant requirement, archiectural, design, code, test and deployment work to decentraland's standard via github.
• Run relevant user testing and trials.
• Operate a streaming test server until mid November 2022 for development and community testing.
• Create test scenes, sdk documentations, blogs and assist Decentraland marketing team on new feature launch.
• Be responsive to community and be present in relevant channel on Discord.
• Attend some sweet dcl live events that use the new technology.

Preliminary Work Completed

A feasibility study was completed prior to grant application request:

• Talking to potential users and understanding their applications.
• Review of code and relevant architectural components.
• Reading Dcl code standards and architecture documentation.
• Research of ECS (sdk) structure and dev of new components.
• Review of existing decentraland video components.
• Review of WebRTC options for Unity and their pros and cons.
• Building new environment across Explorer, Kernel and Unity Renderer. Building and debugging scenes.
• Experimentation and development of prototype including modifications the the sdk Unity Renderer and Kernel.
• Running live streams in Preview scenes in decentraland.
• Testing live streams on Mac, Windows and across browsers. (including mobile testing outside of dcl explorer)
• Setting up professional streaming server in Hong Kong (Azure) and running global tests clients in North America and Asia.
• Achieving sub 1s in a scene in dcl explorer and performance testing (fps-performance).
• Testing audio performance.

Personnel

About the Developer

A 4x CTO with a focus on smaller entrepreneurial companies. Successfully lead and delivered 500+ software projects. Main development language C#.

Last project was an enterprise grade worldwide remote control and entertainment platform, cloud native on Microsoft Azure that includes low latency video and remote control via IoT. Video and audio delivered globally over WebRtc.

Software architect with extensive experience across requirements gathering, analysis, design, coding, testing, deployments, cloud computing.

Projects in many domains including financial software, CTO of software development house, entrepreneur - build a number one major financial software product line in SE Asian country.

Founding member of an early crypto ICO. Lead team delivering Sidechains, Masternodes and early payment channels.

Other relevant software skills Typescipt, Javascript, Unity. Backend APIs.

Education - University of Manchester, UK.

More specific credentials available upon request at: lowlatencydcl@gmail.com Discord: carl#9146

Roadmap and milestones

Scope

• Deliver WebRTC ultra low video and audio streaming components to the Decentraland SDK.
• Support for Dcl web client and feasibility of development on new Desktop client.
• Complete relevant requirement, architectural, design, code, test and deployment work to decentraland's standard.
• Run relevant user testing and trials.
• Create test scenes and sdk documentation.
• Be responsive to community and be present in relevant channel on Discord.
• Attend some sweet dcl live events that use the new technology.

Milestones One

• Goal here is to leverage the experimental prototype as a starting point for feedback from the development community.
  • Commit prototype to test branch in Dcl github repositories (Unity Renderer).
  • Build test scene(s) that run on test servers.
  • Put test scenes live. Document current architecture.

Milestone Two

• Collect all feedback on prototype and formulate detailed plan including:
  • Capture architectural decisions.
  • Feasibility of supporting Desktop Dcl client on supported and future platforms.
  • Full development plan.

Milestone Three

• Main development
  • Execute plan from Milestone Two that delivers on the development scope.

Milestone Four

• Community related activities.
  • Be available on discord and Github to monitor any feedback and issues. Attend dcl events that use the tech and generally gauge success.

I'm looking forward to experiencing all the new cool scenes that can be built with this new tech.

Vote on this proposal on the Decentraland DAO