Interpretation of Lumin OS, SDK and developer port

Magic Leap finally released its Lumin SDK file on the Magic Leap's Creator port this week, and we have the opportunity to learn more about the device and how it can be used.

Here are some explanations of Lumin OS, SDK, and developer ports.

Lumin OS

We know that Magic Leap's operating system is called Lumin. This system is based on Linux and Android open source projects. The app will be built on Magic Leap's own Lumin Runtime engine and run on Lumin. In addition to Lumin Runtime, ordinary 3D engines such as Unity and UE4 engines can also be used to develop Magic Leap content.

Magic Leap software stack

The next level of the Magic Leap One software stack is the OS service layer. At this level, we will go to a higher level with OS Core. For example, functions such as input management, communication management (Bluetooth and Wi-Fi), and power management are handled here.

At this level, Magic Leap also includes the company's service called Perception. These subsystems handle some of the more complex inputs, such as user and environment sensing, as well as graphics rendering pipelines, including Magic Leap for asynchronous time warping (ATW).

Using the Portable Operating System Interface (POSIX) API standard, the platform API layer acts as a gatekeeper between the developer and the spatial computing function, with a user-set permissions system. This layer also includes the Binder framework commonly used in the Android OS as an interprocess communication system, allowing applications to interact (HoloLens is especially missing this feature).

Lumin Runtime Engine

Currently, many immersive applications built on other engines must run independently, but Lumin Runtime allows applications to interact while exhibiting a "single, continuous experience." Developers can access APIs on 13 platforms to create different interactions, including scene graphs (used to determine coordinates and levels in space), collision physics effects, gestures, real-time stereo computing techniques (for rendering objects), controls, Light and shadow and the shadowing effect in the real world. This multi-application form in a single scene sounds similar to the 2016 "Waking Up with Mixed Reality" demo.

At some point, the Windows computer we know now is made up of the MS-DOS operating system. Windows is a graphical user interface (GUI) that runs on top of the operating system, so GUI+ might be a simple way to see what Lumin Runtime is. method. But in addition to providing a simple GUI, Lumin Runtime provides some more important features.

Jumping out, Magic Leap One uses a client/server model that allows multiple devices to share applications. According to the content in the documentation, this seems to be handled automatically. If so, it will bring the problem of controlling whether the underlying elements are shared or not shared, which is like a fresh air for those who have already developed a shared experience in HoloLens.

Other basic elements of Lumin Runtime:

3D model with material and bone 3D model animation

2D picture sprite animation

High security vacuum text rendering for spatial calculation

2D and sound field audio

2D and stereo video

Rigid body collision physics

Real-time particle effects

Input event and haptic control

gesture

Speech Recognition

Time-space computing technology for rendering objects, lighting, shadows, and real-world occlusion

It seems that Lumin Runtime is designed to provide facilities like 3D engines like Unreal or Unity, including things like Lumin Editor and UI Toolkit. In other words, the Lumin Runtime layer also supports both engines.

Lumin Runtime applications can work in parallel or together, while Unreal and Unity applications can only run as stand-alone applications because they have independent control over the device. Of course, they also operate in this form on other platforms, so it's nothing new. But developers are still very much looking forward to making some changes.

In this respect, Magic Leap does have a well-thought-out design for Lumin OS and a range of software. But when we really get the equipment, we can know what we can and can't do.

Platform API

The SDK release notes contain a number of amazing APIs. The eye tracking API can be used to determine the direction of the user's gaze point, the center of the eye, and to monitor the movement of the blink. This SDK also supports hardware masking and has an interactive interface to provide deep information to the platform.

Head posture

The Magic Leap One (ML1) is able to integrate the data collected by its range of sensors, as well as visual observations by users. If the user's field of view is affected by a dim environment or an occluded camera, the ML1 will automatically enable gyroscope tracking until the system re-collects enough visual data.

Image tracking

The ML1 can continuously capture visual data to build a map of the surrounding environment, and automatically recognize and give the previous content when returning to the previously modeled environment. So when you play games that chase small robots in your room, they will stay where you saw them before.

This technique sounds very similar to the previous research published by Magic Leap engineers, using a pair of neural network systems to identify "SLAM-ready" points on the image that can be used for tracking. The specific form is shown in the figure below.

gesture

The Lumin SDK supports 8 different gestures. Microsoft HoloLens supports 3 gestures (the fourth can be achieved with the Mixed Reality Toolkit), while Meta 2 can implement 7 different grab gestures.

Given that Magic Leap recently applied for a patent for translation sign language, we can expect to see more gesture recognition features on the developer platform.

Sound field audio

Just last week, Magic Leap applied for a stereo patent, which sounds like the sound field audio capabilities of the ML1.

Now we have determined that the company's exclusive technology will be open to developers. Stereo solutions based on sound field audio technology may soon land on Unity and UE4 engines, and the sound field API will soon be able to be called by native C++ applications and the company's own Lumin Runtime engine. In addition, Magic Leap also plans to support more middleware, including Wwise and FMOD Studio, which will be implemented after the end of GDC. At the same time, developers can also call the SDK of the Google Resonance Audio project.

illumination

Lumin OS enables surround ray tracing. An app can add light to a scene, but it can't be rendered without light. Despite this, developers can use bright colors to hedge the dark colors to create a dark feeling. At the same time, since the object does not cast shadows, the developer needs to illuminate the object from above. Lumin also supports the self-shadow of virtual objects.

Close to the cutting plane

Magic Leap officially stated that content that is far away from the user can achieve the best experience, and also reminds developers not to place virtual objects within 30 cm of the user for a long time. The company also encourages developers to “fall away” when users are too close to virtual objects, rather than suddenly letting the scene disappear.

In general, developing for Magic Leap isn't much different for people accustomed to developing Unity, but with some extra features, such as eye tracking. We are looking forward to what developers can do with the Lumin Runtime engine.

Nano Core For Electronic Watt-hour Meter

Anyang Kayo Amorphous Technology Co.,Ltd. , https://www.kayoamotech.com