See What Alipay’s Doing with 3D Imaging

11.11 Big Sale for Cloud. Get unbeatable offers with up to 90% off on cloud servers and up to a $300 rebate for all products! Click here to learn more.

By Wen Lou.

Image for post
Image for post

As a fundamental piece of the visual world, images are an important means for us to obtain, express, and transmit visual data in the information age. Arguably, today the most efficient and accurate way to generate images is to generate, display, and draw images with computers. These technologies are collectively referred to as computer graphics. Computer graphics technology has become an important aspect in several different industries, including animation, film special effects, gaming, design, advertising, AR/VR, and data visualization. And, to better simulate human senses and improve the overall sensory experience, increasingly more industries are turning to 3D imaging.

Image for post
Image for post
Image source: https://www.vmovier.com/48189
Image for post
Image for post
Image source: http://movie.xunlei.com/reviews/103994

In the past several years, computer graphics technology has been a key area of research all over the world, not just China. It is a sophisticated and fast-moving field of research, with both theoretical and practical, industry applications being explored intensively. Whereas other fields like gaming and film are relatively stable and mature areas, web-based computer graphics technology is constantly evolving, full of future potential.

So, what is Ant Financial and Alipay doing with computer graphics technology and 3D imaging?

Well, at Ant Financial, we have always attached great importance to the interactive experiences of our customers. Therefore, computer graphics technology, especially web-based computer graphics technology, is also a key area of research and development at Ant Financial. Initially, we only thought of computer graphics as a means to enrich our existing product lines, but later we reimagined what the value of computer graphics is. And in doing so, we have developed a complete set of new solutions, covering infrastructural capability building, tool platforms, workflows, and implementation scenarios.

In this article, I will share, as a member of the Ant Financial’s research and development team, the problems that we encountered during our exploration into the above solutions and the ways we resolved these problems.

But before we get ahead of ourselves, let’s also cover some basic concepts so that we’re all on the same page.

What’s Real-time Image Rendering

In reality, the rendering of 3D images involves both offline rendering and real-time rendering.

Offline rendering is great for producing photo-realistic renders. However, its main disadvantage is that it may take hours or even days for a computer to render just one frame in more complex jobs. Offline rendering is often used in industry applications that require above all else-high image quality-like animated films and advertising illustrations.

Real-time rendering, on the other hand, is all about fast rending and time efficiency. In a human-machine interaction scenario, for example, rending must be quick so to allow for quick refresh rates and smooth animations so that the interactions between the user and the machine appear normal to the user. In such scenarios, a computer must be able to render images in a very short time, display it instantly, and continue on to prepare the next image. For these sorts of rending jobs, the computing capacity of CPUs alone is insufficient. Rather, they will use the parallel computing power of multiple GPUs. Real-time rendering is mainly used in human-machine interaction scenarios where images are displayed based on data in real time, in gaming, data visualization, and AR/VR applications.

In the Internet space, most scenarios in real life require real-time rendering. Today, although real-time rendering may compromise on image quality, we believe that the development of various technologies will boost the quality of real-time renders, making them appear more photo-realistic. In the future, what used to be achieved only through offline rendering will also be feasible through real-time rendering. Future AR/VR applications will definitely wow us.

How about Web-based 3D Image Rendering

To render 3D images efficiently, controlling both the hardware and graphic drawing interfaces are musts. To give browsers the same ability, the WebGL Working Group published WebGL specifications in 2011, which was then quickly implemented in various browsers. With WebGL, you can create dazzling 3D effects on web browsers. And, by using the cross-terminal and cross-client features of web browsers, you can obtain 3D content at low costs and load them on demand without downloading any 3D-application clients. WebGL also can be integrated with product content in super apps such as our own Alipay at a low cost to get the most out of their mobile traffic.

The combination of several dynamic web features and 3D rendering has made for dramatic changes and interesting applications:

  1. The commodity showcase area on Taobao can display 3D product models that give users a panoramic view of products.
  2. For one-time use needs such as building a 3D design, you no longer need to use a specialized, pro-level software program. Rather, you can just simply use the browser to receive a panoramic experience.
  3. Web-based 3D games can be downloaded instantly, allowing users to try them out at low cost and be distributed effortlessly, making it easier to reach more target users.
  4. Maps in browsers can be displayed in 3D, with models loaded on demand and dynamically rendered.
Image for post
Image for post
A 3D map on Amap, Alibaba’s mobile map app.

There are, of course, many other web-based 3D application scenarios, although they aren’t listed here.

While some scenarios may not seem complex, but implementing them directly with WebGL isn’t exactly easy. The catch is that WebGL only provides the most basic of graphic drawing APIs. And it also requires that you understand the internal details of WebGL and that you learn how to complete GPU programming using the WebGL shader language and have a good grasp of some complex mathematics to solve spatial relation and light-shadow effect calculation problems.

Therefore, at Ant Financial, it’s been our goal to package WebGL into a developer-friendly engine, which provides many common APIs and simplifies the overall construction process of the entire 3D project.

So, the below sections outline what we’ve been up to, providing a bit of a background story on how Oasis 3D came into being.

What Is Oasis 3D

Oasis 3D is a lightweight 3D engine that we developed at Ant Financial. It provides a rich set of functional components and supports complex 3D graphics rendering. With Oasis 3D, developers can easily create complex 3D environments without needing to have a background in complex graphics, making it much easier to create 3D environments.

Oasis 3D boasts the following features:

1. Microkernel architecture. The Oasis 3D kernel provides some basic rendering capabilities among many other things. You can extend Oasis 3D capabilities to build different engines for different scenarios, such as for the game engine, animation engine, data visualization rendering engine, and 3D rendering engine for AR environments. Drawing on UNIX’s ideas, Oasis 3D follows the philosophy of “do one thing, and do it well” in its design of every module. By integrating a complex and cohesive set of functions and exposing only user-friendly interfaces, Oasis 3D makes for a powerful and complex application whose capabilities do not negatively affect the maintainability of its overall functionality. Currently, Oasis 3D is mainly used in some interactive games and big screens at the Ant Financial. To support these scenarios, Oasis 3D packages common graphic functional modules, which provide the necessary support for our daily application development:

  • SceneGraph scenario structure: In complex graphic applications, the spatial relationship between scenes can be managed in SceneGraph.
  • Basic 3D rendering functions: They include light source, grid, geometry, material system, and camera.
  • Enhanced rendering functions: They include shadow, post-processing system, and atmospheric effects for scenes.
  • The external model supports glTF 2.0 and the supporting PBR material.
  • Rich animation systems: They include the skeletal animation, interpolation animation, and particle animation systems.
  • A set of powerful functional components: It includes collision detection, HUD parallel display, ray casting, framebuffer fetching, finite state machine, and math library.

2. Component-based development. Both component-based development (CBD) and the traditional object-oriented programming (OOP) can be used to extend the functionality of a class, but, at their core, they exhibit very different concept designs. In the traditional OOP, when a functionality extension is required, a base class is designed to create derived classes that inherit from the base class. While this method is very intuitive, it becomes problematic when more functionalities need to be extended.

For example, consider a base class, such as Animal. For animals that can fly, you must generate a derived class, FlyAnimal, and for animals that can swim, you must generate another derived class, SwimAnimal. Then, you may have to consider this question: what about the animals that can both fly and swim? First of all, you cannot inherit FlyAnimal and SwimAnimal at the same time, because both classes share the same base class. Second, if you generate a derived class named FlySwimAnimal based on Animal, you must rewrite the code of FlyAnimal and SwimAnimal, which means writing the same code twice.

CBD is free of this problem in terms of functionality extension. Again, consider a base class, Animal. To extend the flight ability, implement the component FlyComponent. To extend the swimming ability, implement the component SwimComponent. To use one or both components for an additional ability, simply add the appropriate one to Animal. For animals that can both fly and swim, simply add FlyComponent and SwimComponent to the animal for ability extension.

The following diagrams show the comparison between the structures of the OOP and CBD programs:

Image for post
Image for post
Comparison between OOP and CBD

You can see that, for 3D applications, CBD, which features a simple and clear structure, is more suitable for functionality extension than OOP. CBD also effectively avoids the code coupling caused by inheritance, making it ideal for visual development.

3. Mobile-friendly. As a lightweight engine, Oasis 3D enables on-demand loading and can load content for display even when the network connectivity of mobile devices is limited. Oasis 3D is also optimized for mobile browsers, providing some best practices.

After supporting services of Ant Financial at a massive scale, Oasis 3D is now compatible with thousands of mobile devices, being stable and reliable.

To sum up, Oasis 3D adopts several of the modes and concepts from the original web-based 3D engine, while also making several adjustments for the mobile environment, and therefore it has several advantageous over other engines on the market.

How Oasis 3D Has Been Used

Ant Financial has been exploring the implementation of 3D projects for years. Oasis 3D has been applied in several projects for continuous iteration and optimization with excellent results, allowing us to set up an efficient 3D workflow.

When Oasis 3D was first released in 2017, at which time it was entrusted to support several projects in succession, including Ant Farm’s built-in game “star ball”, which had particle array special effects for the second anniversary of MYbank, and also Ant Credit Pay’s Double 11 special effects, and whale special effects for Ant Cash Now.

Image for post
Image for post
Ant Farm’s star ball game

In 2018, the focus of Oasis 3D was to optimize APIs, release new versions, launch the official website, and promote it within Alibaba Group. During that period, we also explored the implementation of mobile projects, such as Bonus Planet and Ant Farm’s chicken hill climbing game, which were designed in the mobile-friendly low poly style. We used a large number of baked textures to replace complex lighting and shadows. This greatly optimized the visual effects and performance, and dramatically improved the user’s overall experience.

Image for post
Image for post
Bonus Planet
Image for post
Image for post
Ant Farm’s chicken hill climbing game

We also tried our hand at the big screen visualization project and successfully undertook the task of supporting the business big screen.

Image for post
Image for post

Alipay big screen

Oasis Workflow

Although Oasis 3D has made major contributions to the launch of 3D projects, its development has not been easy. In fact, it helped expose a series of problems in web-based 3D project creation:

  • Loads of customized development. This is particularly true for mobile projects, in which the rendering capability is greatly limited by the computing power of the device. To implement smooth operation without compromising on effects, developers usually take shortcuts by using customized solutions. This requires users to have a deep understanding of the principles of the Oasis 3D engine and customize the development when needed. As a result, the overall development is difficult and inefficient.
  • Difficulty with getting started. Front-end engineers are not familiar with computer graphics technology and game development workflows. Therefore, only a few engineers can handle 3D projects.
  • A lack of relevant references and clear innovation. Most product designers are not experienced at 3D design. They either are unsure about how to complete 3D design projects or make unattainable designs. Designers need to understand the characteristics of the actual running environment for model design, and it is best to directly extend according to the effects supported by the current environment. At present, there are few successful web-based 3D cases for reference, making the implementation of 3D projects even more difficult.
  • Cumbersome workflow. It is cumbersome to convert art models to high-quality and operational resources. In this case, Unity is additionally required for performing conversion after configuration conversion, making the workflow cumbersome and inefficient.

These challenges can hamper the implementation of web-based 3D projects, making it difficult to promote and reuse this technology in the Ant Financial unit.

For the solution, we must not only solve the problems at the engine level, but also simplify the implementation of web-based 3D projects and its accessibility to more users. This forces us to simultaneously work on three aspects: engine functionality, tools and workflow, and platform ecosystem.

Therefore, we develop Oasis Editor, a 3D content editor based on Oasis 3D.

Oasis Editor is an online editor that redefines the workflow of 3D projects. With the workflow provided by Oasis Editor, you can visually edit 3D environments and interaction logic. Functions such as scene setup, scene beautification, animation editing, and game logic editing can all be completed online.

Image for post
Image for post

Oasis Editor

Oasis Editor has the following advantages:

  • A rich set of functional components. Oasis Editor supports various types of resources. You can use the custom extension function to extend customized requirements into new functional components.
  • Seamless transformation of art assets. Oasis Editor automatically converts and uploads most 3D resources, saving the Unity conversion work.
Image for post
Image for post

Comparison of different art assets pipelines

  • Visual editing of various animations. Oasis Editor provides engineers or designers with various animation editing functions, such as interpolation animation editing and particle animation editing. This allows us to get rid of the original workflow of repeatedly adjusting parameters in code and then pass on these changes to the designer for review. Designers now can directly use the editor to adjust parameters and participate in project development, leaving the professional tasks to the professionals.
  • Online platform ecosystem. Oasis Editor integrates with the 3D content platform to provide the project iteration management and resource sharing capabilities.

In addition, we tried to cover all aspects of 3D development with Oasis Editor. For example, in computer graphics technology, the most specialized part is the use of various graphics algorithms and programmable shader interfaces to implement various special effects and simulation effects. It is very difficult to involve application developers in this part. However, many professional shaders can be involved in multiple places, and the visual shader editor and the shader content management platform are also very important. In the future, we plan to integrate these functions with Oasis Editor engineering so that we can not only demonstrate the effects of our supported shaders in a holistic way, but also reduce a lot of the workload of custom development.

In general, we hope to use Oasis Editor to simplify the 3D development process, making developers transparent to the Oasis 3D engine. In this way, they can develop the entire project simply by going through the Oasis workflow, and can enjoy the ease of 3D development.

Summary

To achieve better versatility, browsers provide users with the most basic rendering interfaces. With them, users can freely control the display of each source point on the canvas. However, with freedom also comes problems. Your imagination and graphics knowledge system determine what you can render, and most people don’t have such wild imaginations. As a result, it can be extremely hard to implement web-based 3D projects. It’s sort of like trying to build skyscraper with only some gravel.

Like building a skyscraper, web-based 3D engineering needs to consider more advanced issues. The Oasis system undertakes the underlying professional tasks and provides users with efficient and easy-to-use tools and a rich platform.

We have been working hard on building the Oasis system, which will soon be available to users in China to serve all web-based 3D project developers. So, stay tuned.

Original Source

Follow me to keep abreast with the latest technology news, industry insights, and developer trends.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store