Nexus Protocol, the Driving Force Behind Xianyu’s Integrated Development
By Haizhu
Serverless is an emerging concept. It can help developers reduce or do away with the setup and O&M of server devices required by the conventional backend application development. It also provides developers with the required functions by using service interfaces. Serverless expects developers to focus more on the application logic itself rather than getting “kidnapped” by trivial details of the infrastructure.
Function-as-a-service (FaaS) is a good way to implement “serverless.” Since Amazon Web Service (AWS) launched Lambda in 2014, FaaS, a backend development mode, has been quickly accepted and applied by developers. FaaS is more lightweight and event-driven.
Xianyu chose the Flutter + FaaS system to implement a cloud-client integrated development mode because Flutter and FaaS technologies are all lightweight and application-oriented. These features of the Flutter + FaaS system tally with the purpose of integration to let developers focus on the overall business logic as much as possible.
Xianyu has used the cloud-client integrated development model based on the Flutter + FaaS system for some time. Previously, my colleagues wrote several articles to introduce the evolution and implementation process of integrated development at Xianyu. All these articles mentioned words such as LogicEngine and Nexus_Framework. These have been supporting business developers in the implementation and advancement of integrated development.
In my article, I will talk about the Nexus protocol that is the driving force behind this integration, as well as the frameworks and libraries derived from this protocol.
The Origin of the Nexus Protocol
When deciding to conduct the Flutter + FaaS integrated development, we only had a vague understanding of “integration.” We only knew that client-side developers could use the Dart language to write FaaS functions. This enabled language integration. Before the emergence of the integration protocol, we only knew that what FaaS could do might be similar to what the front-end had done at the back-end for the front-end (BFF) level for a long time. At that time, we were confused about what to do with FaaS.
My colleagues at Alibaba always say,
“You don’t know what you can do because you don’t clearly understand the concepts.”
Keeping this in mind, our integration team often got together to discuss integration. We finally decided that we would first make clear of the concept of integration.
Gradually, the concept of integration became clear during our discussions. First, we defined integration, which should undergo the following phases:
- Language integration
- Development mode-architecture integration
Eventually, there will be no obvious gap between the development of Flutter pages and FaaS functions. This experience is just like developing a whole application.
Language Integration
Dart is used as the development language in Flutter, so naturally, we chose it as the development language for FaaS. Xianyu previously practiced Dart Server and therefore gathered much experience in the Dart runtime and relevant development tools. Our team modified the Dart runtime and migrated it to Gaia, the FaaS platform of Alibaba.
Accordingly, developers could use hot reload not only on the client-side for quick page debugging but also on the FaaS platform for quick deployment and debugging. This greatly improved the deployment and debugging experience.
Development Mode-Architecture Integration
Based on the language integration, we expect that developers can stay mentally consistent when developing Flutter pages and FaaS functions. That is if developers use the same data model, language stack, and logic scheduling method.
In the conventional development mode that separates the front-end from the back-end, the client-side development is quite different from the back-end development. The client-side development obtains data for page rendering and user data processing by specifying a data structure in conjunction with the back-end development. In this mode, both sides rely only on data while belonging to different systems.
In the integrated development mode, we expect developers to regard client-side pages and FaaS functions as a single system. They should be an organic integrated entity and work together to implement page functionality. In terms of responsibility, the client-side code handles user interface (UI) rendering, whereas FaaS functions deal with logic and side effects.
Developers can call the client-side functions and FaaS functions freely as if they were in the same system. This is as natural as when you call an object’s functions in one system.
However, in reality, the client-side functions and FaaS functions still belong to two individual systems. How can we call them as naturally and conveniently as calling functions in one system? In what way can they be triggered?
Event-Driven
In the development of a common client-side page, the client-side logic always revolves around three operations. No matter how much code or in what way the code is written, the logic code will eventually generate the following three effects:
- Initiate a network request (remote req).
- Call a common function (native api).
- Modify the data on a page and render this page (state change).
The page initialization process is a typical remote req => state change => render process.
Of course, this process has been simplified. The data returned from an HTTP request cannot directly act on the page state, so the data usually needs to be processed.
These actions are triggered by an obvious event, which is usually a user interaction event. For example, the actions of initiating a request, rendering a page, popping up a dialog, and jumping between pages cannot proceed spontaneously. Otherwise, it would seem a little strange.
A client-side event may be passed on to the FaaS platform to drive logical functions on this platform to process this event. When the client-side and FaaS are considered as a whole, this event moves within a single system.
Therefore, the first figure came out as follows:
Logic Normalization and Mutual Invocation
In the traditional development model, the data flow from the logic, state, to the rendering of pages is performed on the client-side. The backend is responsible for part of logic processing. Usually, this part of logic requires the invocation of various domain interfaces.
In addition, the client and the back-end will also implement the logic of converting partial domain data to the UI state. These two parts of logic are scattered at the client and the back-end and connected over a weak protocol.
After FaaS was introduced, logic can be normalized to FaaS functions. As such, the requested data can be directly used for page rendering.
If we go further, we might as well let FaaS direct the client-side UI. FaaS is like a director whereas the client-side UI is like a marionette. The UI changes by following whatever FaaS says. If all the business-related logic is moved to the FaaS platform, the client-side focuses on rendering the state to the UI. The combination of UI rendering and business logic into one page leads to deeper integration.
Here comes the second figure:
After the logic is normalized to FaaS functions, FaaS can skip the traditional weak protocol and directly connect with client-side pages. For the back-end, a request can be mapped to a specific processing function. Loosely speaking, the client is can call the back-end functions. Now that we want FaaS to control the UI changes on the client-side, we must enable FaaS to call the client-side functions.
We abstract a call as an action. For each action, a specific function provides real logic. That is, a specific action can be used to describe a specific function and the logic code behind this function, and this action itself is a function signature.
How many functions does the client-side have to provide for FaaS? After the logic is normalized to FaaS functions, we find that most implementations on the client-side focus on the following two parts:
- UI presentation
- Handling of side effects
The UI presentation is “pure” and can be described by the state data of a page. That is, in most cases, a state describes the state of the current UI. Therefore, theoretically, the client-side only needs to provide a state-to-UI mapping function so that FaaS can update the UI on the client-side. In other words, assume that a FaaS function intends to update a page, this function only needs to issue a state change action along with all the state data required by the page to achieve the desired effect. In real scenarios, the state data volume on some pages may be huge, making it difficult to directly transmit the data. Therefore, we built a Json Patch library to resolve this problem. If FaaS modifies only some data in a state, a patch is issued to synthesize a new and complete state on the client-side.
For the handling of side effects, most of the side effects come from operations such as popping up a dialog and jumping between pages. These operations are universal and have mutual properties. We also use native API actions to describe these operations. These actions and the processing functions behind them provide the capability of using native API to describe for all pages that use the Nexus protocol.
The abstraction of these two types of functions can already cover 80% of page requirements. For the remaining 20% of requirements from complex interactive pages, we provide the custom action type for developers.
Summary
The Nexus protocol emerged based on the definition of integration and the split of required functions. It is an Action-based protocol that supports mutual invocation between the client and the FaaS system.
LogicEngine — the Dispatcher of Action
We now have a protocol that supports invocation between the two systems, which means that we have a language. This language is known only by us, so we also need an interpreter to execute it.
LogicEngine is one such executor.
Let’s define it. LogicEngine is a library that is based on the Action protocol of Nexus and supports invocation between the client and FaaS.
LogicEngine itself does not provide any specific logic capabilities. All logic capabilities must be registered to LogicEngine in the form of functions and bound to specific types of actions.
Therefore, the design of LogicEngine is relatively clear:
- Externally, it provides function registration and message (action)-based execution of function calls.
- Internally, it parses messages, matches functions, and manages the execution context.
Developers use the post function to issue an action. That is, a function is called through LogicEngine, which may be in the current environment or on the FaaS platform. However, developers do not need to care about this or even the impact of this call.
The reason is that the initiator sends only one of its own intents. For example, in practice, an intent (action) is submitted when a user clicks the Buy button.
This intent will eventually produce some UI changes. FaaS will directly call the impact of the event on the client-side to the target implementation function through a state change or native API action.
The client-side does not register many functions on LogicEngine. As mentioned earlier, the logic in most of UI programming can fall into three categories. Therefore, we usually only need to register three fixed types of processing functions.
Prospects
Based on the Nexus protocol, the abstraction of three types of general processing functions, LogicEngine, and the process from the intent to effect can become transparent.
However, these are far from enough. We want to upgrade from the existing JSON protocol to a more type-safe protocol in the future. We also hope to have an IDL tool that can automatically convert action calls to interface calls, so that developers can have a better calling experience.
Moreover, we hope to change the existing one-way request-response model so that FaaS can freely call client-side functions. As such, the gap between the two systems can be closed again and become more integrated.
