OAM and Crossplane: The Next Stage for Building Modern Application
In Mar 2020, with collaboration and significant contributions from the Crossplane community, the Open Application Model (OAM) project released its v1alpha2 spec which aims at bringing great extensibility to both the model itself and any Kubernetes application platform adopting it. In May 2020, with the latest v0.11 release, Crossplane now becomes the standard implementation of the OAM specification. We are very excited to see the collaboration between these two communities which now brings standard application and infrastructure definitions with an implementation to the cloud native community.
The start of the journey
From the perspective of Kubernetes engineers like us, we are happy with the current level of Kubernetes’ abstraction: containers and infrastructure API resources. But they are still too low level for the end users of the platform.
To somewhat improve the end user experience, many teams tried to hide the Kubernetes API from the end user by introducing a “PaaS” or GUI. At first glance, it seems like a good idea. But in reality, this greatly limits the capability of the platform. The Kubernetes Resource Model emphasizes that every capability in the system should be represented as “data”, i.e. API object. Hiding these objects from end users essentially makes your PaaS lack of extensibility as they can’t leverage the countless plug-in capabilities in the ecosystem.
With the idea that we should enable platform builders to define application level abstractions without introducing limitations to platform extensibility, we began to explore in this field.
Model your application, not just describe
As we want to define an application level abstraction, the first question is: what is an application?
A modern application is mostly composed of multiple pieces (see the diagram above). This pattern widely exists in the real world: multi-tier application, machine learning training application (parameter server and worker), and not to mention the microservice architecture. But what is often forgotten is that every component of this application also needs to be bound to a list of operational strategies. In addition, grouping strategy is also a special kind of operational strategy. For example, we need to set up the security group for multiple components in one group.
So the most straightforward approach is to use a CRD as a higher level abstraction to describe your application. And it will combine every other piece (operational strategy, infrastructure etc) the application need to run into a mega YAML file as below:
This sample above is exactly the 1.0 version of “Application” in Alibaba. As you can imagine, developers complain this “Application” is way too complex though it was designed to make their lives simpler. Also, we found that it is very chaotic to maintain this object and it’s basically impossible to extend. It got worse as more and more capabilities are installed in our Kubernetes cluster and need to be added to this object — the Kubernetes community is growing too fast!
Actually, if you check the YAML file above carefully, what developers really care about is that small piece of “commands” and “package” section to run their application.
So why don’t we split this YAML into multiple pieces? The developer only defines their own piece as “what to run”, the operator (or the system) defines operational strategies, and infrastructure operators to handle the infrastructure pieces.
After approaching various companies in the community, we found this idea of “separation of concerns” aligns very well with Microsoft teams. With several weeks collaboration with Microsoft, we defined a high level draft as below:
See? Instead of an all-in-one CRD mixing everything together, the core idea of OAM is essentially a “frame”, so that the developers and operators can fill in their own pieces of data in the “blanks” to form the whole application. This flexibility ensures that any platform that adopts this definition will not be restricted to certain workload and capability types, and the system can support any workload (container, function, or even VM) with operational capabilities (autoscaling, ingress, security policy, etc).
We call this approach the “application model”, because when a user wants to compose several pieces into an application following this spec, they need to think about which blank that piece fills in, e.g. does it describe “what to run”? Or is it an operational strategy? This process is very similar to mathematical modeling which uses mathematical concepts and language to describe systems. We are now using OAM concepts to describe pieces of an application. The benefit is that the platform now understands the category of these pieces, so it can guarantee the topology of the pieces, or check compatibility of its operational strategies — this discoverability and manageability is essential for a production application platform.
We finally published this idea as OAM spec v1alpha1.
Crossplane + OAM: build the modern application model for Kubernetes
OAM v1alpha1 spec got quick adoption in the Alibaba Cloud Enterprise Distributed Application Service (EDAS) as well as platforms internally. However, we also identified an issue in the “what to run” piece (previously called ComponentSchematic), where you need to publish a new version of ComponentSchematic for any change in the YAML. This is because it is designed as a schematic object, so developers can define what type of workload they want to deploy and share it with others. A similar issue also exists in the operational strategy part (namly “traits”) — its schema has to be leaked to end users as well.
During the following KubeCon NA in Dec, we met with Crossplane maintainers from Upbound.io. We talked about OAM, and how OAM spec can be integrated seamlessly with Crossplane by leveraging CRDs as schemas. We both thought this direction is promising and after months of brainstorm sessions, proposals, and numerous sometimes spirited discussions, this idea eventually evolves into the OAM spec v1alpha2 as below:
OAM spec v1alpha2 adopted the Kubernetes Resource Model, so any piece of data in Kubernetes can now be seamlessly referenced as workload or trait in OAM by simply defining a WorkloadDefinition or TraitDefinition. A deep dive blog about OAM v1alpha2 will be published soon, here's also a detailed slide for you to check.
On the implementation front, we developed a brand new Go based implementation, called the oam-kubernetes-runtime, as part of crossplane. Now we have a standard Kubernetes runtime for OAM.
Composition: completing the whole picture
As you may have noticed, there’s still a missing part with OAM: how can I define infrastructure pieces required by the component, for example, a MySQL database instance (RDS) from Alibaba Cloud? How can I make this definition portable between clouds just like the OAM component?
Defining such application centric and portable infrastructure is never easy in Kubernetes, there are some operators or products to do this in the wild, but nothing is as neat as Composition in Crossplane. Composition allows you to combine and then publish multiple infrastructure pieces into a platform-agnostic CRD, for example, compose CRDs to describe VPC and RDS into a new Database CRD. This CRD, can then be referenced as OAM WorkloadDefinition and become part of the application. Done!
The combined result is a powerful, team-centric platform that enables infrastructure operators to define and compose vendor-neutral infrastructure for applications, and enables application developers and application operators to define, run, and manage portable applications with OAM without having to deal with the complexities of infrastructure. Infrastructure operators now have a way to manage the infrastructure on which those applications run. OAM and Crossplane together provides an elegant solution to both application developers and infrastructure operators.
What’s next?
The core idea of OAM is that developers describe their apps, and then they can take that app and run it on a serverless platform, or a Kubernetes cluster that might be on-prem without modifying that description of the application. It is the part of cloud, edge consistency story that Alibaba and Microsoft have been working on. Obviously, the collaboration with Crossplane is the missing piece to make this story really come together by unifying standard application definition and infrastructure definition in the same system. We will continue to work together to make Crossplane the standard Kubernetes implementation for OAM with better workload/trait portability, interoperability, abundant operational capabilities, and build an open community focused on standard application and infrastructure.
