Intelligently Generate Frontend Code from Design Files: imgcook

Background Analysis

Competitive Product Analysis

Problem Resolution

View Code

  • Reasonable Layout: Include absolute position relative position, redundant node deletion, reasonable grouping, loop judgment, etc.
  • Element Self-adaption: Extensibility of the element itself, alignment between elements, maximum width, and high fault tolerance of elements.
  • Semantic: Multi-level semantics of class names.
  • CSS expression: The background color, rounded corners, lines, etc.
  • The industry has been working in this direction for a long term. The basic information of elements in a design document can be exported through the design tool’s plugin. But the problem remains in the aspect of the high requirement for the design document and poor maintainability of generated code.

Logical Code

  • Data field binding: This is quite feasible. You can determine the candidate field based on the design document’s text or image. But the cost-performance ratio is not high because this is more about business logic, which is not general logic.
  • Dynamic effect: The input of this part is a design document. Generally, the delivery forms of dynamic effect are various. Some of which are animated gif demonstrations, while some are text description or even oral. The generation of dynamic code is more suitable for visual generation. There is no reference for direct, intelligent generation, considering that the input-output ratio is not a short-term problem.
  • Business logic: This part of the development is mainly based on PRD, and even the product manager’s logic. If you want to generate this part of logic code intelligently, there is too much input. Specifically, we need to see what problems intelligentization can solve in this sub-field.

Logical Code Generation

  • Analyzing and guessing the semantics of high-frequency function blocks (logical blocks) based on historical source code. In this way, we can recommend code blocks while editing code.
  • We can guess some of the reusable logical points from the design draft. For instance, to bind the image or text data to view, we can use NLP classification or image classification to recognize the elements’ contents.
  • Field binding: Use deep learning to intelligently identify the semantic classification of text and images in the design draft, especially the text part.
  • Reusable business logic points: It is intelligently identified based on views. It contains small logic points (one line of expression, or several lines of code that are generally insufficient to be encapsulated into components), basic components, and business components.
  • New business logic that cannot be reused: Structured (visualized) collection of PRD requirements is a difficult task and is still being tried.

Summary

Technical Solution

  • Recognition capability: The ability to identify the design document. This is to intelligently analyze multiple dimensions of information from the design document, including layers, basic components, business components, layouts, semantics, data fields, and business logic. If the intelligent recognition is not accurate, then human intervention is used to correct errors. On the one hand, the high-availability code is generated from low-cost intervention. On the other hand, the artificial corrections can also be used as samples for online training.
  • Expression ability: Mainly output the data and access the engineering part.
  • Use DSL to make the standard structured description of Schema2Code.
  • Perform Project Access through IDE plugins.
  • Algorithm engineering: To better support the intelligentization D2C requires, high-frequency capabilities are served, mainly including data generation, processing, and model services.
  • Sample generation: Mainly process each channel’s sample data and generate samples.

Intelligent Identification Layer

  • Material identification layer: To identify the materials in the image through image recognition, including module recognition, atomic module recognition, basic component identification, and business component identification.
  • Layer processing layer: Mainly separate the layers in the design document or image, and combine the previous layer’s recognition results to sort out the layer meta information.
  • Layer reprocessing layer: Further normalize the data from previous layers.
  • Layout algorithm layer: Convert the absolute position to layout relative position and Flex layout.
  • Semantic layer: The layer’s multi-dimensional features are used to make semantic expressions on the generated code.
  • Field binding layer: Bind and map the static data in the layer with the actual backend data.
  • Business logic layer: Generates the business logic codes through the business logic identification and expresser.
  • Output engine layer: Finally, output the code intelligently processed by each layer’s various DSL.

Technical Difficulties

Problem Definition

  • Step 1: Find out as many relevant design documents as possible. Enumerate the types of images.
  • Step 2: Reasonably summarize and classify the types of pictures, which is the easiest place to be controversial. Bad definition and ambiguity will lead to the model’s problem.
  • Step 3: Analyze the features of each type of picture — whether these features are typical or not and whether they are the core feature points — because they are related to the inference generalization ability of subsequent models.
  • Step 4: Whether the data sample source of each type of image is available or not, and if not, whether it can be automatically created or not. If the data sample cannot be available, it is not suitable to use the model. And you can replace the hard rules to see the effect first.

Sample Quality

Data sample engineering system

Model

D2C scenario

Thoughts of the Process

  • At the beginning, we can reach the agreement of the loop manually in the design document.
  • Based on the layer’s context information, you can make some rule judgments to determine whether it is a loop body.
  • Using the layer features of machine learning, you can try to optimize rules.
  • Generate some positive and negative samples of the loop to learn through the deep learning model.

Business Landing: 2019 Double 11

D2C code generation user changes

Overall Landing Situation

  • The number of modules is 12,681, and about 540 are newly added this week
  • The number of users is 4,315, and about 150 new users are newly added every week
  • Number of teams: 24
  • Custom DSLs: 109

Follow-up Planning

  • Continue to reduce the design documents’ requirements, in which the intelligent identification accuracy of grouping and loop are improved, and the manual intervention cost of design document is reduced.
  • The component identification accuracy has been improved. Currently, the accuracy is only 72%, and the business application availability is low.
  • The page-level and project-level restoration capabilities are improved, depending on the accuracy of page segmentation capabilities.
  • Improve page-level restoration of mini-apps and PC programs, and improve overall restoration of complex forms, tables, and charts.
  • Improve the ability to generate code from static images, which can be used in the production environment.
  • Algorithm engineering products are improved, and sample generation channels are more diversified.
  • Open source.

Contact us

Original Source:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com