Lyft’s Large-scale Flink-based Near Real-time Data Analytics Platform

1) Streaming Data Scenarios at Lyft

About Lyft

Streaming Data Scenarios at Lyft

Lyft’s Previous Data Analytics Platform and Architecture

Problems with the Previous Platform

2) Near Real-time Data Analytics Platform and Architecture

Architecture of the Near Real-time Platform

Platform Design

Platform Features and Applications

Flink-based Near Real-time Data Persistence

Multi-tier Compaction and Deduplication During ETL

3) In-depth Analysis of Platform Performance and Fault Tolerance

EventTime-driven Partition Sensing

Challenges in Schema Evolution

Deep Dive into AWS S3

Optimization Solution for Parquet

Data Backfilling-based Fault Tolerance for the Platform

4) Summary and Future Outlook

Experience and Lessons Learned

Future Outlook

Original Source:

--

--

--

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Essential Technologies for Java Developers: I/O and Netty

Designing Repeatable DAGs in Airflow (Part 2)

PIR: “Passive Infrared”, “Pyroelectric”, or “IR motion” sensor application

READ/DOWNLOAD$! Analysis and Design of Energy Systems FULL BOOK PDF & FULL AUDIOBOOK

BloomRpc Missed Gprc GUI Client

Aeroponics — Software development

Decision Making in Python

Expert talk…

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

More from Medium

Airflow on Google Cloud Composer

Streaming NFT Analytics with Apache NiFi and Apache Pulsar (FLiPN Stack)

Building a customized Spark metrics collection tool for a global enterprise technology provider

What’s the Difference Between DevOps & MLOps? — Interview With Alexandra Abbas