LinkedIn built and open-sourced Apache Kafka to handle its internal streams of activity data, later followed by the related stream-processing framework Samza.
What we cover in the quiz
- LinkedIn engineers created Kafka to handle the company's internal streams of activity data at high volume, then open-sourced it; it has since become standard infrastructure well beyond LinkedIn.
- LinkedIn built Pinot to power user-facing analytics such as "who viewed your profile", which required low-latency queries over very large, constantly updating datasets.
- LinkedIn built Samza to process the continuous streams of data flowing through Kafka, giving it a way to build features like real-time analytics on top of that data, before donating the project to Apache.