Fundamentals on Stream Processing: Part 03

• There are many concepts we need to aware of before we jump into the world of Stream Processing. Topics that we will be discussing
  • Time Semantics
    • Processing Time
    • Event Time
    • Watermarks
  • State and Consistency Models
  • Failures

Fundamentals on Stream Processing: Part 02

• There are many concepts we need to aware of before we jump into the world of Stream Processing.

Brief on Stream Processing

• Stream Processing engine usually perform three types of operations
  • Ingestion (Input)
    • Fetching Raw data from external source (Data Source)
  • Processing (Transformations, Aggregation, etc).
  • Sinking (Output)
    • Dumping processed and formatted data to external s̀ystem (Data Sink).
• We combine these operation into single flow called DataFlow processing graph.

Introduction: Part 01

In the era of real-time applications, stream processing has emerged as a critical component of modern software architecture. Today, nearly every product or service incorporates some form of stream processing pipeline. From analyzing live data and monitoring system performance to processing financial transactions and tracking user activities on websites, stream processing has become an essential driver of business growth.

As companies strive to make faster, data-driven decisions, stream processing plays a pivotal role in delivering real-time insights and actions. In this blog, we’ll dive into the world of Stream Processing, exploring its internal workings and understanding why it has become indispensable in today’s fast-paced, data-rich landscape.