Skip to content
  1. We help small teams achieve big things.
  2. Our Technical Fellow just became a Microsoft MVP!
  3. We run the Azure Weekly newsletter.
  4. We just won 2 awards.
  5. We just published Programming C# 8.0 book.
  6. We run the Power BI Weekly newsletter.
  7. Our NDC London 2020 talk is now available online!
  8. We are school STEM ambassadors.
Alice Waddicor By Alice Waddicor Software Engineering Apprentice III
Azure data services part 2: Stream Insight

This blog is part of a series where I'm writing up my notes from a training session on Azure's data services. The previous post dealt with Azure's Hadoop implementation, HDInsight. This week, I'm going to write about Stream Insight.

What it's for:

Stream Insight is an Azure service for real-time event processing.

Use cases for Stream Insight would include real-time processing of sensor data (the archetypal example used in training sessions is a traffic management system), financial data, for example for fraud detection, server logs, clickstream data, or social media activity.

What it consists of:

The Stream Analytics service consists of:

  • Event Hubs, used to ingest data from producers (applications or devices)
  • Stream Analytics jobs, which specify the query to be used on the ingested data. Queries are written using a SQL variant (Stream Analytics Query Language) which lets you specify time-based constraints
  • A storage account used to store the monitoring data for Stream Analytics jobs

Queries are constructed using the Query Editor built into the Azure portal.

Stream-Analytics-query-editor

The Stream Analytics Query Editor.

How you get data into and out of it:

Live data is ingested through an Azure Event Hub. This can be augmented with slower-moving reference data from storage. Alternatively, blob storage can be used as a data stream.

Stream-Analytics-input-data-stream-o

The results of a Stream Analytics job can be saved to storage - Azure Blob Storage, Azure Table Storage or Azure SQL Database - or passed on to an Event Hub for further processing by the service. They can also be exported to Power BI, if you have an organisational account.

Stream-Analytics-output-sinks_thumb1

How it's charged:

The service is charged by volume of data processed by the streaming job and the number of streaming units required to process it.

Resources:

Sign up to the Azure Weekly to receive Azure related news and articles direct to your inbox or follow on Twitter:@azureweekly

Alice Waddicor

Software Engineering Apprentice III

Alice Waddicor

Alice was the second person to complete our Apprenticeship programme.

Alice came from a technical writing background, and re-trained because of an interest in technology, particularly data processing, information extraction, and automation.

During her apprenticeship Alice became passionate about Tech for Good and became an environmental campaigner.

Alice worked at endjin between 2014-2017.