Browse our archives by topic…
Talking about Azure Synapse on Microsoft Mechanics!
I was recently invited on to Microsoft Mechanics to talk about the new on-demand SQL Serverless offering within Azure Synapse. If you have been following along with my previous blog posts you will know that we've been hard at work applying Azure Synapse against real customer workloads. In the video I take you through the service by solving a real-world IoT problem for one of our telco customers.
Benchmarking Azure Synapse Analytics - SQL Serverless, using .NET Interactive
There is a new service in town that promises to transform the way you query the contents of your data lake. Azure Synapse Analytics comes with a new offering called SQL Serverless allowing you to query your data on-demand with no need for pre-provisioned resources.When we heard about the new service we were keen to get involved, so for the last 10 months we've been working with the SQL Serverless product group to provide feedback on the service and to help ensure it meets our customers needs. During this time we've put it through it's paces by implementing a range of real-world use cases. We were particularly interested to see how it stacked up as a replacement for Data Lake Analytics, where to date there has been no clear and easy migration path.
Using multiple azure-cli credentials within automation
Have you ever needed an automated process to use alternative credentials for a subset of tasks? This post will demonstrate a technique that allows you setup multiple, concurrent authenticated sessions when using the azure-cli and switch freely between them.
Does Azure Synapse Analytics spell the end for Azure Databricks?
Have you or are you about to invest in Azure Databricks? If so, the new Spark offering in Azure Synapse Analytics is likely to have grabbed your attention and rightly so. Why is Microsoft putting yet another Spark offering on the table and what does it mean for you?
Why Power BI developers should care about the new read/write XMLA endpoint
Whilst "read/write XMLA endpoint" might seem like a technical mouthful, its addition to Power BI is a significant milestone in the strategy of bringing Power BI and Analysis Services closer together. As well as closing the gap between IT-managed workloads and self-service BI, it presents a number of new opportunities for Power BI developers in terms of tooling, process and integrations. This post highlights some of the key advantages of this new capability and what they mean for the Power BI developer.
How does Azure Key Vault help me secure my data?
Azure Key Vault is used to protect encryption keys and secrets. These keys and secrets can be used to access encrypted data and protected services. Individual Key Vaults can be used to preserve security information for isolating keys and secrets. The keys stored can be either hardware or software protected. Access to the keys and secrets is controlled using Azure Active Directory, RBAC and access policies.
Testing Power BI reports using SpecFlow and .NET
Despite being inherently difficult to test, the need to validate data modelling, business rules and security boundaries in Power BI reports is important, as well as the need for ensuring that quality doesn't regress over time as the insights evolve. This post explains that, by connecting to the underlying tabular model, it is possible to execute scenario-based specifications to add quality gates and build confidence in Power BI reports, just as any other software project.
Recording of Azure Oxford talk on combatting illegal fishing with Azure (for less than £10/month)
Jess and Carmel recently gave a talk at Azure Oxford on "Combatting illegal fishing with Machine Learning and Azure - for less than £10 / month). The recording of that talk is now available for viewing!The talk focuses on the recent work we completed with OceanMind. They run through how to construct a cloud-first architecture based on serverless and data analytics technologies and explore the important principles and challenges in designing this kind of solution. Finally, we see how the architecture we designed through this process not only provides all the benefits of the cloud (reliability, scalability, security), but because of the pay-as-you-go compute model, has a compute cost that we could barely believe!
Testing Power BI Dataflows using SpecFlow and the Common Data Model
Whilst testing Power BI Dataflows isn't something that many people think about, it's critical that business rules and associated data preparation steps are validated to ensure the right insights are available to the right people across the organisation. Data insights are useless, even dangerous, if they can't be trusted, so despite the lack of "official support" or recommended approaches from Microsoft, endjin treat Power BI solutions just as any other software project with respect to testing - building automated quality gates into the end to end development process. This post outlines an approach that endjin has used to test Power BI Dataflows to add quality gates and build confidence in large and complex Power BI solutions.
Azure Analysis Services - how to save money with automatic shutdown
Azure Analysis Services provides an enterprise-grade analytical platform with massive scale and flexibility. But, as one of the more expensive services in the Azure platform, consideration should be given to cost management, especially in multi-environment ALM scenarios. This post explains how to massively reduce running costs through automation using Powershell and orchestration tools like Azure DevOps.
Building a proximity detection pipeline
At endjin, our approach focuses on using scientific experimental method to support the creation of fully proved and tested decision making, and the use of scientific research to support our work. This post runs through how we applied that process to creation a pipeline to detect vessel proximity.This is an example which is based around the project we recently worked on with OceanMind. In this project we helped them to build a #serverless architecture which could detect vessel proximity in close to real time. The vessel proximity events we detected were then fed into machine learning algorithms in order to detect illegal fishing!Carmel also runs through some of the actual calculations we used to detect proximity, how we used #data projections to efficiently process large quantitities of incoming data, and the use of #durablefunctions to orchestrate the processing.
Azure Analysis Services: How to update the expression for a calculated column from .NET
This post explains how to update Azure Analysis Services model schemas from inside custom .NET applications. Whilst not a common scenario for most, it shows that this is easy to do using the AMO SDK. So, there's nothing stopping you from developing complex and rich end-user functionality over the top of your data analysis solutions - providing run-time, user-driven schema changes like "what if" analysis.
Azure Analysis Services - How to process an asynchronous model refresh from .NET
Integrating Azure Analysis Services into custom applications doesn't just mean read-only data querying. But if your application changes the underlying model, it will need to be re-processed before the changes take effect. This post describes how to use the REST API for Azure Analysis Services inside a custom .NET application to perform asynchronous model refreshes, meaning your applications can reliably and efficiently deal with model updates.
Introducing Ais.Net - High-Performance Parsing in C#
As part of our work with OceanMind, endjin wrote a high performance .NET AIS parser. AIS (Automatic Identification System) is how commercial ships report location information. This blog describes the parser, and the performance techniques it uses.
Azure Analysis Services: How to execute a DAX query from .NET
Being able to construct DAX queries dynamically in C# means the possibilities are endless in terms of integrating Azure Analysis Services queries into your custom applications, and with the code samples in this post, you have everything you need to get started.
British Science Week - inspiring the next generation of data scientists
The theme of this year's British Science Week (6 - 15 March 2020) is "Our Diverse Planet". We'll be getting involved by speaking to school children about the work we've been doing with Oxfordshire-based OceanMind (part of the Microsoft AI for Good programme) to help them combat illegal fishing, hopefully inspiring some of the next generation of data scientists!
How can I contribute to the Azure CNAB Quickstarts Library?
The Azure CNAB Quickstart Templates we've created are only half the story. Much of the work we've done over the last few months involved the authoring, contribution and DevOps pipelines required to support an open source project. The project is inspired by the original Azure Quickstart Templates - which over the last 5 years has grown to over 850 templates. In this post we're going to explain how you can author CNAB templates and contribute them.
Setting up Porter on Windows
Porter is a tool based on the CNAB (Cloud Native Application Bundle) spec. It can be used for building, managing, and installing application bundles. This guide will walk you through how to get set up with Porter on Windows.
Introducing the Azure CNAB Quickstarts Library
In partnership with Microsoft, we have released Azure CNAB Quickstarts Library on GitHub. CNAB (Cloud-Native Application Bundle) is a new specification designed for facilitating the packaging, installation, upgrading and uninstallation of cloud-native solutions in the cloud, on-premise or on the edge. We've created a number of quickstarts covering Apache Airflow, Azure Kubernetes Service, Ghost, Kubeflow, SQL Server Always On and Wordpress to help demonstrate the power of CNAB and Porter.
Exposing legacy batch processing code online using Azure Durable Functions, API Management and Kubernetes
In this post we show how a combination of Kubernetes, Azure Durable Functions and Azure API Management can be used to make legacy batch processing code available as a RESTful API. This is a great example of how serverless technologies can be used to expose legacy software to the public internet in a controlled way, allowing you to reap some of the benefits of a cloud first approach without fully rewriting and migrating existing software.
Azure Analysis Services - How to query all the measures in a model from .NET
Integrating Azure Analysis Services into custom applications means more than just querying the data. By surfacing the metadata in your models, you can build dynamic and customisable UIs and APIs, tailored to the needs of the client application. This post explains how easy it is to query model metadata from .NET, so you can create deeper integrations between your data insights and your custom applications.
Azure Analysis Services: How to open a connection from .NET
One of the first steps in integrating Azure Analysis Services into your applications is creating and opening a connection to the server - just like any other database technology. This post explains the ins and outs of creating Azure Analysis Services connections, including code samples for each of the key scenarios.
NDC London 2020 - My highlights
Ed attended NDC London 2020, along with many of his endjin colleagues. In this post he summarises and reflects upon his favourite sessions of the conference including; "OWASP Top Ten proactive controls" by Jim Manico, "There's an Impostor in this room!" by Angharad Edwards, "How to code music?" by Laura Silvanavičiūtė, "ML and the IoT: Living on the Edge" by Brandon Satrom, "Common API Security Pitfalls" by Philippe De Ryck, and "Combatting illegal fishing with Machine Learning and Azure - for less than £10 / month" by Jess Panni & Carmel Eve.
NDC London Day 3 Retrospective - from personal projects to developer comedy
Along with several of my endjin colleagues, I attended NDC London in January this year - here's a run through of the sessions I attended on Day 3 and my thoughts. This final day was a mixed bag, taking in talks on drumming and AKKA.net, as well as something a bit more close to home - a session from endjin's own Jess Panni and Carmel Eve on our recent project for OceanMind.
NDC London Day 2 Retrospective - Full Stack, from SPAs to high performance .NET Core via Websockets
Along with several of my endjin colleagues, I attended NDC London in January this year - here's a run through of the sessions I attended on Day 2 and my thoughts. This day was UI heavy, with sessions on Vuejs and UI testing, but I also learned more about GraphQL and writing high performance C# code.
NDC London Day 1 Retrospective - dangerous AI, microservices, diagnostics and Blazor
A retrospective of my first day at NDC London 2020, taking in sessions on AI and Machine Learning, Capability Mapping, Micro Frontends, Diagnostics and Blazor.
Azure Analysis Services - integration options using .NET, REST APIs and PowerShell
With a variety of integration support through client SDKs, PowerShell cmdlets and REST APIs, it can be hard to know where to start with integrating Azure Analysis Services into your custom applications. This posts walks through the options, and lays out a simple guide to choosing the right framework.
Azure Analysis Services: 8 reasons why you might want to integrate into a custom application
We've done a lot of work at endjin with Azure Analysis Services over the last couple of years - but none of it has been what you'd call "traditional BI". We've pulled, twisted and bent it in all sorts of directions, using it's raw analytical processing power to underpin bespoke analysis products and processes. This post explains some of the common (and not-so-common) reasons why you might want to do similar things, and how Azure Analysis Services might be the key to unlocking your data insights.
AI for Good Hackathon
Towards the end of last year, Microsoft invited endjin along to a hackathon session they hosted at the IET in London as part of their AI for Good initiative. I've been thinking about the event and the broader work Microsoft is doing here a lot lately, because it gets to the heart of what I love about working in this industry: computers can magnify our power to do to good.
Building a secure data solution using Azure Data Lake Store (Gen2)
In this blog from the Azure Advent Calendar 2019 we discuss building a secure data solution using Azure Data Lake. Data Lake has many features which enable fine grained security and data separation. It is also built on Azure Storage which enables us to take advantage of all of those features and means that ADLS is still a cost effective storage option!This post runs through some of the great features of ADLS and runs through an example of how we build our solutions using this technology!
Speaking at NDC London: Combatting illegal fishing with Machine Learning and Azure
In January 2020, Carmel is speaking about creating high performance geospatial algorithms in C# which can detect suspicious vessel activity, which is used to help alert law enforcement to illegal fishing. The input data is fed from Azure Data Lake Storage Gen 2, and converted into data projections optimised for high-performance computation. This code is then hosted in Azure Functions for cheap, consumption based processing.
Troubleshooting "NotImplemented / Access is denied" error when integrating Azure App Service with Azure Virtual Network (VNet)
Are you configuring your Azure App Service to use a VNet? The regional VNet integration for an Azure App Service is a preview feature, and so comes with some quirks. One of the documented limitations of this preview feature is that "The feature is only available from newer App Service scale units that support PremiumV2 App Service plans." This has interesting implications as to how you need to deploy your App Service, otherwise you could end up with a rather perplexing pattern of errors. Read about how we came about this error, and how we ended up fixing it.
Comparing GitHub Actions and Azure Pipelines: High-level concepts
GitHub Actions is GitHub's new CI/CD platform. It is comparable with Azure Pipelines, which forms part of the Azure DevOps suite. In this post, Mike Larah looks at the similarities and differences in the high-level concepts and terminology between the two platforms.
Long Running Functions in Azure Data Factory
While on first inspection Azure Function look like a good candidate for long running operations as they can run for 10 minutes on a Consumption plan, HTTP Triggers only run for 230 seconds because of a limitation with the Azure Load Balancer. In this blog post Jess Panni demonstrates how Durable Functions can be used instead.
How Azure DevTestLabs is helping me climb Everest
Remote working allows us to work from anywhere we want. This brings a huge amount of flexibility in freedom, however we do need the help of a working laptop! When Carmel's laptop gave in just before a trip, she used Azure DevTestLabs to allow her to continue to work using a 10 year old Mac that probably couldn't wouldn't have been up to the task alone...
Running Azure functions in Docker on a Raspberry Pi 4
For one of my first experiments with the Raspberry Pi 4, I decided to get an Azure Function running in a Docker container. This post gives a step-by-step guide on how to do it, as well as providing code you can use a starting point for your own experiments.
Secure function-to-function authentication in Azure without the need for credentials
Building a secure solution on Azure can be a daunting task. Using Azure Functions and Managed Identities, we have built up a pattern for giving services access to one another, woithout the need to store credentials. These managed identities can be given access to necessary resources. For example, they can be granted roles and added to access control lists in ADLS Gen2 accounts, or the ability to access keys in key vault. This means that data can be securely accessed without needing to store connection strings or app passwords.
How to use the AzureAD module in PowerShell Core
Whilst some of the Azure Active Directory PowerShell for Graph module (AzureAD) functionality has been rolled into the new Azure PowerShell Az module, it's not currently (and might never be) a replacement for the full power of what you can achieve with AzureAD. So, there's every chance you'll find yourself needing to use both side-by-side. This post explains how to do that using the new cross-platform PowerShell Core.
How to create a Power BI workspace in an Azure DevOps Pipeline using Powershell
A Power BI based solution typically consists of a variety of technologies - for example Azure data platform services containing source data. As such, automation of Power BI resources needs to be considered as part of a wider DevOps strategy. This post describes the specific steps needed in order to fully automate the creation and security of Power BI workspaces using Powershell and Azure DevOps pipelines.
Using Databricks Notebooks to run an ETL process
Here at endjin we've done a lot of work around data analysis and ETL. As part of this we have done some work with Databricks Notebooks on Microsoft Azure. Notebooks can be used for complex and powerful data analysis using Spark. Spark is a "unified analytics engine for big data and machine learning". It allows you to run data analysis workloads, and can be accessed via many APIs. This means that you can build up data processes and models using a language you feel comfortable with. They can also be run as an activity in a ADF pipeline, and combined with Mapping Data Flows to build up a complex ETL process which can be run via ADF.
Exploring Azure Data Factory - Mapping Data Flows
Mapping Data Flows are a relatively new feature of ADF. They allow you to visually build up complex data transformation sequences. This can aid in the streamlining of data manipulation and ETL processes, without the need to write any code! This post gives a brief introduction to the technology, and what this could enable!
Snowflake Connector for Azure Data Factory - Part 2
Enforce resource tagging with Azure Policy
This blog post details how we used Azure Policy to enforce Azure resources were tagged with appropiate tags and ensured tags were inherited from parent resource groups where possible.
Avoiding deployment locking errors by running Web and Functions Apps from packages
This post walks through the fix for DLL locking errors when trying to deploy an Azure Function. The solution was to switch over to the new "deploy from package" option when deploying the functions. This fixes the file locking problem because instead of deploying the DLLs, the function will run from a package file added to its directory.
How to run a script on an existing Azure Virtual Machine
Using PowerShell and the VM Custom Script Extension to download and execute scripts on existing Azure VMs
Managing applications using Azure AD, service principals and managed identities: A permissions story
The complexities around Azure Active Directory can be difficult to understand. This post runs through some of the key concepts - AAD apps, service principles, managed identities, and walks through an example of how to set some of this up!
Cosmos DB - Request Units charged for processing a Gremlin API request
Thoughts about .NET, The Cloud, AI, ML, and teaching software engineers
OpenAPI Document Converters for Visual Studio 2017
How we set up daily Azure spending alerts and saved $10k
Kickstart your API proposition with the API Maturity Matrix
How to plan your cloud transformation journey
Azure Tech Selector - 2017 Edition
Best of the Recorded Azure Sessions at BUILD 2017
Benchmarking the Cloud against on-premise data centres
Advanced Azure Resource Manager template patterns – t-shirt sizing and optional resources
One very useful but little used pattern when working with Resource Manager templates, is the ability to use parameters to optionally deploy resources, constrain certain resource configurations based on other user defined parameters, or to toggle parameters based on other values. This blog post explores some of these options.
Automating office security with Synology, Surveillance Station, OneDrive and Power Automate
Read about how we set up automated backups of our office security camera footage and used Power Automate to alert us if anything went wrong
AWS vs Azure vs Google Cloud Platform - Networking
How to use Power BI Embedded with AspNetCore
AWS vs Azure vs Google Cloud Platform - Mobile Services
AWS vs Azure vs Google Cloud Platform - Internet of Things
AWS vs Azure vs Google Cloud Platform - Analytics & Big Data
AWS vs Azure vs Google Cloud Platform - Database
AWS vs Azure vs Google Cloud Platform - Storage & Content Delivery
AWS vs Azure vs Google Cloud Platform - Compute
AWS vs Azure vs Google Cloud Platform
Cloud Adoption: A Deep Dive into the Swiss Cheese Model
Automating creation of new ALM environments using PowerShell and Azure DevOps
Did you know that you can fully automate the bootstrapping process of setting up an Azure DevOps enviornment? This post shows you how.
"But it works on my cloud!" - are your developers still making the same mistakes in a world of DevOps and PaaS services?
In the world of DevOps, cloud and platform services, how does a developer's "definition of done" need to change? This post argues that as the silos of development and operations are broken down, the responsibility of understanding the whole solution increases meaning, to truly take advantage of the cloud, the need for quality and professionalism is critical for success.
Deploying to Azure using Azure Resource Manager templates and Octopus Deploy
Learn how to combine the power of Azure Resource Manager and Octopus Deploy for a frictionless Azure DevOps experience.
Cloud Adoption: Risks & Mitigations Analysis
Using Postman to load test an Azure Machine Learning web service
Azure Machine Learning Studio is a fantastic service for experimentation, but you can also easily productionize your machine learning models. In this post we show how to create an Azure ML Studio web service and test it using Postman.
TeamCity MetaRunner for creating Release Annotations in Azure Application Insights
Using Azure Automation to run VMs during office hours only – using graphical runbooks
We've improved our approach for saving money by turning off our virtual machines outside of office hours. This post explored how to do it using graphical runbooks.
Embracing Disruption - Financial Services and the Microsoft Cloud
We have produced an insightful booklet called "Embracing Disruption - Financial Services and the Microsoft Cloud" which examines the challenges and opportunities for the Financial Service Industry in the UK, through the lens of Microsoft Azure, Security, Privacy & Data Sovereignty, Data Ingestion, Transformation & Enrichment, Big Compute, Big Data, Insights & Visualisation, Infrastructure, Ops & Support, and the API Economy.
The 100 Year Start-up: Embracing Disruption in Financial Services
Hymans Robertson was set up in Glasgow in 1921 and is one of the longest established independent firms of consultants and actuaries in the UK. Hymans Robertson soon realised that the computational requirements of their models exceeded the capacity of their on-premise datacentres and that the most cost effective solution would be to use the cloud to perform their Big Compute. But before they could harness the cloud to help them solve their Big Data problems, the business needed to understand the ramifications of moving to the cloud; everything from regulatory, risk and compliance concerns, to how their internal Ops team would need to evolve and adapt, and how to deal with moving data from on-prem into the cloud.
Why is blockchain revolutionising Financial Services?
There is a lot of hype about the blockchain - usually wrapped up with talk about Bitcoin and crypto-currencies. In this article, we look at its impact on trust, and auditability in financial services, and why it may (or may not) be appropriate for your solutions.
Regulatory Compliance and Cloud Adoption
In this post we review the FCA's guidelines for the adoption of cloud services by FinTech businesses, and help you to understand their impact across the value chain.
FinTech Week and the Microsoft Cloud
Azure Resource Manager authentication from a PowerShell script
Learn how to authenticate against Azure Resource Manager using PowerShell, for fully automated deployments.
Azure Weekly turns 1 & we're celebrating by giving away 5 Windows IoT Raspberry Pi 2 starter kits
Azure Batch - Time is Money in Big Compute
Using Azure Resource Manager and PowerShell DSC to create and provision a VM
Azure Resource Manager and PowerShell Desired State Configuration are powerful tools for provisioning and then configuring your virutal machines in Azure. This post delves into the details.
Spinning up 16,000 A1 Virtual Machines on Azure Batch
SpecFlow Extensions for Azure Storage Emulator
Speed kills: decrease page load time and increase conversions with Azure CDN
A CDN, short for Content Delivery Network, is essentially a network of servers set-up, in various geographic locations, with the aim of speeding up the delivery of content to end-users across the Internet.
Azure Automation scheduler and daylight saving time
Everything in Azure runs in Coordinated Universal Time; learn how to deal with Daylight saving time in Azure Automation, so your jobs still run on time.
Visualise your Azure Table Storage data with Power BI
Azure Web Apps: Package Restore against a private NuGet Feed
Using a different New Relic application name per environment with Azure Cloud Services
Just because you're using the cloud, doesn't mean you don't use traditional DTAP environments. But you do need to configure your cloud services to understand them.
Reporting errors to New Relic from an Azure Worker Role
In this post we demonstrate how to use New Relic to monitor an Azure Cloud Service Worker Role, step by step.
A step by step guide to choosing the right Azure technology for your solution
To help guide the client through the selection process (and to act as an aide memoire when we come to try and remember why particular choices were made), we created a handy flow-chart to take you through the technology landscape, and suggest some recommended-practice solutions.
Generating and using a certificate to authorise Azure Automation
Diagnostic logging with Azure SDK 2.5
Learn how to use diagnostics in Azure Virtual Machines and Cloud Service Web or Worker Roles using the Azure SDK.
Using Azure Automation to run VMs during office hours only
One of the strongest selling points of cloud computing is that you only pay for what you use. Running virtual machines can become expensive, but you can save significant money if you turn those VMs off outside of office hours.
Elasticsearch gotchas and tips - part 2
We've delivered a number of Elasticsearch solutions on Azure. In this second post in the series, we share some useful tips.
Have you signed up for the Azure Weekly newsletter?
Azure Weekly Newsletter
Troubleshooting Twilio with New Relic
We created an an automated, telephony-based candidate interviewing system using .NET and Azure. We used New Relic and Twilio to simplify first line support. In this post we delve into some of the technical details.
Restrict access to Azure Websites by whitelisting
By utilising the IP and Domain Restrictions feature in IIS, it is possible to lock down your Azure Website to only allow access to IP addresses and domains that you have specified in a whitelist. Learn how in this post.
Gotchas when installing an Elasticsearch cluster on Microsoft Azure
We've built a number of Elasticsearch clusters on Microsoft Azure. This blog post contains some of our top tips and gotchas.
Azure Tip: Cannot connect to Virtual Machine after changing Virtual Machine Size
Configuring TeamCity and YouTrack to use Office 365 for Email Notifications
Downloading Windows Azure Subscription Files
If you use Azure Management Studio, you can quickly setup your connections by importing your publish settings...