Browse our archives by topic…
Why Power BI developers should care about the new read/write XMLA endpoint
Whilst "read/write XMLA endpoint" might seem like a technical mouthful, its addition to Power BI is a significant milestone in the strategy of bringing Power BI and Analysis Services closer together. As well as closing the gap between IT-managed workloads and self-service BI, it presents a number of new opportunities for Power BI developers in terms of tooling, process and integrations. This post highlights some of the key advantages of this new capability and what they mean for the Power BI developer.
C# 8.0 nullable references: getting started in an existing codebase
C# 8.0's nullable references feature dramatically changes a fundamental aspect of the language. In this post, Ian explains how to you can soften the impact by enabling gradually across your projects.
Testing Power BI reports using SpecFlow and .NET
Despite being inherently difficult to test, the need to validate data modelling, business rules and security boundaries in Power BI reports is important, as well as the need for ensuring that quality doesn't regress over time as the insights evolve. This post explains that, by connecting to the underlying tabular model, it is possible to execute scenario-based specifications to add quality gates and build confidence in Power BI reports, just as any other software project.
Testing Power BI Dataflows using SpecFlow and the Common Data Model
Whilst testing Power BI Dataflows isn't something that many people think about, it's critical that business rules and associated data preparation steps are validated to ensure the right insights are available to the right people across the organisation. Data insights are useless, even dangerous, if they can't be trusted, so despite the lack of "official support" or recommended approaches from Microsoft, endjin treat Power BI solutions just as any other software project with respect to testing - building automated quality gates into the end to end development process. This post outlines an approach that endjin has used to test Power BI Dataflows to add quality gates and build confidence in large and complex Power BI solutions.
Azure Analysis Services - how to save money with automatic shutdown
Azure Analysis Services provides an enterprise-grade analytical platform with massive scale and flexibility. But, as one of the more expensive services in the Azure platform, consideration should be given to cost management, especially in multi-environment ALM scenarios. This post explains how to massively reduce running costs through automation using Powershell and orchestration tools like Azure DevOps.
Building a proximity detection pipeline
At endjin, our approach focuses on using scientific experimental method to support the creation of fully proved and tested decision making, and the use of scientific research to support our work. This post runs through how we applied that process to creation a pipeline to detect vessel proximity.This is an example which is based around the project we recently worked on with OceanMind. In this project we helped them to build a #serverless architecture which could detect vessel proximity in close to real time. The vessel proximity events we detected were then fed into machine learning algorithms in order to detect illegal fishing!Carmel also runs through some of the actual calculations we used to detect proximity, how we used #data projections to efficiently process large quantitities of incoming data, and the use of #durablefunctions to orchestrate the processing.
Using complex objects in BDD Scenarios with SpecFlow
During our projects at endjin, we often find ourselves evangelising Behaviour Driven Development, and specifically SpecFlow. In this post we look at a technique for defining complex test data objects in your Gherkin feature files, which we've also made available via the endjin-sponsored Corvus.NET project.
Azure Analysis Services: How to update the expression for a calculated column from .NET
This post explains how to update Azure Analysis Services model schemas from inside custom .NET applications. Whilst not a common scenario for most, it shows that this is easy to do using the AMO SDK. So, there's nothing stopping you from developing complex and rich end-user functionality over the top of your data analysis solutions - providing run-time, user-driven schema changes like "what if" analysis.
C# teamwork: share project config with common Nuget Build Asset Packages
See how to manage consistent default configuration across all your .NET projects by using NuGet build assets.
Integration Testing Azure Functions with SpecFlow and C#, Part 5 - Using Corvus.SpecFlow.Extensions in a build pipeline
If you use Azure Functions on a regular basis, you'll likely have grappled with the challenge of testing them. Even now, several years after their introduction, the testing story for Functions is not hugely well defined. In the final post in this series, we show how to ensure specs written using Corvus.SpecFlow.Extensions can run as part of your build pipeline.
Integration Testing Azure Functions with SpecFlow and C#, Part 4 - Controlling your functions with additional configuration
If you use Azure Functions on a regular basis, you'll likely have grappled with the challenge of testing them. Even now, several years after their introduction, the testing story for Functions is not hugely well defined. In the fourth of this series of posts, we look at how configuration can be supplied from your tests to the functions apps being tested.
Integration Testing Azure Functions with SpecFlow and C#, Part 3 - Using hooks to start Functions
If you use Azure Functions on a regular basis, you'll likely have grappled with the challenge of testing them. Even now, several years after their introduction, the testing story for Functions is not hugely well defined. In the third of a series of posts, we look at using classes in the Corvus.SpecFlow.Extensions library to run functions apps via scenario and feature hooks.
Integration Testing Azure Functions with SpecFlow and C#, Part 2 - Using step bindings to start Functions
If you use Azure Functions on a regular basis, you'll likely have grappled with the challenge of testing them. Even now, several years after their introduction, the testing story for Functions is not hugely well defined. In the second of a series of posts, we look at using step bindings provided by the Corvus.SpecFlow.Extensions library to run functions apps as part of your SpecFlow scenarios.
Integration Testing Azure Functions with SpecFlow and C#, Part 1 - Introduction
If you use Azure Functions on a regular basis, you'll likely have grappled with the challenge of testing them. Even now, several years after their introduction, the testing story for Functions is not hugely well defined. In the first of a series of posts, we look at some different approaches to testing your functions apps, and introduce the Corvus.SpecFlow.Extensions library.
Optimising C# for a serverless environment
In our recent project with OceanMind we used #AzureFunctions to process marine vessel telemetry from around the world. This involved processing huge quantities of data in close to real time. We optimised our processing for a #serverless environment, the outcome of which being that the compute would cost less than £10 / month!This post summarises some of the techniques we used, including some concrete examples of optimisations we made.#bigdata #dataprocessing #dataanalysis #bigcompute
High-performance C#: a test pattern for ref structs
C# 7.2 introduce ref structs, a new kind of type (Span<T> is a ref struct) designed to support certain high performance scenarios. There are constraints around their use, and when writing unit tests for our Ais.Net parser, this caused some challenges. This blog describes the technique we used to work around the constraints.
An experimental approach to software development
The application of scientific experimental process to software development leads to the development of fully-validated solutions. This approach provides you with confidence in designs and means that you can quickly identify ideas which are not worth pursuing. At endjin we use the ideas of hypotheses and experimentation when designing any solution and this gives us full confidence in the designs we produce. In this post we outline the steps and advantages of using this approach.
Azure Analysis Services - How to process an asynchronous model refresh from .NET
Integrating Azure Analysis Services into custom applications doesn't just mean read-only data querying. But if your application changes the underlying model, it will need to be re-processed before the changes take effect. This post describes how to use the REST API for Azure Analysis Services inside a custom .NET application to perform asynchronous model refreshes, meaning your applications can reliably and efficiently deal with model updates.
Power BI Data Type Mappings
If you've worked with Power BI at all, you'll have probably realised that there are numerous mediums through which you work with (potentially the "same") data. Data types across these mediums can be called different things, but actually refer to the same thing. They can also (unsurprisingly) be called different things and actually mean different things. It's useful to know what the corresponding data types are across these mediums, as you may need to, for example, convert queries from one format to another. This blog and containing report intend to clarify what the corresponding data types are across each of the separate mediums within Power BI.
C# developers! Your scoped components are more dangerous than you think.
There are hidden pitfalls with dependency injection, particularly when managing the lifetime of scoped components. What is safe? And are there other approaches we can take to managing scoped object lifetimes?
Azure Analysis Services: How to execute a DAX query from .NET
Being able to construct DAX queries dynamically in C# means the possibilities are endless in terms of integrating Azure Analysis Services queries into your custom applications, and with the code samples in this post, you have everything you need to get started.
An Overview of the Azure CNAB Quickstarts Library
The Azure CNAB Quickstarts Library helps you get up and running with CNAB and Porter. We've built quickstarts covering solutions like WordPress, Ghost, Mattermost, and data platforms like Apache Airflow, SQL Server AlwaysOn clusters and Kubernetes features like an nginx ingress controller and an Azure AD enabled OAuth2 Proxy. We've condensed all our learnings from the past 9 months of working on the project and turned them into a 10 minute video which explains all the key concepts. We hope this video helps accelerate your own CNAB & Porter epiphanies!
Introducing the Azure CNAB Quickstarts Library
In partnership with Microsoft, we have released Azure CNAB Quickstarts Library on GitHub. CNAB (Cloud-Native Application Bundle) is a new specification designed for facilitating the packaging, installation, upgrading and uninstallation of cloud-native solutions in the cloud, on-premise or on the edge. We've created a number of quickstarts covering Apache Airflow, Azure Kubernetes Service, Ghost, Kubeflow, SQL Server Always On and Wordpress to help demonstrate the power of CNAB and Porter.
Azure Analysis Services - How to query all the measures in a model from .NET
Integrating Azure Analysis Services into custom applications means more than just querying the data. By surfacing the metadata in your models, you can build dynamic and customisable UIs and APIs, tailored to the needs of the client application. This post explains how easy it is to query model metadata from .NET, so you can create deeper integrations between your data insights and your custom applications.
Azure Analysis Services: How to open a connection from .NET
One of the first steps in integrating Azure Analysis Services into your applications is creating and opening a connection to the server - just like any other database technology. This post explains the ins and outs of creating Azure Analysis Services connections, including code samples for each of the key scenarios.
NDC London Day 1
In this post, Ian describes some of the highlights from the NDC London conference
Comparing GitHub Actions and Azure Pipelines: High-level concepts
GitHub Actions is GitHub's new CI/CD platform. It is comparable with Azure Pipelines, which forms part of the Azure DevOps suite. In this post, Mike Larah looks at the similarities and differences in the high-level concepts and terminology between the two platforms.
Design patterns in C# - The Builder Pattern
This is the second blog in a series around design patterns. This post focuses on the builder pattern. The builder pattern is used when there is complex set up involved in creating an object. Like the other creational patterns, it also separates out the construction of an object from the object's use.
How to use the AzureAD module in PowerShell Core
Whilst some of the Azure Active Directory PowerShell for Graph module (AzureAD) functionality has been rolled into the new Azure PowerShell Az module, it's not currently (and might never be) a replacement for the full power of what you can achieve with AzureAD. So, there's every chance you'll find yourself needing to use both side-by-side. This post explains how to do that using the new cross-platform PowerShell Core.
How to create a Power BI workspace in an Azure DevOps Pipeline using Powershell
A Power BI based solution typically consists of a variety of technologies - for example Azure data platform services containing source data. As such, automation of Power BI resources needs to be considered as part of a wider DevOps strategy. This post describes the specific steps needed in order to fully automate the creation and security of Power BI workspaces using Powershell and Azure DevOps pipelines.
Agile Architecture: Painting by Numbers
A code review with NDepend Part 2: The initial review
At endjin we have a high quality bar when it comes to our code. As part of this we carry out regular code reviews. One of the tools we have used for these code reviews is NDepend. This is the second in a blog series written as we carried out that process. This post focuses on the insight you can quickly gain just by glancing at the NDepend UI.
Enforce resource tagging with Azure Policy
This blog post details how we used Azure Policy to enforce Azure resources were tagged with appropiate tags and ensured tags were inherited from parent resource groups where possible.
A code review with NDepend Part 1: Quality metrics
At endjin we have a high quality bar when it comes to our code. As part of this we carry out regular code reviews. One of the tools we have used for these code reviews is NDepend. This is the first in a blog series written as we carried out that process. This post runs through the different metrics used by NDepend, and the reasons that each of these can be an indication of code quality.
ML.NET, Azure Functions and the 4th Industrial Revolution
A conversation about .NET, The Cloud, Data & AI, teaching software engineers and joining endjin with Ian Griffiths
When he joined endjin, Technical Fellow Ian sat down with founder Howard for a Q&A session. This was originally published on LinkedIn in 5 parts, but is republished here, in full. Ian talks about his path into computing, some highlights of his career, the evolution of the .NET ecosystem, AI, and the software engineering life.
Rx operators deep dive part 2: Slowly aggregating knowledge
This is the second blog in a series which delves into how the Rx operators work under the covers. This series aims to provide a greater understanding of Rx and its operators. This post focuses on the AGGREGATE operator.
Explicit interface implementation
Two of the main use-cases for explicit interface implementation are: 1. to hide members of the interface in a class which inherits from that interface, and 2. to work around the scenario when a class is inheriting from two interfaces which share a member of the same name. Take a look at this blog to go into more depth about each of those scenarios.
Using Step Argument Tranformations in SpecFlow to pass null values into scenarios
A good test suite should validate behaviour across your code base, testing as many edge cases as reasonably possible. A common "edge case", is passing a null value when a value isn't expected to be null. In Specflow, all values used in scenario examples are treated as strings, so it isn't possible (by default) to pass in a null value. However, using "Step Argument Transformations", we can achieve the desired behaviour. Read this blog to learn how this can be implemented.
Understanding Rx: Making the interfaces, subscribing and other Subject<T>s finally click
At endjin we've done quite a lot of work around reactive data processing. This post is a great introduction to the main concepts in reactive programming. It runs through the main classes in Rx.NET and provides an easy-to-follow explanation of this powerful (and somewhat complex) area of .NET.
Overflowing with dataflow part 2: TPL Dataflow
This is the second blog in a series about data flow. This post delves into TPL dataflow.The task parallel library is a .NET library which aims to make parallel processing and concurrency simpler to work with. The TPL dataflow library is specifically aimed at making parallel data processing more understandable via a pipeline-based model.
Async pitfalls: deferred work and resource ownership
Whose package is it anyway? Why it's important to minimise dependencies in your solutions...
This is a post about the importance of reducing your dependencies as much as possible when designing solutions.If as much as possible is under your own control, there is far less room for synchronization errors and update conflicts.
Thoughts about .NET, The Cloud, AI, ML, and teaching software engineers
Garbage collection, a memorandum on memory in C#
Garbage collection is the process in .NET of cleaning up unused memory. Here is a great post which explains how it works under the covers. If you have ever wanted to get an understanding of exactly what is going on behind the scenes when you write C# code, then this is the post for you!
Branches, builds and modern data pipelines. Let's catch-up!
As an apprentice engineer at endjin, you cover a lot of ground, especially at a consultancy which works with the latest and greatest tools to solve its clients' problems. Learn about endjin's Modern Data Platform, which is a culmination of IP, processes and knowledge built from years of implementing high-performance data-driven solutions. Also learn about the types of tools an apprentice gets to use, and the types of things an apprentice learns along the way.
OpenAPI code generators for Visual Studio
How to choose the right .NET technology or "learn to stop worrying and love NetStandard2.0"
Advanced Azure Resource Manager template patterns – t-shirt sizing and optional resources
One very useful but little used pattern when working with Resource Manager templates, is the ability to use parameters to optionally deploy resources, constrain certain resource configurations based on other user defined parameters, or to toggle parameters based on other values. This blog post explores some of these options.
Automating creation of new ALM environments using PowerShell and Azure DevOps
Did you know that you can fully automate the bootstrapping process of setting up an Azure DevOps enviornment? This post shows you how.
"But it works on my cloud!" - are your developers still making the same mistakes in a world of DevOps and PaaS services?
In the world of DevOps, cloud and platform services, how does a developer's "definition of done" need to change? This post argues that as the silos of development and operations are broken down, the responsibility of understanding the whole solution increases meaning, to truly take advantage of the cloud, the need for quality and professionalism is critical for success.
Automating R Unit Tests With Azure DevOps
Many organisations are starting to adopt the R Programming Language for their data science and financial modelling scenarios. But just because the language is being used for modelling, doesn't mean you should write unit tests that can be exercised as part of your CI/CD pipeline. In this blog post Jess Panni demonstrates how you can run R unit tests inside Azure DevOps.
Deploying to Azure using Azure Resource Manager templates and Octopus Deploy
Learn how to combine the power of Azure Resource Manager and Octopus Deploy for a frictionless Azure DevOps experience.
Using Postman to load test an Azure Machine Learning web service
Azure Machine Learning Studio is a fantastic service for experimentation, but you can also easily productionize your machine learning models. In this post we show how to create an Azure ML Studio web service and test it using Postman.
TeamCity MetaRunner for creating Release Annotations in Azure Application Insights
Year 2 as a software engineering apprentice at endjin
Machine Learning - the process is the science
What do machine learning and data science actually mean? This post digs into the detail behind the endjin approach to structured experimentation, arguing that the "science" is really all about following the process, allowing you to iterate to insights quickly when there are no guarantees of success.
Improve your Windows Command Prompt and PowerShell experience with ConEmu
Mike Larah shares his tips for how to best customize ConEmu to improve your terminal experience for Windows Command Prompy and PowerShell
SpecFlow 2.0, NUnit3, Visual Studio and TeamCity
ILMerge Meta-Runner for TeamCity
Azure Resource Manager authentication from a PowerShell script
Learn how to authenticate against Azure Resource Manager using PowerShell, for fully automated deployments.
CucumberJS Visual Studio Test Runner
We've built and open-sourced a Visual Studio Extension that provides basic support for discovering and running CucumberJS tests with Node.JS, in the Visual Studio Test Environment.
Chaos with a rhythm - controlling innovation
Running a start-up or bootstrapping a new product within an established business can feel pretty chaotic. We talk about strategies for managing the time of a team, and ensuring that you become predictable.
A Step-by-step guide to setting up CucumberJS in Visual Studio 2015
This is a quick guide to setting up cucumber JS in Visual Studio with grunt.
Reduce Merge Conflicts in .NET Solutions with PowerShell based Git Hooks
SpecFlow Extensions for Azure Storage Emulator
Improve your SpecFlow scenarios with custom step argument transformations
One of the great benefits of using SpecFlow is that it allows you to write your specifications in a human readable format. Learn how you can create reusable step argument transformations to apply custom transformations to your parameters, so that you can keep your specifications easy to read.
You're hiring the wrong people: 10 tips to find great developers
Hiring developers is hard. Really hard. Here are some tips on how to make it easier on yourself.
A step by step guide to choosing the right Azure technology for your solution
To help guide the client through the selection process (and to act as an aide memoire when we come to try and remember why particular choices were made), we created a handy flow-chart to take you through the technology landscape, and suggest some recommended-practice solutions.
Generating and using a certificate to authorise Azure Automation
Refactoring by pattern using Regular Expressions and ReSharper search by pattern
Using SmartGit to follow the GitFlow branching and workflow model
Elasticsearch gotchas and tips - part 2
We've delivered a number of Elasticsearch solutions on Azure. In this second post in the series, we share some useful tips.
A step by step guide to developing Visual Studio item templates using SideWaffle
In this post, we explore the use of SideWaffle to create a frictionless experience for developers wanting to create repeatable plugins.
A Step by Step Guide to Testing NuGet Packages Locally
There's a lot of documentation available around NuGet and how to create/publish packages, etc. But when I looked for a simple step-by-step guide on how to test a package locally, I couldn't find any.
Create a custom TeamCity PowerShell MetaRunner to Notify NewRelic that new deployment has occurred
Using Git for .NET Development: Part 4 - Resolving Merge Conflicts
Understanding dependency injection
Getting started with semantic logging
Semantic logging enables much richer diagnostics and analysis. It is supported in Windows and in .NET. This post explores the problem space. Others in the series explore how we can use Reactive Extensions to enable some powerful scenarios.