Polyglot Notebooks for Ops

Regular visitors here may have seen many posts from us about why we love using notebooks for data wrangling and the like, so much so that we treat them as first class development artefacts and think they should be tested like any other code.

For those of you not familiar with these notebooks, they provide an interactive compute environment that allow you to create and share documents that combine code, text explanations, visualizations, and more. The classic Jupyter notebooks are accessed via a browser connecting to either a hosted service or a web server you can run locally, however, now they can likely be accessed from inside your favourite editor via an extension. This gives a more seamless development experience that easily fits with your existing dev inner loop.

Given the original focus on data wrangling and experimentation, the languages supported by these notebooks were typically python, scala etc. However, a team at Microsoft have been working on a project called .NET Interactive that amongst other things adds support for the .NET ecosystem inside notebooks. It also integrates with Visual Studio Code via an extension called Polyglot Notebooks. Given that it supports .NET this also includes PowerShell, which is where things get interesting so far as this post is concerned.

Whilst the desire is to automate as many operational processes as possible, there are often situations where this isn't possible and/or desirable, for example:

• The process itself requires checks and/or approvals at various stages.
• The process is run infrequently, making it difficult to justify the cost of full automation and the robustness/resilience that it implies.
• The process requires different configuration each time it is run, potentially only known at the point that it is being run.
• The process is highly sensitive and requires human oversight to verify certain configuration or actions.
• The process requires permissions that cannot be granted to an unattended security identity due to security policy. (i.e. service account, service principal etc.)
• Where a process is being handed-over to another team or organisation, they may not have the facilities to run or capabilities to support a fully automated solution.
• The process is, initially, more exploratory in nature but will subsequently need to be repeatable by others

Typically we see these types of scenarios supported using something like a Word or PDF document with a series of steps and screenshots that walk the operator through every step. However, such documents when used in this way exhibit common limitations:

• They are quite brittle to change, especially when they contain lots of screenshots
• Directing operators around a user interface can be error prone
• Difficult to document a process that changes subtlety between runs or in different contexts, without duplicating swathes of the document

Instead we believe that it is important that these types of operational scenarios are implemented such that:

• As much of the 'work' as possible is performed by automated steps (i.e. avoiding the operator's manual interactions with a user interface)
  • In this way the operator is more of an orchestrator of the overall process
• The automated steps use the security context of the person running the process
• The operator has context-sensitive guidance and instructions for how to run the process
• Configuration required by the process is maintained as part of or directly alongside the process itself
• It is stored in version control (i.e. treated as a source code artefact)

Given the title of this post, it should come as no surprise that interactive notebooks lend themselves to be used in this way quite splendidly!

Let's look at an example as a way to illustrate what we're talking about.

Installing Polyglot Notebooks Extension for Visual Studio Code

Assuming you already have Visual Studio Code installed, then getting started with a new interactive notebook is as easy as:

1. Open the 'Extensions' tab
2. Search for 'polyglot'
3. Install the 'Polyglot Notebooks' extension (it should be the first result)

Creating PowerShell-based Polyglot Notebook

For this example, let's imagine we need to run an audit of some Azure resources across our tenant. The notebook below shows how we can combine a commentary on the audit process and leveraging PowerShell to do the heavy lifting.

• Use CTRL+SHIFT+P (CMD+SHIFT+P on a Mac) to open the command palette
• Type 'polyglot' to show the available Polyglot Notebook commands
• Select the 'Polyglot Notebook: Create new blank notebook' option
• Next choose the format you want to use:
  • dib - this is a markdown-based format
    • Pro: A more readable format, especially when comparing source control diffs
    • Con: The format is not supported by the preview features in GitHub or Azure DevOps
  • ipynb - this is the standard Jupyter Notebooks format
    • Pro: Previews of the notebook can be rendered in GitHub or Azure DevOps
    • Con: As an JSON-based format, it is not so human-readable when comparing raw source control diffs
  • NOTE: The VS Code extension we're using has its own notebook-aware visual diff tooling which works for either file format.
• Choose 'PowerShell' as the default language
• You should now have a new document tab open with a notebook containing an empty code cell

Using PowerShell in a Polyglot Notebook

Now you have an empty notebook waiting to be filled, let's take a look at an example to show how they can be used.

NOTE: The italicised asides should be treated as blog commentary, rather then notebook content.

Other Scenarios

Hopefully that example has given you a taste of how you can use PowerShell with interactive notebooks (including scripts that you've previously written), but let's wrap-up this post with some other suggestions to get you thinking about other scenarios where you could use one.

• A 'one-off' migration process that needs to be run in different environments
• Security incident analysis - the ability to save the cell output as part of the notebook is particularly useful here as the notebook doubles as a permanent record of the analysis that others can review afterwards.
• Azure resource 'clean-up' process
• Customising and deploying Azure Landing Zones
• Implementing a 'wizard-style' guide that walks you through data gathering steps before using that data to generate a configuration artefact to drive another fully-automated process.