Adventures in Dapr: Episode 5 - Debugging Containers with Visual Studio

At the end of the previous episode we had containerised the Dapr Traffic Control sample, using Docker Compose and the Visual Studio Container Tools. However, the Fine Collection Service was throwing the following exception:

fail: Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.AggregateException: One or more errors occurred. (Secret operation failed: the Dapr endpoint indicated a failure. See InnerException for details.)
       ---> Dapr.DaprException: Secret operation failed: the Dapr endpoint indicated a failure. See InnerException for details.
       ---> Grpc.Core.RpcException: Status(StatusCode="InvalidArgument", Detail="failed finding secret store with key kubernetes")
         at Dapr.Client.DaprClientGrpc.GetSecretAsync(String storeName, String key, IReadOnlyDictionary`2 metadata, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at Dapr.Client.DaprClientGrpc.GetSecretAsync(String storeName, String key, IReadOnlyDictionary`2 metadata, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
         at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
         at System.Threading.Tasks.Task`1.get_Result()
         at FineCollectionService.Controllers.CollectionController..ctor(ILogger`1 logger, IFineCalculator fineCalculator, VehicleRegistrationService vehicleRegistrationService, DaprClient daprClient) in /pp/Controllers/CollectionController.cs:line 27
         at lambda_method8(Closure , IServiceProvider , Object[] )
         at Microsoft.AspNetCore.Mvc.Controllers.ControllerActivatorProvider.<>c__DisplayClass7_0.<CreateActivator>b__0(ControllerContext controllerContext)
         at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass6_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isComplted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
         at Dapr.CloudEventsMiddleware.ProcessBodyAsync(HttpContext httpContext, String charSet)
         at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

Leveraging Dapr's Zipkin integration, we used its dashboard to establish that the service was attempting to access a Kubernetes secret, despite the fact that we aren't using Kubernetes!

So where to start?

If we drill-down into the stack trace and get underneath the outer Dapr layers, we notice this item on the stack:

at FineCollectionService.Controllers.CollectionController..ctor(ILogger`1 logger, IFineCalculator fineCalculator, VehicleRegistrationService vehicleRegistrationService, DaprClient daprClient) in /pp/Controllers/CollectionController.cs:line 27

This is pointing us towards the constructor for the Fine Collection Service, so let's look at that part of the code:

This is looking hopeful; line 27 is indeed trying to lookup a Kubernetes secret, so let's start by debugging this part of the code in our quest to understand why.

Setting Environment Variables for Visual Studio

As you'll recall from the last episode, we still have dependencies on some environment variables and these need to be set so that they are visible to Visual Studio. The easiest way to do this is using our handy deploy.ps1 script:

PS:> ./src/bicep/deploy.ps1 -ResourcePrefix <prefix> -Location <location> -SkipProvision

When the above completes the required environment variables will have been set, which we can double-check as follows:

PS:> gci env:/AZURE_*

Name                           Value
----                           -----
AZURE_CLIENT_ID                <some-guid>
AZURE_CLIENT_OBJECTID          <some-guid>
AZURE_CLIENT_SECRET            <some-strong-password>
AZURE_TENANT_ID                <some-guid>

The important final step, is to launch Visual Studio from this same PowerShell session as this will ensure it can access the above environment variables. With a default Visual Studio 2022 installation, the following does the trick:

PS:> & "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\devenv.exe"

Debug Multiple Containers with Visual Studio

If you followed the steps from the previous episode then Visual Studio should be setup and ready, if not, then check the following or refer back to that post for the full guide.

• Ensure Visual Studio will use container-based hosting/debugging:
• Ensure that your 'Startup Project' is set as 'docker-compose':

1. Set a breakpoint at line 23:
2. Confirm all the Docker containers by looking for the following in the 'Container Tools' output window:

   Done!  Docker containers are ready.
   

3. Hit F5 and Visual Studio will spring into life by rebuilding the solution and launching the services inside the containers that it prepared above.
4. If you notice the following message in the 'Container Tools' output window, then you need re-run the steps above to setup the required environment variables:

   The AZURE_TENANT_ID variable is not set. Defaulting to a blank string.
   The AZURE_CLIENT_ID variable is not set. Defaulting to a blank string.
   The AZURE_CLIENT_SECRET variable is not set. Defaulting to a blank string.
   

5. Once everything has been running for a little while the Traffic Control Service will eventually detect a pesky speeder and will trigger the Fine Collection Service. At this point Visual Studio should land on our breakpoint. If you hover of the runningInK8s variable you should that it is set to true:
6. Line 23 contains the logic that is now invalid when running in a container, but outside of Kubernetes. As you might expect the DOTNET_RUNNING_IN_CONTAINER environment variable is not designed to distinguish when running inside Kubernetes, so we need a different detection mechanism.
7. As it happens, when containers run inside of Kubernetes there is an environment variable that you can reasonably expect to be set:

   KUBERNETES_SERVICE_HOST
   

8. Armed with that knowledge, we can make the following changes:
9. Hit F5 and let's see if we've fixed the issue. This time when we hit the breakpoint we should see that it correctly identifies we are not running in Kubernetes:
10. Remove the breakpoint and press F5 to let everything run without interruption - NOTE: Even after removing the breakpoint you find execution halts a couple of times due to the Simulation and Traffic Control Service continuing to run whilst the Fine Collection Service is suspended at the breakpoint.
11. As in previous episodes, you can be happy that things are working as required once you start seeing e-mails arriving in the MailDev interface:

Review

In this post we've used the familiar debugging features of Visual Studio to debug an entire set of independent services. In this particular case we only needed to debug a single service, but we could have set breakpoints in any of them if required. A key point to remember is that we could not have reproduced this particular issue using a Dapr Self-Hosted runtime environment. It's a useful reminder that for as much testing as we can do with mocking, stubs, emulators and such like there really is no substitute for testing your code in the same runtime environment as it runs in 'the real world'. With the Visual Studio Container Tools you can wield all its debugging might just as effectively in a complex multi-container environment as you can when running .NET applications 'locally'.