Extending Endjin.Retry with custom Retry Policies

By Howard van Rooijen Co-Founder 3rd March 2015

Extending Endjin.Retry with custom Retry Policies

Update: This project has been replaced by Corvus.Retry.

Someone tweeted @endjin this week to ask if we were still supporting our Endjin.Retry framework; they asked because they were worried it had been abandoned because it had only had 6 minor commits in 2014. I replied to confirm that we have not abandoned this project in the slightest, in fact it's one of our most used, core bits of IP (along with Endjin.Composition), that we use in practically every project we develop whether it's related specifically to Azure or not. We have found that it is easier to extend Endjin.Retry, through its IRetryPolicy and IRetryStrategy inside the consuming project rather than pollute the core library – this means all of these customisations happen in our private project repos, not in the public Endjin.Retry repo.

Below are a series of examples of custom retry policies we've implemented in various projects:

HttpRequestExceptionRetryPolicy

This a very simple retry policy that will retry if a HttpRequestException is encountered. A good general purpose transient HTTP error retry strategy

#region Using Directives

using System;
using System.Net.Http;

using Endjin.Core.Retry.Policies;

#endregion

public class HttpRequestExceptionRetryPolicy : IRetryPolicy
{
    public bool CanRetry(Exception ex)
    {
        return ex is HttpRequestException;
    }
}

DoNotRetryOnConflictPolicy

This retry policy was specifically designed to work with the Azure Storage SDK will retry on any exception except for a HTTP 409 Conflict status code – a scenario whereby the resource you're trying to create already exists.


#region Using Directives

using System;

using Endjin.Core.Retry.Policies;

using Microsoft.WindowsAzure.Storage;

#endregion 

public class DoNotRetryOnConflictPolicy : IRetryPolicy
{
    public bool CanRetry(Exception exception)
    {
        var storageException = exception as StorageException;

        if (storageException != null && storageException.RequestInformation.HttpStatusCode == 409)
        {
            return false;
        }

        return true;
    }
}

DoNotRetryOn304NotModified

Similar to the previous policy, this retry policy will retry except for a HTTP 304 Not Modified status code.

#region Using Directives

using System;
using System.Net;

using Endjin.Core.Retry.Policies;

using Microsoft.WindowsAzure.Storage;

#endregion

public class DoNotRetryOn304NotModified : IRetryPolicy
{
    public bool CanRetry(Exception exception)
    {
        var storageException = exception as StorageException;

        if (storageException == null)
        {
            return true;
        }

        return storageException.RequestInformation.HttpStatusCode != (int)HttpStatusCode.NotModified;
    }
}

StorageWriteExceptionRetryPolicy

The following retry policy was created to retry except in the event of a HTTP Protocol error or a HTTP 409 Conflict status code

#region Using Directives

using System;
using System.Diagnostics;
using System.Net;

using Endjin.Core.Retry.Policies;
using Endjin.Kpi.Core.Extensions;

using Microsoft.WindowsAzure.Storage;

#endregion

public class StorageWriteExceptionRetryPolicy : IRetryPolicy
{
    public bool CanRetry(Exception ex)
    {
        Trace.TraceError(ex.GetFullExceptionMessage());

        var we = ex as WebException;
        
        if (we != null && we.Status == WebExceptionStatus.ProtocolError)
        {
            return false;
        }

        var se = ex as StorageException;

        if (se != null && se.RequestInformation.HttpStatusCode == (int)HttpStatusCode.Conflict)
        {
            return false;
        }

        return ex is LeaseException;
    }
}

DoNotRetryOnLeaseAcquisitionUnsuccessfulPolicy

This retry policy is part of our Endjin.Leasing framework, which implements Mutex operations over Azure Blob Storage and will retry unless a LeaseAcquisitionUnsuccessfulException is detected, and would enable a scenario whereby you'd have a series of competing actors who want perform an action against a specific resource, the first to obtain the Mutex wins and the other actors fail fast.

#region Using Directives

using System;

using Endjin.Core.Retry.Policies;

#endregion 

public class DoNotRetryOnLeaseAcquisitionUnsuccessfulPolicy : IRetryPolicy
{
    public bool CanRetry(Exception exception)
    {
        var storageException = exception as LeaseAcquisitionUnsuccessfulException;

        if (storageException != null)
        {
            return false;
        }

        return true;
    }
}

RetryUntilLeaseAcquiredPolicy

This retry policy is also part of our Endjin.Leasing framework but enables the opposite scenario than the previous policy; if multiple actors want to perform an action on a particular resource, the should retry until the obtain their Mutex:

#region Using Directives

using System;

using Endjin.Core.Retry.Policies;
    
#endregion

public class RetryUntilLeaseAcquiredPolicy : IRetryPolicy
{
    public bool CanRetry(Exception exception)
    {
        var actualException = exception as LeaseAcquisitionUnsuccessfulException;

        return actualException != null;
    }
}

Hopefully these examples show how easy it is to implement your own retry policies, which nicely encapsulate your success and failure conditions.