Use Resilience Patterns in Microservices

Resilience patterns help software systems recover gracefully from failures, ensuring minimal disruption to the user experience. In this post, we’ll explore what resilience means in a microservices architecture, common resilience patterns, and how to implement them effectively.

Use Resilience Patterns in Microservices

Getting Started

In the world of distributed systems, failure is inevitable. Microservices, by nature, introduce complexity due to their interdependencies and networked communication. As systems grow in scale, ensuring stability and availability becomes increasingly critical. This is where resilience patterns come into play.

What is Resilience in Microservices?

Resilience is the system's capacity to withstand failures and continue operating. It doesn’t imply that failures won’t occur, but that the system is built to manage them gracefully without crashing or causing widespread issues.

For example, if one microservice fails or responds slowly, others depending on it should degrade gracefully or retry intelligently, rather than failing catastrophically.

Resilience patterns in microservices are essential for building systems that are robust, fault-tolerant, and capable of recovering gracefully from failures. Here's a clear breakdown of why you should use resilience patterns in microservices:

Why Resilience is Critical in Microservices

Microservices are:

Distributed: Services run on different machines or environments.
Network-Dependent: Communication relies heavily on networks, which can be slow or unreliable.
Decentralized: Each service is independently deployed, scaled, and updated.
Interdependent: Failures in one service can cascade and bring down others.

So without resilience patterns, your system is vulnerable to:

Downtime (from cascading failures)
Poor user experience (slow or broken services)
Hard-to-diagnose issues (intermittent or hidden faults)

Benefits of Resilience Patterns

Fault Isolation Prevents one failing service from taking down the entire system.
Graceful Degradation Keeps core functionality available even if some parts fail.
Improved Uptime Automatically recovers from transient errors.
Better User Experience Fewer failures or unresponsiveness visible to users.
Operational Insights Helps monitor, retry, and report failures effectively.

Common Resilience Patterns in Microservices

Circuit Breaker Prevents repeated calls to a failing service by quickly falling back or returning default data.
Retry Automatically retries a failed request after a short delay often combined with exponential backoff.
Timeout Prevents the system from hanging by limiting how long a service waits for a response.
Fallback Provides an alternative response (e.g., cached data) when the main service is unavailable.
Bulkhead Isolates services so failure in one doesn't overload others (like compartments in a ship).
Rate Limiting Protects services from being overwhelmed by too many requests.
Fail Fast Immediately returns an error if the service knows it can’t fulfill the request.

Create a Microservice Using Resilience Patterns

Creating a microservice using the Resilience Pattern in C# typically involves implementing retry, circuit breaker, timeout, fallback, and bulkhead isolation patterns.

We'll build a simple .NET microservice that that fetches product data from a remote InventoryService and uses Polly, a .NET resilience and transient-fault-handling library, to implement resilience patterns and apply above patterns.

Technologies Used

.NET 7 or 8 (applies to both)
ASP.NET Core Web API
Polly (for resilience)

Create the Project

Open Visual Studio
Click Create a new project
Select ASP.NET Core Web API
Click Next
Name your project (e.g., MyWebApi)
Choose .NET 6 or later (recommend .NET 8 if available)
Click Create

Add Polly NuGet Package

Right-click on your project in Solution Explorer
Choose Manage NuGet Packages
Click on the "Browse" tab
Search for: Microsoft.Extensions.Http.Polly
Click Install

Configure HttpClient with Polly

Program.cs

using Polly;
using Polly.Extensions.Http;
using Polly.Timeout;
var builder = WebApplication.CreateBuilder(args);
// Polly policies
var retryPolicy = HttpPolicyExtensions
.HandleTransientHttpError()
.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));
var circuitBreakerPolicy = HttpPolicyExtensions
.HandleTransientHttpError()
.CircuitBreakerAsync(2, TimeSpan.FromSeconds(30));
var timeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(2); // 2 seconds timeout
var fallbackPolicy = Policy<HttpResponseMessage>
.Handle<Exception>()
.FallbackAsync(
fallbackAction: (ct) =>
{
  var response = new HttpResponseMessage(System.Net.HttpStatusCode.OK)
  {
    Content = new StringContent("{ \"message\": \"Fallback response from Polly\" }")
    };
    return Task.FromResult(response);
    });
    builder.Services.AddHttpClient("InventoryClient", client =>
    {
      client.BaseAddress = new Uri("https://mock-inventory-service/api/");
      })
      .AddPolicyHandler(fallbackPolicy)
      .AddPolicyHandler(retryPolicy)
      .AddPolicyHandler(circuitBreakerPolicy)
      .AddPolicyHandler(timeoutPolicy);
      // Add services to the container.
      builder.Services.AddControllers();
      builder.Services.AddEndpointsApiExplorer();
      builder.Services.AddSwaggerGen();
      var app = builder.Build();
      app.UseSwagger();
      app.UseSwaggerUI();
      app.UseAuthorization();
      app.MapControllers();
      app.Run();

api resilience patterns

Create a Service

Services/IInventoryService.cs

public interface IInventoryService
{
  Task<string> GetInventoryStatusAsync(string productId);
}

Services/InventoryService.cs

public class InventoryService : IInventoryService
{
  private readonly IHttpClientFactory _httpClientFactory;
  private readonly ILogger<InventoryService> _logger;
  public InventoryService(IHttpClientFactory httpClientFactory, ILogger<InventoryService> logger)
  {
    _httpClientFactory = httpClientFactory;
    _logger = logger;
  }
  public async Task<string> GetInventoryStatusAsync(string productId)
  {
    var client = _httpClientFactory.CreateClient("InventoryClient");
    try
    {
      var response = await client.GetAsync($"inventory/{productId}");
      response.EnsureSuccessStatusCode();
      var content = await response.Content.ReadAsStringAsync();
      return content;
    }
    catch (Exception ex)
    {
      _logger.LogError(ex, "Error fetching inventory for product {ProductId}", productId);
      return "Inventory status unavailable.";
    }
  }
}

builder.Services.AddScoped<IInventoryService, InventoryService>();

Create Controller

Controllers/ProductController.cs

using Microsoft.AspNetCore.Mvc;
[ApiController]
[Route("api/[controller]")]
public class ProductController : ControllerBase
{
  private readonly IInventoryService _inventoryService;
  public ProductController(IInventoryService inventoryService)
  {
    _inventoryService = inventoryService;
  }
  [HttpGet("{id}")]
  public async Task<IActionResult> GetProduct(string id)
  {
    // Normally, fetch product details from DB here
    var inventoryStatus = await _inventoryService.GetInventoryStatusAsync(id);
    var product = new
    {
      Id = id,
      Name = "Sample Product",
      InventoryStatus = inventoryStatus
      };
      return Ok(product);
    }
  }

Testing

Run project.
Navigate to http:/localhost:port/api/ endpoint.
Simulate slow responses or failures in that service.
Observe retries, circuit breaker, and fallback in action.

Summary

Resilience is essential not optional in a microservices architecture. By adopting resilience patterns, you enhance the robustness, reliability, and user experience of your system. Thoughtful implementation of these patterns enables services to fail gracefully and recover swiftly, minimizing downtime and promoting overall system stability.

Thanks

Use Resilience Patterns in Microservices