How do you configure liveness and readiness probes for your ASP.NET Core Web API application in AKS?

Liveness and readiness probes are essential Kubernetes features configured within your Azure Kubernetes Service (AKS) deployment YAML to monitor the health of your ASP.NET Core Web API. Liveness probes ensure your application process is running and trigger restarts upon failure, while readiness probes check if it’s ready to accept traffic, removing unhealthy pods from service. Both typically utilize dedicated HTTP endpoints in your API for comprehensive health checks, leveraging ASP.NET Core’s built-in health check system.

Understanding Liveness and Readiness Probes

In a containerized environment like Kubernetes, applications need robust mechanisms to report their health. Liveness and readiness probes serve this critical purpose, enabling Kubernetes to manage the lifecycle and traffic routing for your ASP.NET Core Web API applications effectively.

What is a Liveness Probe?

A liveness probe determines if the application process inside a container is alive and running. Its primary function is to detect situations where an application might be running but is in a broken state, such as a deadlock, an unresponsive loop, or a memory leak that causes it to hang. If a liveness probe fails, Kubernetes assumes the application is unhealthy and takes corrective action by restarting the pod. This self-healing mechanism is crucial for recovering from internal application failures without manual intervention. It generally performs a lightweight check, focusing on the basic responsiveness of the application.

What is a Readiness Probe?

A readiness probe determines if the application is ready to handle incoming requests and accept traffic. Unlike liveness probes, a failing readiness probe does not trigger a pod restart. Instead, it signals to Kubernetes that the pod is not yet ready (or is no longer ready) to serve traffic. Consequently, Kubernetes removes the pod from the service’s load balancer, preventing new requests from being routed to it. This is vital for scenarios such as:

• Application startup, where dependencies (like databases or external services) might still be initializing.
• Temporary outages of critical backend services.
• Graceful shutdown procedures, allowing existing connections to drain before traffic is completely cut off.

Once the readiness probe passes again, the pod is automatically added back to the service and begins receiving traffic. Readiness checks are typically more comprehensive than liveness checks, often validating external dependencies.

Implementing Health Checks in ASP.NET Core

ASP.NET Core provides a robust and extensible health check system that integrates seamlessly with Kubernetes probes. The general approach involves creating specific HTTP endpoints within your application that Kubernetes can poll.

1. Setting Up Health Check Endpoints

It’s best practice to create separate endpoints for liveness and readiness checks to provide granular control over what each probe evaluates:

• /health/live: This endpoint should perform very basic checks, such as ensuring the application process is running and responsive. It should not check external dependencies, as a failure here would lead to an unnecessary pod restart.
• /health/ready: This endpoint should perform more comprehensive checks, including connectivity to critical external dependencies like databases, message queues, or other microservices. If any critical dependency is unavailable, this endpoint should report as unhealthy, signaling that the application cannot serve traffic effectively.

2. Configuring Health Checks in Your ASP.NET Core Application

You configure health checks in your ASP.NET Core application’s startup logic (e.g., in Program.cs for .NET 6+ or Startup.cs for earlier versions).

First, register the health check services and any custom health checks:

// Program.cs (for .NET 6+) or Startup.cs's ConfigureServices method

builder.Services.AddHealthChecks()
    // Add a basic liveness check
    .AddCheck("live-check", () => HealthCheckResult.Healthy("A basic liveness check is healthy."))
    // Add a readiness check for SQL Server (requires Microsoft.Extensions.Diagnostics.HealthChecks.SQLServer)
    .AddSQLServer(
        connectionString: builder.Configuration.GetConnectionString("DefaultConnection"),
        name: "sqlserver-check",
        failureStatus: HealthStatus.Unhealthy,
        tags: new[] { "ready" })
    // Add a readiness check for an external API (requires a custom IHealthCheck implementation)
    .AddCheck<ExternalApiServiceHealthCheck>(
        name: "external-api-check",
        failureStatus: HealthStatus.Unhealthy,
        tags: new[] { "ready" });
    

Next, map the health check endpoints in your application’s request pipeline:

// Program.cs (for .NET 6+) or Startup.cs's Configure method

app.MapHealthChecks("/health/live", new HealthCheckOptions
{
    Predicate = healthCheck => healthCheck.Name == "live-check", // Only run the 'live-check'
    ResponseWriter = HealthCheckResponseWriters.JsonResponseWriter
});

app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = healthCheck => healthCheck.Tags.Contains("ready"), // Run all checks tagged 'ready'
    ResponseWriter = HealthCheckResponseWriters.JsonResponseWriter
});
    

For custom health checks, implement the IHealthCheck interface:

public class ExternalApiServiceHealthCheck : IHealthCheck
{
    private readonly IHttpClientFactory _httpClientFactory;

    public ExternalApiServiceHealthCheck(IHttpClientFactory httpClientFactory)
    {
        _httpClientFactory = httpClientFactory;
    }

    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        try
        {
            var client = _httpClientFactory.CreateClient();
            var response = await client.GetAsync("https://api.example.com/status", cancellationToken);

            if (response.IsSuccessStatusCode)
            {
                return HealthCheckResult.Healthy("External API is healthy.");
            }
            else
            {
                return HealthCheckResult.Unhealthy($"External API is unhealthy: {response.ReasonPhrase}");
            }
        }
        catch (Exception ex)
        {
            return HealthCheckResult.Unhealthy($"External API check failed: {ex.Message}");
        }
    }
}
    

Configuring Probes in Kubernetes Deployment YAML

Once your ASP.NET Core application exposes the health check endpoints, you configure the livenessProbe and readinessProbe within the container definition of your Kubernetes Deployment YAML.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-web-api-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-web-api
  template:
    metadata:
      labels:
        app: my-web-api
    spec:
      containers:
      - name: my-web-api
        image: youracr.azurecr.io/my-web-api-image:v1.0.0 # Replace with your image
        ports:
        - containerPort: 80
        livenessProbe: # Checks if the app is alive
          httpGet:
            path: /health/live # The liveness endpoint defined in ASP.NET Core
            port: 80
          initialDelaySeconds: 15 # Wait 15 seconds before the first probe
          periodSeconds: 10       # Probe every 10 seconds
          timeoutSeconds: 5       # Probe times out after 5 seconds
          failureThreshold: 3     # Allow 3 consecutive failures before restart
        readinessProbe: # Checks if the app is ready to serve traffic
          httpGet:
            path: /health/ready # The readiness endpoint defined in ASP.NET Core
            port: 80
          initialDelaySeconds: 30 # Wait 30 seconds before the first probe
          periodSeconds: 10       # Probe every 10 seconds
          timeoutSeconds: 5       # Probe times out after 5 seconds
          failureThreshold: 3     # Allow 3 consecutive failures before removing from service
    

Key Probe Parameters Explained:

• httpGet: Specifies an HTTP GET request to be performed. For ASP.NET Core Web APIs, this is the most common and recommended probe type.
  • path: The path to the health check endpoint (e.g., /health/live or /health/ready).
  • port: The port on which the container is listening for HTTP requests.
• initialDelaySeconds: The number of seconds after the container has started before liveness/readiness probes are initiated. This is crucial to give your application enough time to initialize and start listening on its port. Setting this too low can lead to premature restarts or pods never becoming ready.
• periodSeconds: How often (in seconds) Kubernetes should perform the probe. A shorter period detects issues faster but increases resource consumption.
• timeoutSeconds: Number of seconds after which the probe times out. If the probe takes longer than this, it’s considered a failure.
• failureThreshold: How many consecutive probe failures Kubernetes must observe for the probe to be considered failed.
  • For liveness probes, exceeding this threshold triggers a pod restart.
  • For readiness probes, exceeding this threshold removes the pod from the service’s load balancer.
• successThreshold: (Default: 1) How many consecutive successes are required for the probe to pass after having failed. This is particularly relevant for readiness probes to bring a pod back into service.

Best Practices and Advanced Considerations

Enhancing Resilience and Self-Healing

Properly configured liveness and readiness probes significantly improve the resilience and self-healing capabilities of your application in AKS. For instance, in a microservice architecture, if a central database experiences a brief outage, well-configured readiness probes on dependent services can detect the connection failure. This prevents them from receiving traffic, thus avoiding cascading failures and potential data corruption. Once the database recovers, the readiness probes pass, and the services automatically rejoin the load balancer, ensuring continuous availability.

Choosing the Right Probe Type

Kubernetes offers three main probe types:

• HTTP Probes: (Most preferred for Web APIs) They perform an HTTP GET request to a specified path and port. A successful HTTP response (status code 200-399) indicates success. This is ideal for ASP.NET Core Web APIs as it allows checking the actual application logic and responsiveness.
• TCP Probes: Attempts to open a TCP socket on the specified port. If a connection can be established, the probe is considered successful. Useful for applications that don’t expose HTTP endpoints but need port availability checked.
• Command Probes: Executes a command inside the container. If the command exits with status code 0, the probe is successful. Useful for custom health scripts or checking internal processes (e.g., cat /tmp/healthy or checking disk space).

For ASP.NET Core Web APIs, HTTP probes are generally preferred as they provide a more accurate picture of application health by interacting with the application layer.

Setting Appropriate Thresholds and Timeouts

Fine-tuning probe parameters is crucial for optimal stability. If the failureThreshold is too low, transient network blips or temporary resource spikes can cause unnecessary pod restarts or removals. Conversely, if it’s too high, a genuinely failing application might remain in service too long, leading to degraded user experience. Similarly, adjust periodSeconds to balance the frequency of checks with the overhead they introduce, and set timeoutSeconds to ensure probes don’t hang indefinitely.

Importance of Separate Endpoints

As highlighted, using distinct endpoints for liveness and readiness is a critical best practice. An application might be running (liveness passes) but temporarily unable to connect to a database or external service (readiness fails). Separate endpoints ensure that Kubernetes correctly distinguishes between a process crash (restart) and a temporary unavailability of dependencies (remove from service), preventing unnecessary restarts and improving overall system stability.

Leveraging ASP.NET Core’s Health Check System for Custom Checks

Beyond basic connectivity, ASP.NET Core’s health check system allows you to implement custom health checks using the IHealthCheck interface. This enables you to monitor the status of specific application components, such as a message queue connection, a cache service, or even the health of internal background processes. This provides a more comprehensive and accurate assessment of your application’s operational readiness.

Conclusion

Configuring liveness and readiness probes for your ASP.NET Core Web API in AKS is a fundamental practice for building resilient and self-healing cloud-native applications. By combining ASP.NET Core’s built-in health check capabilities with robust Kubernetes probe configurations, you ensure that your applications are always available, responsive, and capable of gracefully handling transient failures and dependency issues.