How would you troubleshoot a scenario where your ASP.NET Core Web API pods are constantly crashing in AKS ?

Question

How would you troubleshoot a scenario where your ASP.NET Core Web API pods are constantly crashing in AKS ?

Brief Answer

When ASP.NET Core Web API pods are constantly crashing in AKS, I’d follow a systematic troubleshooting approach:

  1. Analyze Pod Logs First: Crucially, use kubectl logs <pod-name> --previous to find application-specific exceptions, stack traces, or configuration errors that directly led to the crash.
  2. Evaluate Resource Consumption & Limits: Check kubectl top pod and kubectl describe pod for OOMKilled events. Verify that requests and limits for CPU/memory are appropriately defined in your deployment YAML to prevent resource starvation or evictions.
  3. Verify Liveness & Readiness Probes: Misconfigured probes can cause unnecessary restarts. Ensure initialDelaySeconds is adequate and the probe endpoints (e.g., /healthz, /ready) accurately reflect application health, preventing false positives.
  4. Review Recent Deployments: Use kubectl rollout history to identify if a recent code or configuration change introduced a regression, and be prepared to kubectl rollout undo if suspected.
  5. Assess AKS Node Health: Briefly check kubectl get nodes for any NotReady nodes, as underlying node issues can affect pods.

Beyond these steps, I always emphasize using kubectl describe pod for deep insights into pod events, and stress the importance of proactive monitoring and alerting to catch issues before they escalate, demonstrating a comprehensive understanding of operational stability.

Super Brief Answer

To troubleshoot crashing ASP.NET Core API pods in AKS:

  1. Check Pod Logs: Use kubectl logs --previous for crash details (exceptions, errors).
  2. Review Resource Limits: Verify CPU/memory limits and check for OOMKilled events.
  3. Inspect Liveness/Readiness Probes: Ensure they aren’t causing false restarts or unreadiness.
  4. Assess Recent Deployments: Identify if a new deployment caused the issue and consider a rollback.

Detailed Answer

When your ASP.NET Core Web API pods are constantly crashing in Azure Kubernetes Service (AKS), a systematic troubleshooting approach is crucial. Begin by examining pod logs for application-specific errors. Next, investigate resource constraints (CPU/memory) and the configuration of liveness and readiness probes. Review recent deployments for regressions, and finally, check the health of the underlying AKS nodes. This comprehensive strategy will help pinpoint the root cause of the instability.

Related Topics: AKS, ASP.NET Core Web API, Kubernetes, Pod Troubleshooting, Crashing Pods, Azure, Container Orchestration

Systematic Troubleshooting Steps for Crashing Pods

Follow these steps to diagnose and resolve persistent crashes in your ASP.NET Core Web API pods within AKS:

1. Analyze Pod Logs

The first and most critical step is always to check the application logs. These logs provide direct insight into what the application was doing immediately before the crash.

  • Action: Use kubectl logs <pod-name> to view the logs of the currently running pod.
  • Crucial Tip: If the pod has restarted, use kubectl logs <pod-name> --previous (or -p) to retrieve logs from the terminated instance. This is vital as the previous instance’s logs often contain the error that led to the crash.
  • Look For: Specific ASP.NET Core exceptions, detailed error messages, stack traces, and any unusual log entries or patterns preceding the crash. Common issues include database connection failures, unhandled exceptions, or configuration errors.

2. Evaluate Resource Consumption and Limits

Resource exhaustion (CPU or memory) is a very common reason for pod crashes. Kubernetes can terminate pods that exceed their defined limits.

  • Check Current Usage: Use kubectl top pod <pod-name> to see the current CPU and memory usage of your pods.
  • Examine Node Capacity: Use kubectl describe node <node-name> to understand the overall resource capacity and allocation of the underlying nodes where your pods are scheduled.
  • Verify Resource Definitions: Ensure your deployment YAML correctly defines requests (guaranteed resources) and limits (maximum allowed resources) under the spec.containers.resources section.
  • Example Configuration: 
    resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    cpu: 200m
    memory: 512Mi

    Impact of Incorrect Limits: If limits are too low, your application might crash due to insufficient resources (e.g., an “OOMKilled” event for memory). If limits are too high or not set, a single misbehaving pod could starve other pods or nodes.

3. Verify Liveness and Readiness Probes

Misconfigured or overly aggressive liveness and readiness probes can lead to unnecessary pod restarts, making them appear to be “crashing.”

  • Liveness Probes: These determine if your application inside the pod is still running and healthy. If a liveness probe fails, Kubernetes will restart the pod. Ensure your liveness probe checks a fundamental health endpoint (e.g., /healthz) that confirms the application process is alive.
  • Readiness Probes: These determine if the pod is ready to accept incoming network traffic. If a readiness probe fails, Kubernetes removes the pod from the service’s endpoints, preventing new traffic from being directed to it until it becomes ready again. This prevents traffic from hitting an unready or unhealthy instance.
  • Configuration Example for a Web API: 
    livenessProbe:
  httpGet:
    path: /healthz
    port: 80
  initialDelaySeconds: 15
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /ready
    port: 80
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3

    Considerations: Set appropriate initialDelaySeconds to give your application enough time to start up. Adjust periodSeconds, timeoutSeconds, and failureThreshold based on your application’s startup time and responsiveness to avoid false positives.

4. Review Recent Deployments

A recently deployed code change is a frequent culprit for new crashes. A bad deployment can introduce bugs or configuration errors that destabilize pods.

  • Check Rollout History: Use kubectl rollout history deployment/<deployment-name> to view the history of your deployments. This helps identify if a recent change correlates with the start of the crashes.
  • Rollback if Necessary: If a recent deployment is suspected, you can quickly revert to a previous, stable version using kubectl rollout undo deployment/<deployment-name>. This can restore service quickly while you thoroughly investigate the problematic deployment.

5. Assess AKS Node Health

While less common, issues with the underlying AKS nodes can also lead to pod instability and crashes.

  • Check Node Status: Use kubectl get nodes to quickly see the status of all nodes in your AKS cluster.
  • Look For: Any node with a NotReady status. A NotReady node indicates a problem (e.g., resource pressure, network issues, or Kubelet problems) that could be causing pods on that node to crash.
  • Further Investigation: If a node is unhealthy, you’ll need to investigate further using Azure portal diagnostics, AKS built-in insights, or by checking node logs.

Preparing for an Interview: Demonstrating Troubleshooting Expertise

When discussing troubleshooting scenarios in an interview, go beyond just listing commands. Show your thought process and understanding of the underlying principles.

1. Articulate a Systematic Troubleshooting Process

Interviewers appreciate a structured approach. Describe how you would systematically go through the problem, starting with immediate symptoms and drilling down.

Example Answer: “In a previous role, we faced intermittent API pod crashes. I followed a systematic approach. First, I checked the pod logs using kubectl logs, which revealed repeated database connection errors. Next, I used kubectl describe pod to confirm resource usage was within limits. Finally, I investigated the database, identifying a temporary network glitch affecting connectivity. This structured method allowed us to quickly pinpoint and resolve the issue.”

2. Leverage kubectl describe pod for Deep Insights

Demonstrate your familiarity with this powerful command and its output.

Example Answer: “The kubectl describe pod <pod-name> command is invaluable. It provides a wealth of information: the pod’s current status, recent events (like restarts, scheduling failures, or OOMKilled events), resource usage, and even the events related to liveness/readiness probes. For instance, I once used it to diagnose a pod crashing due to exceeding its memory limit; the ‘Events’ section clearly showed the OOMKilled event, indicating a memory exhaustion issue.”

3. Explain the Importance of Resource Limits

Beyond just knowing how to set them, explain why they are crucial.

Example Answer: “Resource limits are essential for preventing a single pod from monopolizing node resources, which can lead to instability for other applications. Without proper limits, a runaway process in one pod could cause resource starvation for others or even lead to Kubernetes evicting pods to reclaim resources, resulting in application downtime. I’ve seen scenarios where forgetting to set memory limits caused cascading pod failures during peak traffic, underscoring their critical role in cluster stability.”

4. Showcase Understanding of Liveness and Readiness Probes

Explain their distinct roles and how they contribute to application resilience.

Example Answer: “Liveness and readiness probes are fundamental to Kubernetes’ self-healing capabilities. A liveness probe checks if the application process is alive and responsive; if it fails, Kubernetes restarts the pod. A readiness probe, on the other hand, determines if the application is ready to serve traffic; if it fails, the pod is temporarily removed from service endpoints. For our ASP.NET Core API, we implemented a liveness probe hitting a basic /healthz endpoint and a readiness probe checking /ready after essential services (like database connections) were established. This ensures only healthy pods receive traffic and unhealthy ones are promptly recovered.”

5. Emphasize Proactive Monitoring and Alerting

Show that you think beyond immediate fixes to long-term stability.

Example Answer: “While troubleshooting is reactive, true operational excellence comes from being proactive through robust monitoring and alerting. We used Azure Monitor to track key metrics like CPU/memory usage, pod restarts, and application-specific logs. We also configured alerts for anomalies, such as sudden spikes in resource consumption or sustained pod restarts. This allowed us to identify and address potential issues before they escalated into user-impacting incidents. For open-source alternatives, Prometheus with Grafana offers powerful monitoring and visualization capabilities.”