Kubernetes 04 - Inspecting Resources

Effectively inspecting your Kubernetes resources is crucial for maintaining a healthy cluster and troubleshooting issues as they arise. This guide will walk you through various commands and techniques to thoroughly examine your Kubernetes resources, providing you with the insights needed to manage your applications confidently.

Creating a Deployment

Let’s start with creating a deployment. This deployment will be used as an example throughout the guide.

kubectl create deployment my-apache --image httpd --replicas 3

This command creates a deployment named my-apache with two replicas using the httpd image.

Inspecting the Deployment

Basic Information

To get basic information about the deployment, use the following command:

kubectl get deploy/my-apache

This command outputs a summary of the deployment, including the number of replicas that are ready, up-to-date, and available, as well as the age of the deployment.

NAME        READY   UP-TO-DATE   AVAILABLE   AGE
my-apache   3/3     3            3           10m

Here’s what each column means:

READY: Shows how many replicas are ready out of the desired number
UP-TO-DATE: Indicates the number of replicas updated to the latest configuration
AVAILABLE: Displays the number of replicas available to users
AGE: Shows how long the deployment has been running

Detailed Information

For more detailed information, including the containers, images, and selectors used by the deployment, use the -o wide flag:

kubectl get deploy/my-apache -o wide

This provides additional details such as the containers, images, and selectors.

NAME        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES   SELECTOR
my-apache   3/3     3            3           8m24s   httpd        httpd    app=my-apache

The extra columns provide insights into:

CONTAINERS: The name of the container(s) in the pod
IMAGES: The Docker image(s) used
SELECTOR: The label selector used to identify pods managed by this deployment

Selectors are used to identify a set of objects in Kubernetes. They help match labels assigned to resources (like Pods) with the labels specified in the deployment or service specifications. In this case, the selector app=my-apache is used to ensure that the deployment manages pods labeled with app=my-apache.

Full Configuration in YAML

To examine the complete configuration of your deployment, use:

kubectl get deploy/my-apache -o yaml

This command outputs the entire configuration in YAML format, which is particularly useful for understanding all aspects of the resource, including any custom configurations or annotations.

Describing the Deployment

While kubectl get provides a summary, kubectl describe offers a more detailed view that combines related resources and events.

Deployment Summary

kubectl describe deploy/my-apache

This command provides a comprehensive summary of the deployment, including:

Deployment details
ReplicaSet status
Pod template
Events related to the deployment

The events section is particularly valuable for troubleshooting, as it shows a timeline of significant occurrences related to the deployment.

Here is an example output:

Name:                   my-apache
Namespace:              default
CreationTimestamp:      Mon, 22 Jul 2024 09:47:52 +0900
Labels:                 app=my-apache
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=my-apache
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=my-apache
  Containers:
   httpd:
    Image:        httpd
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   my-apache-5bd7979764 (3/3 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  20m   deployment-controller  Scaled up replica set my-apache-5bd7979764 to 1
  Normal  ScalingReplicaSet  20m   deployment-controller  Scaled up replica set my-apache-5bd7979764 to 3 from 1

Inspecting Individual Pods

To get detailed information about a specific pod within the deployment:

kubectl describe pod/my-apache-5bd7979764-55cbx

This command provides in-depth information about the pod, including its status, IPs, container details, and events.

Name:             my-apache-5bd7979764-55cbx
Namespace:        default
Priority:         0
Service Account:  default
Node:             docker-desktop/192.168.65.3
Start Time:       Mon, 22 Jul 2024 09:47:52 +0900
Labels:           app=my-apache
                  pod-template-hash=5bd7979764
Annotations:      <none>
Status:           Running
IP:               10.1.0.10
IPs:
  IP:           10.1.0.10
Controlled By:  ReplicaSet/my-apache-5bd7979764
Containers:
  httpd:
    Container ID:   docker://cd8273a2a9310dea3245e9fe36923b84491b374a2e8ca603daff7192e0ce42a2
    Image:          httpd
    Image ID:       docker-pullable://httpd@sha256:3584a496c4c0b2460755ac97e65caea2e7900be2a303cfb2a59f149aea858c78
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 22 Jul 2024 09:47:59 +0900
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gmdw9 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  kube-api-access-gmdw9:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  25m   default-scheduler  Successfully assigned default/my-apache-5bd7979764-55cbx to docker-desktop
  Normal  Pulling    25m   kubelet            Pulling image "httpd"
  Normal  Pulled     25m   kubelet            Successfully pulled image "httpd" in 5.796s (5.796s including waiting)
  Normal  Created    25m   kubelet            Created container httpd
  Normal  Started    25m   kubelet            Started container httpd

Inspecting Cluster Nodes

To get a high-level overview of the nodes in your cluster, use the following command:

kubectl get nodes

To get a quick overview of all nodes:

NAME             STATUS   ROLES           AGE   VERSION
docker-desktop   Ready    control-plane   31m   v1.29.2

For more detailed information about a specific node, use:

kubectl get node/docker-desktop -o wide

This provides additional details such as internal and external IP addresses, OS image, kernel version, and container runtime.

NAME             STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION    CONTAINER-RUNTIME
docker-desktop   Ready    control-plane   32m   v1.29.2   192.168.65.3   <none>        Docker Desktop   6.6.31-linuxkit   docker://26.1.4

To describe a node in detail, including its conditions, allocated resources, and events, use:

kubectl describe node/docker-desktop

This command provides a comprehensive overview of the node’s status and configuration.

Real-time Resource Monitoring with Kubectl Watch

While point-in-time status checks are useful, real-time monitoring can be useful for tracking changes as they occur. The kubectl get command can be combined with the -w or --watch flag to watch resources in real-time.

Using `kubectl get pods -w`

The kubectl get pods -w command allows you to observe changes to your pods in real-time. This is particularly useful when you’re making changes to your deployments or when you want to monitor the health of your pods.

kubectl get pods -w

When you run this command, you’ll see a live feed of your pods’ statuses. Let’s see what happens when we delete a pod:

Now, let’s delete one of the pods:

kubectl delete pod/my-apache-5bd7979764-gpdtl

You should see the pod being terminated and a new pod being created in real-time.

NAME                         READY   STATUS    RESTARTS        AGE
my-apache-5bd7979764-55cbx   1/1     Running   1 (7m53s ago)   24h
my-apache-5bd7979764-df2tr   1/1     Running   1 (7m53s ago)   24h
my-apache-5bd7979764-gpdtl   1/1     Running   1 (7m53s ago)   24h
my-apache-5bd7979764-gpdtl   1/1     Terminating   1 (8m44s ago)   24h
my-apache-5bd7979764-jzm4z   0/1     Pending       0               0s
my-apache-5bd7979764-jzm4z   0/1     Pending       0               0s
my-apache-5bd7979764-jzm4z   0/1     ContainerCreating   0               0s
my-apache-5bd7979764-gpdtl   0/1     Terminating         1 (8m46s ago)   24h
my-apache-5bd7979764-jzm4z   1/1     Running             0               2s
my-apache-5bd7979764-gpdtl   0/1     Terminating         1 (8m47s ago)   24h
my-apache-5bd7979764-gpdtl   0/1     Terminating         1 (8m47s ago)   24h
my-apache-5bd7979764-gpdtl   0/1     Terminating         1 (8m47s ago)   24h

As you can observe, Kubernetes immediately starts the process of terminating the deleted pod and creating a new one to maintain the desired state.

Watching Only New Events

If you’re only interested in new events without seeing the initial state, you can use the --watch-only flag. This is particularly useful when you want to focus on changes.

kubectl get events --watch-only

This command will show you real-time events as they occur in your cluster, such as pod creation, deletion, or any issues that arise.

kubectl delete pod/my-apache-5bd7979764-jzm4z

You should see the event of the pod being deleted in real-time.

LAST SEEN   TYPE     REASON    OBJECT                           MESSAGE
0s          Normal   Killing   pod/my-apache-5bd7979764-jzm4z   Stopping container httpd
0s          Normal   SuccessfulCreate   replicaset/my-apache-5bd7979764   Created pod: my-apache-5bd7979764-2zfrr
0s          Normal   Scheduled          pod/my-apache-5bd7979764-2zfrr    Successfully assigned default/my-apache-5bd7979764-2zfrr to docker-desktop
0s          Normal   Pulling            pod/my-apache-5bd7979764-2zfrr    Pulling image "httpd"
0s          Normal   Pulled             pod/my-apache-5bd7979764-2zfrr    Successfully pulled image "httpd" in 1.734s (1.734s including waiting)
0s          Normal   Created            pod/my-apache-5bd7979764-2zfrr    Created container httpd
0s          Normal   Started            pod/my-apache-5bd7979764-2zfrr    Started container httpd

Container Logs

It’s important to note that Kubernetes doesn’t store logs in its API or database server. Instead, logs are stored by default on each node. When you request logs, Kubernetes coordinates with the kubelet on each node to retrieve and present the logs to you.

kubectl logs deploy/my-apache

Following log entries in real-time

This command starts with the latest log entry and continues to stream new entries as they come in.

kubectl logs deploy deploy/my-apache --follow -tail 1

Accessing logs from a specific container in a pod

Use this when you need to target a specific container in a multi-container pod.

kubectl logs pod/my-apache-5bd7979764-df2tr -c httpd

Retrieving logs from all containers in a pod

This is useful when you need a comprehensive view of all activities within a pod.

kubectl logs pod/my-apache-5bd7979764-df2tr --all-containers=true

Getting logs from multiple pods using labels

This allows you to aggregate logs from all pods with a specific label, giving you a broader view of your application’s behavior.

kubectl logs -l app=my-apache

To effectively use label-based log retrieval, you need to know your pods’ labels. You can find these using:

kubectl get pods --show-labels

This command will show you the labels associated with each pod.

NAME                         READY   STATUS    RESTARTS      AGE   LABELS
my-apache-5bd7979764-2zfrr   1/1     Running   0             23m   app=my-apache,pod-template-hash=5bd7979764
my-apache-5bd7979764-55cbx   1/1     Running   1 (35m ago)   24h   app=my-apache,pod-template-hash=5bd7979764
my-apache-5bd7979764-df2tr   1/1     Running   1 (35m ago)   24h   app=my-apache,pod-template-hash=5bd7979764

This command displays all pods along with their associated labels, making it easy to construct targeted log queries.

Third-party Tools for Log Management

While kubectl logs is useful for basic log retrieval, third-party tools like Fluentd, Elasticsearch, and Kibana (EFK) or the ELK stack can provide more advanced log management capabilities, including log aggregation, search, and visualization.

Cleaning up

You can clean up the resources created in this guide using the following commands:

kubectl delete deployment my-apache