Step by Step detailed guide to setup Apache Skywalking on kubernetes

This guide will be covering the setup of following components of skywalking version 8.5.0 on kubernetes.

1. Backend
2. Java Agent
3. K8's monitoring

SkyWalking is an open source observability platform used to collect, analyze, aggregate and visualize data from services and cloud native infrastructures. SkyWalking provides an easy way to maintain a clear view of your distributed systems, even across Clouds. It is a modern APM, specially designed for cloud native, container based distributed systems.

You can read more about the detailed concepts and design here.

Requirements

1. Access to working Kubernetes Cluster
2. Helm Charts
3. Working Java Application JDK 8–14(spring boot web)

Install Backend

Installing backend requires a storage to store its data and following are the supported storages. I’m using Elasticsearch in this guide.

• H2
• OpenSearch
• ElasticSearch 6, 7
• MySQL
• TiDB
• InfluxDB
• PostgreSQL

You can refer this link for more detailed guide on backend storage.

Clone the github repo https://github.com/apache/skywalking-kubernetes

$ git clone https://github.com/apache/skywalking-kubernetes
$ cd skywalking-kubernetes/chart

If you want to use the elasticsearch as storage you two options to choose

1. skywalking helm chart comes with elasticsearch installation.
2. you can also point to the skywalking configuration to existing running supported elasticsearch.

Run the following command the add and download the elastic helm charts

$ helm repo add elastic https://helm.elastic.co
$ helm dep up skywalking

Export the following variables

$ export SKYWALKING_RELEASE_NAME=skywalking 
$ export SKYWALKING_RELEASE_NAMESPACE=skywalking

Run the following command to Install Skywalking on kubernetes

$ helm install "${SKYWALKING_RELEASE_NAME}" skywalking -n "${SKYWALKING_RELEASE_NAMESPACE}" \
  --set oap.image.tag=8.5.0-es7 \
  --set oap.storageType=elasticsearch7 \
  --set ui.image.tag=8.5.0 \
  --set elasticsearch.imageTag=7.5.1 \
--set elasticsearch.persistence.enabled=true

This following option will create the statefulset elasticsearch

--set elasticsearch.persistence.enabled=true

This option is to create the elasticsearch indexes name with given pattern

--set oap.env.SW_NAMESPACE=skywalking-dev

If you want to point skywalking to an existing running elasticsearch cluster. Edit the following file and skywalking-kubernetes/chart/skywalking/values-my-es.yaml

update the required configuration accordingly.

oap:
  image:
    tag: 8.5.0-es7      # Set the right tag according to the existing Elasticsearch version
  storageType: elasticsearch7
ui:
  image:
    tag: 8.5.0
elasticsearch:
  enabled: false
  config:               # For users of an existing elasticsearch cluster,takes effect when `elasticsearch.enabled` is false
    host: your.elasticsearch.host.or.ip
    port:
      http: 9200
    user: "xxx"         # [optional]
    password: "xxx"     # [optional]

and run the following command to install skywalking backend.

$ helm install "${SKYWALKING_RELEASE_NAME}" skywalking -n "${SKYWALKING_RELEASE_NAMESPACE}" -f ./skywalking/values-my-es.yaml

If the command is successful. you should see the following pods running.

➜ kubectl get pods -n skywalking
NAME                              READY   STATUS      RESTARTS   AGE
elasticsearch-master-0            1/1     Running     0          26h
elasticsearch-master-1            1/1     Running     0          26h
elasticsearch-master-2            1/1     Running     0          26h
skywalking-es-init-fccpx          0/1     Completed   0          26h
skywalking-oap-684d845ff8-bm5f7   1/1     Running     0          24h
skywalking-oap-684d845ff8-jkhkv   1/1     Running     0          24h
skywalking-ui-59b75cc6b8-56958    1/1     Running     0          26h

From here you can access the Skywalking UI by doing a port-forward.

➜ kubectl port-forward svc/skywalking-ui 8080:80 -n skywalking
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
Handling connection for 8080
Handling connection for 8080

Launch any browser and open http://localhost:8080 you should be able to see the UI loading.

Setup Java Agent

Skywalking support many middleware servers, frameworks and libraries. In this example I will be using spring boot web application to integrate java agent.

Data Flow

Use this link to read more about the supported list

Download the java agent from here

Apache Download Mirrors

Extract the tar file, Copy the required jar files from optional-plugins and bootstrap-plugins folder to plugins folder.

Build your java application(spring boot web) to generate a jar file using mvn command Maven

Following is the Dockerfile

FROM adoptopenjdk:11-jre-hotspot

# copy extracted agent folder from the downloaded archive
ADD agent /opt/agent
# copy the jar file
COPY target/*.jar /app/application.jar 
WORKDIR /app
ENTRYPOINT ["java","-javaagent:/opt/agent/skywalking-agent.jar=agent.namespace=default,agent.service_name=example-spring-boot-app,collector.backend_service=skywalking-oap.skywalking.svc.cluster.local:11800,plugin.jdbc.trace_sql_parameters=true,profile.active=true","-jar","/app/application.jar"]

You can configure you agent in various ways

agent.service_name # name of your application to show on UI
-javaagent:/opt/agent/skywalking-agent.jar path to your agent jar file
agent.namespace=default # kubernetes namespace where application is running
collector.backend_service=skywalking-oap.skywalking.svc.cluster.local:11800 # address of skywalking oap backend
plugin.jdbc.trace_sql_parameters=true # sql prepared statement will be collected
profile.active=true # enable skywalking agent profiling
logging.dir # path where the agent logs will be stored. default is the agent directory

Following link contains more details on configuring the agent

Setup java agent

Once you build the Dockerfile and deploy it to kubernetes. The agent should start collecting metrics and post it to the backend and the service will be available in the Skywalking UI.

Troubleshooting

Agent logs are located in agent directory /logs.

If the agent fails to connect to backend(oap). Make sure the oap is reachable on 11800 port.

Incase if you face any issues with agent started and having any issues with application you need to remove the bootstrap-plugins or the optional plugins

K8s Monitoring

SkyWalking leverages K8s kube-state-metrics and cAdvisor for collecting metrics data from the K8s, and leverages OpenTelemetry Collector to transfer the metrics to OpenTelemetry receiver and into the Meter System. This feature requires authorizing the OAP Server to access K8s’s API Server.
We defined the k8s-cluster as a Service in OAP, use k8s-cluster:: as a prefix to identify.
Defined the k8s-node as an Instance in OAP, the name is k8s node name.
Defined the k8s-service as an Endpoint in OAP, the name is $serviceName.$namespace.

Steps to configure K8s Monitoring in Skywalking

OpenTelemetry fetches data from kube-state-metrics and exports it to skywalking OAP

Data Flow

Install Kube-state-metrics

Run the following commands to install the kube-state-metrics helm chart.

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
$ helm install [RELEASE_NAME] prometheus-community/kube-state-metrics

Install OpenTelemetry collector

This example manifests is provided by Apache Skywalking for quick start.

Download the manifest file otel-collector-config.yaml

$ wget https://skywalking.apache.org/docs/main/v8.5.0/en/setup/backend/otel-collector-config.yaml

You need to update the scrape job config accordingly and also you need to update the exporters accordingly

exporters:
      opencensus:
        endpoint: "skywalking-oap.skywalking.svc.cluster.local:11800" # The OAP Server address
        insecure: true

Then run the kubectl apply command

$ kubectl apply -f otel-collector-config.yaml

Configure Skywalking OpenTelemetry Receiver

Following are the options to configure skywalking OpenTelemetry Receiver.

Note: This Change will require OAP to have access to kubernetes api to query the metadata. This option ( — set oap.envoy.als.enabled=true) will create required clusterrole, clusterrolebinding

--set oap.env.SW_OTEL_RECEIVER=default --set oap.env.SW_OTEL_RECEIVER_ENABLED_OC_RULES="k8s-cluster\,k8s-service\,k8s-node" --set oap.envoy.als.enabled=true

Once this is done you should see the k8s metrics in Skywalking UI.

Tip: If you don’t see the k8s tab open in private session or clear browser cache.

Skywalking Demo:

http://demo.skywalking.apache.org/

User: skywalking Password: skywalking

Skywalking support many other agents for different programming languages.

Language agents in Service