|
1 | | -# eBPF for Observability |
| 1 | +## 📊 What is OpenTelemetry? |
| 2 | +- OpenTelemetry is an open-source observability framework for generating, collecting, and exporting telemetry data (traces, metrics, logs) to help monitor applications. |
| 3 | + |
| 4 | +## 🛠️ How is it Different from Other Libraries? |
| 5 | +- OpenTelemetry offers a unified standard for observability across multiple tools and vendors, unlike other libraries that may focus only on a specific aspect like tracing or metrics. |
| 6 | + |
| 7 | +## ⏳ What Existed Before OpenTelemetry? |
| 8 | +- Before OpenTelemetry, observability was typically managed using a combination of specialized tools for different aspects like |
| 9 | + - `Tracing`: Tools like Jaeger and Zipkin were used to track requests |
| 10 | + - `Metrics`: Solutions like Prometheus and StatsD were popular for collecting metrics |
| 11 | + - `Logging`: Tools like ELK Stack (Elasticsearch, Logstash, Kibana) or Fluentd were used to aggregate and analyze logs. |
| 12 | +- OpenTelemetry unified these by standardizing how telemetry data is collected and exported. |
| 13 | +- Prior to OpenTelemetry, there were OpenTracing and OpenCensus, which OpenTelemetry merged to provide a more comprehensive and standardized observability solution. |
| 14 | + |
| 15 | +## 🌐 Supported Programming Languages |
| 16 | + |
| 17 | +OpenTelemetry supports several languages, including: |
| 18 | + |
| 19 | +- **Go** |
| 20 | +- **Java** |
| 21 | +- **JavaScript** |
| 22 | +- **Python** |
| 23 | +- **C#** |
| 24 | +- **C++** |
| 25 | +- **Ruby** |
| 26 | +- **PHP** |
| 27 | +- **Swift** |
| 28 | +- ...and others. |
| 29 | + |
| 30 | +## Architecture |
| 31 | + |
| 32 | +### 🖥️ Step 1: Create EKS Cluster |
| 33 | + |
| 34 | +```bash |
| 35 | +eksctl create cluster --name=observability \ |
| 36 | + --region=us-east-1 \ |
| 37 | + --zones=us-east-1a,us-east-1b \ |
| 38 | + --without-nodegroup |
| 39 | +``` |
| 40 | +```bash |
| 41 | +eksctl utils associate-iam-oidc-provider \ |
| 42 | + --region us-east-1 \ |
| 43 | + --cluster observability \ |
| 44 | + --approve |
| 45 | +``` |
| 46 | +```bash |
| 47 | +eksctl create nodegroup --cluster=observability \ |
| 48 | + --region=us-east-1 \ |
| 49 | + --name=observability-ng-private \ |
| 50 | + --node-type=t3.medium \ |
| 51 | + --nodes-min=2 \ |
| 52 | + --nodes-max=3 \ |
| 53 | + --node-volume-size=20 \ |
| 54 | + --managed \ |
| 55 | + --asg-access \ |
| 56 | + --external-dns-access \ |
| 57 | + --full-ecr-access \ |
| 58 | + --appmesh-access \ |
| 59 | + --alb-ingress-access \ |
| 60 | + --node-private-networking |
| 61 | + |
| 62 | +# Update ./kube/config file |
| 63 | +aws eks update-kubeconfig --name observability |
| 64 | +``` |
| 65 | + |
| 66 | +### 🔐 Step 2: Create IAM Role for Service Account |
| 67 | +```bash |
| 68 | +eksctl create iamserviceaccount \ |
| 69 | + --name ebs-csi-controller-sa \ |
| 70 | + --namespace kube-system \ |
| 71 | + --cluster observability \ |
| 72 | + --role-name AmazonEKS_EBS_CSI_DriverRole \ |
| 73 | + --role-only \ |
| 74 | + --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \ |
| 75 | + --approve |
| 76 | +``` |
| 77 | +- This command creates an IAM role for the EBS CSI controller. |
| 78 | +- IAM role allows EBS CSI controller to interact with AWS resources, specifically for managing EBS volumes in the Kubernetes cluster. |
| 79 | +- We will attach the Role with service account |
| 80 | + |
| 81 | +### 📝 Step 3: Retrieve IAM Role ARN |
| 82 | +```bash |
| 83 | +ARN=$(aws iam get-role --role-name AmazonEKS_EBS_CSI_DriverRole --query 'Role.Arn' --output text) |
| 84 | +``` |
| 85 | +- Command retrieves the ARN of the IAM role created for the EBS CSI controller service account. |
| 86 | + |
| 87 | +### 📦 Step 4: Deploy EBS CSI Driver |
| 88 | +```bash |
| 89 | +eksctl create addon --cluster observability --name aws-ebs-csi-driver --version latest \ |
| 90 | + --service-account-role-arn $ARN --force |
| 91 | +``` |
| 92 | +- Above command deploys the AWS EBS CSI driver as an addon to your Kubernetes cluster. |
| 93 | +- It uses the previously created IAM service account role to allow the driver to manage EBS volumes securely. |
| 94 | + |
| 95 | + |
| 96 | +### 🧩 Step 5: Understand the Application |
| 97 | +- We have two very simple microservice A (`microservice-a`) & B (`microservice-a`), Built with Golang using the Gin web framework for handling HTTP requests. |
| 98 | +- **Microservice A** API Endpoints: |
| 99 | + - `GET /hello-a` – Returns a greeting message |
| 100 | + - `GET /call-b` – Calls another service (Service B) and returns its response |
| 101 | + - `GET /getme-coffee` – Fetches and returns data from an external coffee API |
| 102 | +- **Microservice B** API Endpoints: |
| 103 | + - `GET /hello-b` – Returns a greeting message |
| 104 | + - `GET /call-a` – Calls another service (Service A) and returns its response |
| 105 | + - `GET /getme-coffee` – Fetches and returns data from an external coffee API |
| 106 | +- Observability: |
| 107 | + - OpenTelemetry SDK integrated for tracing and metrics. |
| 108 | + - Metrics and traces are exported to the OpenTelemetry Collector via OTLP over HTTP. |
| 109 | +- Instrumentation: |
| 110 | + - Uses OpenTelemetry middleware (otelgin) for automatic request tracing. |
| 111 | + - Instruments HTTP clients with otelhttp for distributed tracing of outbound requests. |
| 112 | + |
| 113 | + |
| 114 | +### 🐳 Step 6: Dockerize & push it to the registry |
| 115 | +```bash |
| 116 | +# Dockerize microservice - a |
| 117 | +docker build -t <<NAME_OF_YOUR_REPO>>:<<TAG>> microservice-a/ |
| 118 | +
|
| 119 | +# Dockerize microservice - b |
| 120 | +docker build -t <<NAME_OF_YOUR_REPO>>:<<TAG>> microservice-b/ |
| 121 | +
|
| 122 | +# push both images |
| 123 | +docker push <<NAME_OF_YOUR_REPO>>:<<TAG>> |
| 124 | +docker push <<NAME_OF_YOUR_REPO>>:<<TAG>> |
| 125 | +``` |
| 126 | +
|
| 127 | +
|
| 128 | +### 🗂️ Step 7: Create Namespace for observability components |
| 129 | +```bash |
| 130 | +kubectl create namespace olly |
| 131 | +``` |
| 132 | +
|
| 133 | +### 📚 Step 8: Install Elasticsearch on K8s |
| 134 | +helm repo add elastic https://helm.elastic.co |
| 135 | +
|
| 136 | +helm install elasticsearch \ |
| 137 | + --set replicas=1 \ |
| 138 | + --set volumeClaimTemplate.storageClassName=gp2 \ |
| 139 | + --set persistence.labels.enabled=true elastic/elasticsearch -n olly |
| 140 | +
|
| 141 | +
|
| 142 | +### 📜 Step 9: Export Elasticsearch CA Certificate |
| 143 | +- This command retrieves the CA certificate from the Elasticsearch master certificate secret and decodes it, saving it to a ca-cert.pem file. |
| 144 | +```bash |
| 145 | +kubectl get secret elasticsearch-master-certs -n olly -o jsonpath='{.data.ca\.crt}' | base64 --decode > ca-cert.pem |
| 146 | +``` |
| 147 | +
|
| 148 | +### 🔑 Step 10: Create ConfigMap for Jaeger's TLS Certificate |
| 149 | +- Creates a ConfigMap in the olly namespace, containing the CA certificate to be used by Jaeger for TLS. |
| 150 | +```bash |
| 151 | +kubectl create configmap jaeger-tls --from-file=ca-cert.pem -n olly |
| 152 | +``` |
| 153 | +
|
| 154 | +### 🛡️ Step 11: Create Secret for Elasticsearch TLS |
| 155 | +- Creates a Kubernetes Secret in the tracing namespace, containing the CA certificate for Elasticsearch TLS communication. |
| 156 | +```bash |
| 157 | +kubectl create secret generic es-tls-secret --from-file=ca-cert.pem -n olly |
| 158 | +``` |
| 159 | +
|
| 160 | +### 🔍 Step 12: Retrieve Elasticsearch Username & Password |
| 161 | +```bash |
| 162 | +# for username |
| 163 | +kubectl get secrets --namespace=olly elasticsearch-master-credentials -ojsonpath='{.data.username}' | base64 -d |
| 164 | +# for password |
| 165 | +kubectl get secrets --namespace=olly elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -d |
| 166 | +``` |
| 167 | +- Retrieves the password for the Elasticsearch cluster's master credentials from the Kubernetes secret. |
| 168 | +- 👉 **Note**: Please write down the password for future reference |
| 169 | +
|
| 170 | +
|
| 171 | +### 🕵️♂️ Step 13: Install Jaeger with Custom Values |
| 172 | +- 👉 **Note**: Please update the `password` field and other related field in the `jaeger-values.yaml` file with the password retrieved previous step at step 12: (i.e NJyO47UqeYBsoaEU)" |
| 173 | +- Command installs Jaeger into the olly namespace using a custom jaeger-values.yaml configuration file. Ensure the password is updated in the file before installation. |
| 174 | +```bash |
| 175 | +helm repo add jaegertracing https://jaegertracing.github.io/helm-charts |
| 176 | +helm repo update |
| 177 | +
|
| 178 | +helm install jaeger jaegertracing/jaeger -n olly --values jaeger-values.yaml |
| 179 | +``` |
| 180 | +
|
| 181 | +### 🌐 Step 14: Access UI - Port Forward Jaeger Query Service |
| 182 | +kubectl port-forward svc/jaeger-query 8080:80 -n olly |
| 183 | +
|
| 184 | +
|
| 185 | +
|
| 186 | +### 📈 Step 15: Install Opentelemetry-collector |
| 187 | +helm install otel-collector open-telemetry/opentelemetry-collector -n olly --values otel-collector-values.yaml |
| 188 | +
|
| 189 | +
|
| 190 | +### 📊 Step 16: Install prometheus |
| 191 | +```bash |
| 192 | +helm repo add prometheus-community https://prometheus-community.github.io/helm-charts |
| 193 | +helm repo update |
| 194 | +
|
| 195 | +helm install prometheus prometheus-community/prometheus -n olly --values prometheus-values.yaml |
| 196 | +``` |
| 197 | +
|
| 198 | +### 🚀 Step 17: Deploy the applicaiton |
| 199 | +- ***Note:*** - Review the Kubernetes manifest files located in `./k8s-manifest`. and you should change image name & tag with your own image |
| 200 | +```bash |
| 201 | +kubectl apply -k k8s-manifests/ |
| 202 | +``` |
| 203 | +- 👉 ***Note***: wait for 5 minutes till you load balancer comes in running state |
| 204 | +
|
| 205 | +## 🔄 Step 18: Generate Load |
| 206 | +- Script: `test.sh` takes two load balancer DNS addresses as input arguments and alternates requests between them using curl. |
| 207 | +- `test.sh` Continuously sends random HTTP requests every second to predefined routes on two provided load balancer DNSs |
| 208 | +- ***Note:*** Keep the script running in another terminal to quickly gather metrics & traces. |
| 209 | +
|
| 210 | +```bash |
| 211 | +./test.sh http://Microservice_A_LOAD_BALANCER_DNS http://Microservice_B_LOAD_BALANCER_DNS |
| 212 | +``` |
| 213 | +
|
| 214 | +### 📊 Step 19: Access the UI of Prometheus |
| 215 | +```bash |
| 216 | +kubectl port-forward svc/prometheus-server 9090:80 -n olly |
| 217 | +``` |
| 218 | +- Look for your application's metrics like `request_count`, `request_duration_ms`, `active_requests` and other to monitor request rates & performance. |
| 219 | +
|
| 220 | +
|
| 221 | +### 🕵️♂️ Step 20: Access the UI of Jaeger |
| 222 | +```bash |
| 223 | +kubectl port-forward svc/jaeger-query 8080:80 -n olly |
| 224 | +``` |
| 225 | +- Look for traces from the service name microservice-a, microservice-b and operations such as `[/hello-a, /call-b, and /getme-coffee]` or `[/hello-b, /call-a, and /getme-coffee]` to monitor request flows and dependencies. |
| 226 | +
|
| 227 | +## ✅ Conclusion |
| 228 | +- By following the above steps, you have successfully set up an observability stack using OpenTelemetry on an EKS cluster. This setup allows you to monitor your microservices effectively through integrated tracing, metrics, and logging. |
| 229 | +
|
| 230 | +## 🧼 Clean Up |
| 231 | +```bash |
| 232 | +helm uninstall prometheus -n olly |
| 233 | +helm uninstall otel-collector -n olly |
| 234 | +helm uninstall jaeger -n olly |
| 235 | +helm uninstall elasticsearch -n olly |
| 236 | +
|
| 237 | +<!-- Delete all the pvc & pv --> |
| 238 | +
|
| 239 | +kubectl delete -k k8s-manifests/ |
| 240 | +
|
| 241 | +
|
| 242 | +kubectl delete ns olly |
| 243 | +
|
| 244 | +eksctl delete cluster --name observability |
| 245 | +``` |
0 commit comments