In today’s data-driven world, monitoring application performance and understanding user behavior are crucial for maintaining high availability, improving performance, and ensuring a great user experience. While traditional monitoring tools offer insights into infrastructure metrics like CPU and memory usage, gaining visibility into custom application metrics can provide a deeper understanding of how your application is truly performing.
This is where OpenTelemetry (OTEL) comes in. As an open-source observability framework, OpenTelemetry enables developers to instrument, generate, collect, and export telemetry data (traces, metrics, and logs) from their applications. By integrating OpenTelemetry, you can easily define and collect custom metrics tailored to your specific business needs.
In this blog, we’ll walk you through how to measure custom application metrics using OpenTelemetry and visualize them in Kibana. This combination not only helps in tracking application-specific metrics but also enhances your observability stack, making it easier to troubleshoot, optimize, and ensure the health of your applications.
Whether you’re looking to monitor response times, track API usage, or measure custom business metrics, this guide will equip you with the knowledge to harness the power of OpenTelemetry and Kibana to take your application’s observability to the next level.
Before diving into measuring custom application metrics with OpenTelemetry and visualizing them in Kibana, make sure you have the following components set up:
Elasticsearch and Kibana
To begin integrating OpenTelemetry with Kibana, you’ll need to set up the Elastic APM Server. This server collects and forwards application performance data to Elasticsearch, where it can be visualized in Kibana. Here’s how you can install and configure the APM Server on your Ubuntu server:
Update your package list:
First, update your package list to ensure you have the latest information on available packages:
sudo apt-get update
Install the APM Server:
Next, install the APM Server package using the apt
package manager:
sudo apt-get install apm-server
Configure the APM Server:
After installation, you need to configure the APM Server to suit your environment. Open the APM Server configuration file using a text editor:
sudo vi /etc/apm-server/apm-server.yml
Here’s an example configuration with placeholder values:
######################### APM Server Configuration #########################
################################ APM Server ################################
apm-server:
# Defines the host and port the server is listening on.
# Setting host to "0.0.0.0" allows APM Server to listen on all network interfaces.
host: "0.0.0.0:8200"
#---------------------------- APM Server - Secure Communication with Agents ----------------------------
# Enable SSL to secure communication between APM agents and the server.
# Configure paths to your SSL certificate and key files.
ssl:
enabled: true
certificate: /etc/letsencrypt/live/example.com/fullchain.pem
key: /etc/letsencrypt/live/example.com/privkey.pem
# Configure authorization via a secret token.
# Agents include this token in the Authorization header: Authorization: Bearer .
secret_token: 'your-secret-token-here'
#================================ Outputs =================================
# Configure the output to use when sending the data collected by APM Server.
# Output data to Elasticsearch.
output.elasticsearch:
# Array of hosts to connect to. Modify according to your Elasticsearch setup.
hosts: ["http://localhost:9200"]
username: "elastic" # Elasticsearch username
password: "your-elasticsearch-password" # Elasticsearch password
#================================= Logging =================================
# Configure logging output and level.
# You can choose between syslog, file, or stderr output.
logging.level: debug # Set the logging level to debug for detailed logs
# Enable logging to rotating files.
logging.to_files: true
logging.files:
# Configure the path where logs are written.
path: /var/log/apm-server
# Name of the log files.
name: apm-server
# Configure log file size limit. When this limit is reached, log files are rotated.
rotateeverybytes: 10485760 # 10MB file size limit
# Number of rotated log files to keep.
keepfiles: 7
# Permissions mask for rotated log files.
permissions: 0640
Configuration Breakdown:
host
setting defines the network address and port on which the APM Server listens. Setting it to 0.0.0.0
allows connections from any network interface.certificate
and key
fields should point to your SSL certificate and key files.'your-secret-token-here'
with a strong, unique token.hosts
, username
, and password
fields to match your Elasticsearch setup.debug
for detailed logs and configure file-based logging with rotation and retention settings.
Start and Enable the APM Server Service:
Next, install the APM Server package using the apt
package manager:
sudo systemctl start apm-server
Enable it to start on boot:
Next, install the APM Server package using the apt
package manager:
sudo systemctl enable apm-server
Verify the Installation:
Check the status of the APM Server to ensure it is running properly:
sudo systemctl status apm-server
You should see output indicating that the APM Server is active and running. You can also inspect logs for any errors:Check the status of the APM Server to ensure it is running properly:
sudo journalctl -u apm-server
With the APM Server set up, you’re now ready to configure the OpenTelemetry Collector to send data to it, and eventually visualize this data in Kibana. This setup provides a comprehensive observability solution that integrates application performance metrics seamlessly into your existing Elasticsearch and Kibana stack.
Once the APM Server is up and running, the next step is to instrument your application to collect telemetry data such as traces, metrics, and logs. By using OpenTelemetry, you can achieve a seamless integration with the Elastic APM Server, enhancing your observability stack.
1. Configuring Metrics and OpenTelemetry in Node.js:
In your bin/www
file, metrics and observability are configured using Elastic APM and OpenTelemetry. Below, we’ll focus on that configuration and explain its purpose:
Elastic APM Configuration:
Elastic APM is used to monitor the performance of your application. It provides insights into request latency, errors, and other key performance indicators. The configuration in the bin/www
file includes:
if (config.metrics.serverUrl && config.metrics.secretToken) {
logger.info(`Metrics are enabled (serverUrl=${config.metrics.serverUrl}).`);
require('elastic-apm-node').start({
serverUrl: config.metrics.serverUrl,
secretToken: config.metrics.secretToken,
serviceName: config.metrics.serviceName,
opentelemetryBridgeEnabled: true
});
} else {
logger.info('Metrics are not enabled.');
}
Key Configuration Details:
serverUrl
: The URL of the APM server where data will be sent.secretToken
: A token used to authenticate with the APM server.serviceName
: A unique name for your service, which helps in identifying data related to this specific service.opentelemetryBridgeEnabled
: Enables the bridge to OpenTelemetry, allowing you to use OpenTelemetry instrumentation alongside Elastic APM.
OpenTelemetry Configuration:
OpenTelemetry is a set of APIs and tools for collecting, processing, and exporting telemetry data like traces and metrics. In your application, it’s used alongside Elastic APM to provide more detailed insights.
const otel = require('@opentelemetry/api');
const customMeter = otel.metrics.getMeter('test-meter');
const Metrics = require('../src/metricsV2/metrics').Metrics;
const metrics = new Metrics(customMeter, logger);
Key Configuration Details:
Meter
: A part of OpenTelemetry that allows you to create and record custom metrics.getMeter('test-meter')
: Retrieves or creates a new meter with the name 'test-meter'
.Metrics
Class: A custom class (from your application’s metricsV2/metrics
module) that likely handles the creation, management, and recording of metrics using the provided meter and logger.
The bin/www
file in your Node.js application is a critical component that initializes and configures the server, manages WebSocket connections, and integrates performance monitoring through Elastic APM and OpenTelemetry. By setting up these monitoring tools, you gain valuable insights into your application’s performance and can proactively address issues as they arise.
2. Define Metrics Configuration:
Add the metrics configuration section to your config.json
file. This section will include settings for the metrics server and OpenTelemetry bridge.
"metrics": {
"serverUrl": http://apm.example.com:8200, // URL for the metrics server
"secretToken": "your-secret-token", // Secret token for authentication with the metrics server
"serviceName": "YourServiceName", // Name of your service for metrics tracking
"opentelemetryBridgeEnabled": true // Enable OpenTelemetry bridge
}
Key Configuration Details:
serverUrl
: Specifies the URL where metrics data should be sent. Replace http://metrics.example.com
with the endpoint of your metrics service.secretToken
: The token used for authenticating requests to the metrics server. Ensure this is kept secure.serviceName
: A descriptive name for your service, which will be used to identify metrics data in your APM tool.opentelemetryBridgeEnabled
: Set to true
to enable OpenTelemetry integration, which facilitates observability and performance monitoring.3. Add Required Dependencies:
Incorporating Application Performance Monitoring (APM) and OpenTelemetry metrics into your Node.js application enhances observability and helps in identifying performance bottlenecks. This guide will demonstrate how to configure these tools in your package.json
file.
To begin, ensure that you have the necessary dependencies for OpenTelemetry and APM integrated into your project. For this example, we’ll use the elastic-apm-node
for APM and OpenTelemetry packages.
"dependencies": {
"@opentelemetry/api": "^1.4.1",
"@opentelemetry/sdk-metrics": "^1.15.1",
"elastic-apm-node": "^3.48.0",
// Other dependencies...
}
Key Configuration Details:
@opentelemetry/api
: Provides the core OpenTelemetry API for instrumenting your application.@opentelemetry/sdk-metrics
: SDK for collecting and exporting metrics.elastic-apm-node
: Elastic APM agent for monitoring Node.js applications.4. Initialize OpenTelemetry Metrics:
Now, you will create a new OpenTelemetry MeterProvider and use it in your Metrics class. Here’s a step-by-step implementation guide:
In the metrics.js
file, set up OpenTelemetry’s MeterProvider
and update the Metrics
class to use it:
// Import necessary OpenTelemetry packages
const { MeterProvider } = require('@opentelemetry/sdk-metrics');
// Create a MeterProvider instance
const meterProvider = new MeterProvider();
const meter = meterProvider.getMeter('example-meter');
// Updated Metrics class with OpenTelemetry meter
class Metrics {
constructor(meter, logger) {
this.meter = meter; // Use OpenTelemetry meter
this.logger = logger;
this.tradierTradingPanelMetrics = this.createTradierTradingPanelMetrics();
this.healthMetrics = this.createHealthMetrics();
this.wheelMetrics = this.createWheelMetrics();
this.tradierApiMetrics = this.createTradierApiMetrics();
}
createBullCallSpreadMetrics() {
const matchCounter = this.meter.createCounter('bull_call_spread_match_count', {
description: 'Number of matches'
});
return {
matchCounter
};
}
createMarketIndexRunnerMetrics({ metricsPrefix }) {
const runsCounter = this.meter.createCounter(`${metricsPrefix}_total_runs_count`, {
description: 'Number of total runs'
});
const notificationsSentCounter = this.meter.createCounter(`${metricsPrefix}_notifications_sent_count`, {
description: 'Number of notifications sent'
});
return {
runsCounter,
notificationsSentCounter
};
}
// Add similar updates to other metric creation methods...
createTradierTradingPanelMetrics() {
const generateAccessTokenSuccessCounter = this.meter.createCounter('tradier_trading_panel_generate_access_token_success_counter', {
description: 'Tradier Trading Panel generate access token success counter'
});
const generateAccessTokenFailureCounter = this.meter.createCounter('tradier_trading_panel_generate_access_token_failure_counter', {
description: 'Tradier Trading Panel generate access token failure counter'
});
// Additional counters...
return {
generateAccessTokenSuccessCounter,
generateAccessTokenFailureCounter,
// Include other counters as defined
};
}
// Define other metric groups similar to the existing ones, using the OpenTelemetry meter instance
createHealthMetrics() {
const databaseHealthSuccessCounter = this.meter.createCounter('health_database_success_counter', {
description: 'Instance database health success counter'
});
const databaseHealthFailureCounter = this.meter.createCounter('health_database_failure_counter', {
description: 'Instance database health failure counter'
});
// Additional counters...
return {
databaseHealthSuccessCounter,
databaseHealthFailureCounter,
// Include other counters as needed
};
}
// Continue with other metric setup methods...
}
// Export the metrics module
module.exports = Metrics;
pxoMeter
Access Kibana:
http://localhost:5601
).Create an Index Pattern:
apm-*
.@timestamp
or another appropriate field) and complete the index pattern setup.Navigate to Visualize:
Create a New Visualization:
Add Your Metric Fields:
apm-*
or any other relevant index pattern.Choose Your Visualization Type:
Customize Your Visualization:
Save the Visualization:
Create a Dashboard (Optional):
Measuring custom application metrics using OpenTelemetry (OTEL) and visualizing the data in Kibana provides a robust solution for monitoring your application’s performance. By leveraging OTEL, you can collect detailed metrics from your application, regardless of the technology stack, and send them to Kibana for real-time analysis. This integration empowers you to gain valuable insights, troubleshoot issues more effectively, and optimize your application’s behavior based on precise data.
With these tools, you can set up comprehensive observability, customize your visualizations, and create dashboards that highlight the most critical metrics for your business needs. Start exploring OTEL and Kibana today to take your application monitoring to the next level. If you have any questions or need further guidance, don’t hesitate to reach out. Happy monitoring!