TechAnek

In today’s data-driven world, monitoring application performance and understanding user behavior are crucial for maintaining high availability, improving performance, and ensuring a great user experience. While traditional monitoring tools offer insights into infrastructure metrics like CPU and memory usage, gaining visibility into custom application metrics can provide a deeper understanding of how your application is truly performing.

This is where OpenTelemetry (OTEL) comes in. As an open-source observability framework, OpenTelemetry enables developers to instrument, generate, collect, and export telemetry data (traces, metrics, and logs) from their applications. By integrating OpenTelemetry, you can easily define and collect custom metrics tailored to your specific business needs.

In this blog, we’ll walk you through how to measure custom application metrics using OpenTelemetry and visualize them in Kibana. This combination not only helps in tracking application-specific metrics but also enhances your observability stack, making it easier to troubleshoot, optimize, and ensure the health of your applications.

Whether you’re looking to monitor response times, track API usage, or measure custom business metrics, this guide will equip you with the knowledge to harness the power of OpenTelemetry and Kibana to take your application’s observability to the next level.

Prerequisite:

Before diving into measuring custom application metrics with OpenTelemetry and visualizing them in Kibana, make sure you have the following components set up:

  1. Elasticsearch and Kibana

    • Elasticsearch: The heart of the ELK stack, Elasticsearch is where your data is stored, indexed, and searched. Ensure you have a working Elasticsearch cluster that your OpenTelemetry Collector can send metrics to.
    • Kibana: This is the visualization layer of the ELK stack. Kibana allows you to create dashboards, visualizations, and alerts based on the data ingested into Elasticsearch.

Step1: Install APM Server Using APT

To begin integrating OpenTelemetry with Kibana, you’ll need to set up the Elastic APM Server. This server collects and forwards application performance data to Elasticsearch, where it can be visualized in Kibana. Here’s how you can install and configure the APM Server on your Ubuntu server:

Update your package list:
First, update your package list to ensure you have the latest information on available packages:

				
					sudo apt-get update
				
			

Install the APM Server:
Next, install the APM Server package using the apt package manager:

				
					sudo apt-get install apm-server

				
			

Configure the APM Server:
After installation, you need to configure the APM Server to suit your environment. Open the APM Server configuration file using a text editor:

				
					sudo vi /etc/apm-server/apm-server.yml
				
			

Here’s an example configuration with placeholder values:

				
					######################### APM Server Configuration #########################

################################ APM Server ################################

apm-server:
  # Defines the host and port the server is listening on.
  # Setting host to "0.0.0.0" allows APM Server to listen on all network interfaces.
  host: "0.0.0.0:8200"

  #---------------------------- APM Server - Secure Communication with Agents ----------------------------
  # Enable SSL to secure communication between APM agents and the server.
  # Configure paths to your SSL certificate and key files.
  ssl:
    enabled: true
    certificate: /etc/letsencrypt/live/example.com/fullchain.pem
    key: /etc/letsencrypt/live/example.com/privkey.pem

  # Configure authorization via a secret token.
  # Agents include this token in the Authorization header: Authorization: Bearer <secret-token>.
  secret_token: 'your-secret-token-here'

#================================ Outputs =================================

# Configure the output to use when sending the data collected by APM Server.
# Output data to Elasticsearch.

output.elasticsearch:
  # Array of hosts to connect to. Modify according to your Elasticsearch setup.
  hosts: ["http://localhost:9200"]
  username: "elastic"  # Elasticsearch username
  password: "your-elasticsearch-password"  # Elasticsearch password

#================================= Logging =================================

# Configure logging output and level.
# You can choose between syslog, file, or stderr output.
logging.level: debug  # Set the logging level to debug for detailed logs

# Enable logging to rotating files.
logging.to_files: true
logging.files:
  # Configure the path where logs are written.
  path: /var/log/apm-server

  # Name of the log files.
  name: apm-server

  # Configure log file size limit. When this limit is reached, log files are rotated.
  rotateeverybytes: 10485760  # 10MB file size limit

  # Number of rotated log files to keep.
  keepfiles: 7

  # Permissions mask for rotated log files.
  permissions: 0640

				
			

Configuration Breakdown:

  • Host Configuration: The host setting defines the network address and port on which the APM Server listens. Setting it to 0.0.0.0 allows connections from any network interface.
  • SSL/TLS: SSL is enabled to ensure secure communication between APM agents and the server. The certificate and key fields should point to your SSL certificate and key files.
  • Secret Token: Used for securing APM server endpoints. Replace 'your-secret-token-here' with a strong, unique token.
  • Elasticsearch Output: Configure the hosts, username, and password fields to match your Elasticsearch setup.
  • Logging: Set the logging level to debug for detailed logs and configure file-based logging with rotation and retention settings.

 

Start and Enable the APM Server Service:

Next, install the APM Server package using the apt package manager:

				
					sudo systemctl start apm-server
				
			

Enable it to start on boot:

Next, install the APM Server package using the apt package manager:

				
					sudo systemctl enable apm-server
				
			

Verify the Installation:

Check the status of the APM Server to ensure it is running properly:

				
					sudo systemctl status apm-server

				
			

You should see output indicating that the APM Server is active and running. You can also inspect logs for any errors:Check the status of the APM Server to ensure it is running properly:

				
					sudo journalctl -u apm-server
				
			

With the APM Server set up, you’re now ready to configure the OpenTelemetry Collector to send data to it, and eventually visualize this data in Kibana. This setup provides a comprehensive observability solution that integrates application performance metrics seamlessly into your existing Elasticsearch and Kibana stack.

Step2: Instrument Your Application Using OpenTelemetry and Elastic APM

Once the APM Server is up and running, the next step is to instrument your application to collect telemetry data such as traces, metrics, and logs. By using OpenTelemetry, you can achieve a seamless integration with the Elastic APM Server, enhancing your observability stack.

1. Configuring Metrics and OpenTelemetry in Node.js:
In your bin/www file, metrics and observability are configured using Elastic APM and OpenTelemetry. Below, we’ll focus on that configuration and explain its purpose:

Elastic APM Configuration:

Elastic APM is used to monitor the performance of your application. It provides insights into request latency, errors, and other key performance indicators. The configuration in the bin/www file includes:

				
					if (config.metrics.serverUrl && config.metrics.secretToken) {
  logger.info(`Metrics are enabled (serverUrl=${config.metrics.serverUrl}).`);

  require('elastic-apm-node').start({
    serverUrl: config.metrics.serverUrl,
    secretToken: config.metrics.secretToken,
    serviceName: config.metrics.serviceName,
    opentelemetryBridgeEnabled: true
  });
} else {
  logger.info('Metrics are not enabled.');
}

				
			

Key Configuration Details:

  • serverUrl: The URL of the APM server where data will be sent.
  • secretToken: A token used to authenticate with the APM server.
  • serviceName: A unique name for your service, which helps in identifying data related to this specific service.
  • opentelemetryBridgeEnabled: Enables the bridge to OpenTelemetry, allowing you to use OpenTelemetry instrumentation alongside Elastic APM.


OpenTelemetry Configuration:
OpenTelemetry is a set of APIs and tools for collecting, processing, and exporting telemetry data like traces and metrics. In your application, it’s used alongside Elastic APM to provide more detailed insights.

				
					const otel = require('@opentelemetry/api');
const customMeter = otel.metrics.getMeter('test-meter');

const Metrics = require('../src/metricsV2/metrics').Metrics;
const metrics = new Metrics(customMeter, logger);
				
			

Key Configuration Details:

  • Meter: A part of OpenTelemetry that allows you to create and record custom metrics.
  • getMeter('test-meter'): Retrieves or creates a new meter with the name 'test-meter'.
  • Metrics Class: A custom class (from your application’s metricsV2/metrics module) that likely handles the creation, management, and recording of metrics using the provided meter and logger.


The bin/www file in your Node.js application is a critical component that initializes and configures the server, manages WebSocket connections, and integrates performance monitoring through Elastic APM and OpenTelemetry. By setting up these monitoring tools, you gain valuable insights into your application’s performance and can proactively address issues as they arise.

2. Define Metrics Configuration:
Add the metrics configuration section to your config.json file. This section will include settings for the metrics server and OpenTelemetry bridge.

				
					"metrics": {
  "serverUrl": http://apm.example.com:8200,  // URL for the metrics server
  "secretToken": "your-secret-token",         // Secret token for authentication with the metrics server
  "serviceName": "YourServiceName",           // Name of your service for metrics tracking
  "opentelemetryBridgeEnabled": true          // Enable OpenTelemetry bridge
}

				
			

Key Configuration Details:

  • serverUrl: Specifies the URL where metrics data should be sent. Replace http://metrics.example.com with the endpoint of your metrics service.
  • secretToken: The token used for authenticating requests to the metrics server. Ensure this is kept secure.
  • serviceName: A descriptive name for your service, which will be used to identify metrics data in your APM tool.
  • opentelemetryBridgeEnabled: Set to true to enable OpenTelemetry integration, which facilitates observability and performance monitoring.

3. Add Required Dependencies:
Incorporating Application Performance Monitoring (APM) and OpenTelemetry metrics into your Node.js application enhances observability and helps in identifying performance bottlenecks. This guide will demonstrate how to configure these tools in your package.json file.

To begin, ensure that you have the necessary dependencies for OpenTelemetry and APM integrated into your project. For this example, we’ll use the elastic-apm-node for APM and OpenTelemetry packages.

				
					"dependencies": {
  "@opentelemetry/api": "^1.4.1",
  "@opentelemetry/sdk-metrics": "^1.15.1",
  "elastic-apm-node": "^3.48.0",
  // Other dependencies...
}

				
			

Key Configuration Details:

  • @opentelemetry/api: Provides the core OpenTelemetry API for instrumenting your application.
  • @opentelemetry/sdk-metrics: SDK for collecting and exporting metrics.
  • elastic-apm-node: Elastic APM agent for monitoring Node.js applications.

4. Initialize OpenTelemetry Metrics:

Now, you will create a new OpenTelemetry MeterProvider and use it in your Metrics class. Here’s a step-by-step implementation guide:
In the metrics.js file, set up OpenTelemetry’s MeterProvider and update the Metrics class to use it:

				
					// Import necessary OpenTelemetry packages
const { MeterProvider } = require('@opentelemetry/sdk-metrics');

// Create a MeterProvider instance
const meterProvider = new MeterProvider();
const meter = meterProvider.getMeter('example-meter');

// Updated Metrics class with OpenTelemetry meter
class Metrics {
    constructor(meter, logger) {
        this.meter = meter;  // Use OpenTelemetry meter
        this.logger = logger;

        this.tradierTradingPanelMetrics = this.createTradierTradingPanelMetrics();
        this.healthMetrics = this.createHealthMetrics();
        this.wheelMetrics = this.createWheelMetrics();
        this.tradierApiMetrics = this.createTradierApiMetrics();
    }

    createBullCallSpreadMetrics() {
        const matchCounter = this.meter.createCounter('bull_call_spread_match_count', {
            description: 'Number of matches'
        });

        return {
            matchCounter
        };
    }

    createMarketIndexRunnerMetrics({ metricsPrefix }) {
        const runsCounter = this.meter.createCounter(`${metricsPrefix}_total_runs_count`, {
            description: 'Number of total runs'
        });

        const notificationsSentCounter = this.meter.createCounter(`${metricsPrefix}_notifications_sent_count`, {
            description: 'Number of notifications sent'
        });

        return {
            runsCounter,
            notificationsSentCounter
        };
    }

    // Add similar updates to other metric creation methods...

    createTradierTradingPanelMetrics() {
        const generateAccessTokenSuccessCounter = this.meter.createCounter('tradier_trading_panel_generate_access_token_success_counter', {
            description: 'Tradier Trading Panel generate access token success counter'
        });

        const generateAccessTokenFailureCounter = this.meter.createCounter('tradier_trading_panel_generate_access_token_failure_counter', {
            description: 'Tradier Trading Panel generate access token failure counter'
        });

        // Additional counters...

        return {
            generateAccessTokenSuccessCounter,
            generateAccessTokenFailureCounter,
            // Include other counters as defined
        };
    }

    // Define other metric groups similar to the existing ones, using the OpenTelemetry meter instance

    createHealthMetrics() {
        const databaseHealthSuccessCounter = this.meter.createCounter('health_database_success_counter', {
            description: 'Instance database health success counter'
        });

        const databaseHealthFailureCounter = this.meter.createCounter('health_database_failure_counter', {
            description: 'Instance database health failure counter'
        });

        // Additional counters...

        return {
            databaseHealthSuccessCounter,
            databaseHealthFailureCounter,
            // Include other counters as needed
        };
    }

    // Continue with other metric setup methods...
}

// Export the metrics module
module.exports = Metrics;
pxoMeter 
				
			

Step3: View Metrics in Kibana

Access Kibana:

  • Open your web browser and go to Kibana’s URL (e.g., http://localhost:5601).
 

Create an Index Pattern:

  • Navigate to Stack Management > Index Patterns.
  • Click on Create Index Pattern.
  • Enter the pattern that matches your metrics indices, such as apm-*.
  • Select the time filter field (@timestamp or another appropriate field) and complete the index pattern setup.
 

Navigate to Visualize:

  • Open Kibana and log in if necessary.
  • From the left menu on the Kibana dashboard, click on Visualize Library.
 

Create a New Visualization:

  • Click on Create new visualization.
  • From the list of available visualization types, select Lens.
 

Add Your Metric Fields:

  • In the Lens editor, start by selecting your data source. This will typically be your metrics index, such as apm-* or any other relevant index pattern.
  • From the Fields panel on the left, drag your desired metric fields (e.g., success counters, health database values) onto the main workspace.
  • You can choose how to aggregate these metrics, such as using sum, average, count, etc., depending on what you want to analyze.
 

Choose Your Visualization Type:

  • After adding your fields, Kibana Lens will suggest a default visualization type.
  • You can change this by selecting a visualization type that best fits your data, such as Metric, Line Chart, Bar Chart, etc.
  • For displaying key metrics like success counters, a Metric visualization might be the most suitable choice.
 

Customize Your Visualization:

  • Use the right-hand customization panel to adjust settings like:
    • X-axis and Y-axis: Configure the axes to represent your data appropriately (e.g., plotting metrics over time).
    • Breakdown Fields: Break down the data by fields such as timestamp or categories to gain more insights.
    • Filters: Apply filters to narrow down the data you want to visualize.
 

Save the Visualization:

  • Once you’re happy with how your visualization looks, click Save at the top.
  • Enter a descriptive name for your visualization and save it to the Visualize Library.
 

Create a Dashboard (Optional):

  • If you want to compile multiple visualizations into one view, you can create a dashboard:
    • Click on Dashboard from the left menu.
    • Click Create new dashboard.
    • Use the Add from library button to include your saved visualizations.
    • Arrange and resize them as needed to create a cohesive dashboard layout.
 

Conclusion:

Measuring custom application metrics using OpenTelemetry (OTEL) and visualizing the data in Kibana provides a robust solution for monitoring your application’s performance. By leveraging OTEL, you can collect detailed metrics from your application, regardless of the technology stack, and send them to Kibana for real-time analysis. This integration empowers you to gain valuable insights, troubleshoot issues more effectively, and optimize your application’s behavior based on precise data.

With these tools, you can set up comprehensive observability, customize your visualizations, and create dashboards that highlight the most critical metrics for your business needs. Start exploring OTEL and Kibana today to take your application monitoring to the next level. If you have any questions or need further guidance, don’t hesitate to reach out. Happy monitoring!