Building Production-Ready Prometheus Monitoring for Wireless Infrastructure

In today's telecommunications landscape, monitoring wireless infrastructure is critical for maintaining service quality and operational efficiency. This blog post details the development of a specialized Prometheus metrics exporter designed specifically for wireless infrastructure monitoring at scale.

Monitoring

Building Production-Ready Prometheus Monitoring for Wireless Infrastructure

Introduction

The Challenge

When managing thousands of SIM cards, wireless orders, and billing processes, traditional monitoring approaches fall short. We needed a solution that could:

Monitor wireless-specific metrics like MRC tracking, SIM card inventory, and public IP allocation
Handle dual database connections (management and billing systems)
Provide real-time visibility into order statuses and inventory levels
Scale efficiently without impacting production databases

Technical Architecture

Core Components

The solution is built around a custom Python application that extends the base QueryExporterScript to add wireless-specific functionality:

class HealthQueryExporterScript(QueryExporterScript): async def on_application_startup(self, application: Application): application.router.add_get("/health", self._handle_health) await super().on_application_startup(application) async def _handle_health(self, request: Request) -> Response: return json_response({"status": "OK"})

Database Strategy

The system connects to two primary databases: - wireless_manager: Core management database containing SIM cards, orders, and gateway information - wireless_billing: Billing database with Oban job queue monitoring

Using replica connections ensures zero impact on production systems while maintaining real-time monitoring capabilities.

Key Metrics Delivered

The system provides 15+ critical wireless metrics:

Inventory Management: - wireless_total_sim_cards: Total SIM cards by region and allocation status - wireless_v4_sim_in_stock: v4 SIM cards remaining in stock - wireless_v3_sim_in_stock: v3 SIM cards remaining in stock

Order Tracking: - wireless_sim_card_orders_transient_status: Orders in pending/allocated/dispatched states - wireless_sim_card_orders_delivered_total: Total delivered orders - wireless_total_ordered_sim_cards: Current order pipeline

Resource Management: - wireless_total_sim_card_public_ips: Public IP allocation by region - wireless_available_sim_card_public_ips: Available IP pool monitoring

Implementation Highlights

Configuration Management

Using YAML-based configuration for maintainability:

# queries.yaml
wireless_mrcs:
 interval: 300
 databases: [wireless_manager]
 query: |
 SELECT mrc_type, COUNT(*) as count 
 FROM wireless_mrcs 
 WHERE status = 'active'
 GROUP BY mrc_type

Security & Secrets Management

Integrated with HashiCorp Vault for secure credential management: - Environment-based configuration - Automated secret rotation capabilities - Development and production secret separation

Deployment Strategy

The system uses a hybrid deployment approach: - Containerized with Docker for consistency - CI/CD pipeline deployment for reliability - Kubernetes configurations for scalability - Strategic single-datacenter deployment for non-critical services

Performance Optimizations

Query Efficiency

Optimized query intervals (300-600 seconds) based on metric criticality
Replica database usage to prevent production impact
Careful metric cardinality management to prevent explosion

Resource Management

Single replica deployment for non-critical service classification
Memory-optimized query result handling
Efficient connection pooling

Operational Excellence

Monitoring & Alerting

Health endpoint (/health) for service monitoring
Comprehensive metrics endpoint (/metrics) for Prometheus scraping
Integration with Grafana for visualization

Documentation & Maintenance

Comprehensive README with troubleshooting guides
Schema validation for configuration changes
Local development setup with docker-compose

Lessons Learned

1. Infrastructure Decisions Matter

Migrating from Kubernetes to CI pipeline deployment improved reliability and simplified operations for this specific use case.

2. Security First

Implementing Vault integration from the start prevented security technical debt and enabled proper credential lifecycle management.

3. Monitoring Strategy

Balancing monitoring coverage with performance impact requires careful query optimization and interval tuning.

4. Development Experience

Investing in local development setup and comprehensive documentation pays dividends in team velocity and maintenance.

Results & Impact

The wireless monitoring system now provides: - Real-time visibility into 15+ critical wireless infrastructure metrics - Proactive alerting for inventory levels, order processing, and billing health - Zero production impact through replica database usage - Operational efficiency through automated monitoring and alerting

Future Enhancements

Looking ahead, planned improvements include: - Additional wireless-specific metrics based on operational feedback - Enhanced Grafana dashboard templates - Automated anomaly detection for critical metrics - Integration with incident management systems

Conclusion

Building production-ready monitoring for wireless infrastructure requires careful consideration of performance, security, and operational requirements. By leveraging Prometheus, custom Python extensions, and modern DevOps practices, we created a solution that provides critical visibility while maintaining system reliability.

The key to success was focusing on wireless-specific requirements while building on proven monitoring foundations. This approach delivered a monitoring solution that truly serves the needs of telecommunications operations teams.

This implementation represents 25 hours of development work across 22 commits, resulting in a production-ready monitoring solution that provides crucial visibility into wireless infrastructure operations.

Future Imperfect