DevOps Security: From Hardcoded Secrets to Vault Integration

One of the most critical aspects of production deployment is secure secrets management. This post chronicles the journey from hardcoded credentials to a production-ready Vault integration, highlighting the security improvements and operational benefits achieved.

DevOps

DevOps Security: From Hardcoded Secrets to Vault Integration

Introduction

One of the most critical aspects of production deployment is secure secrets management. This post chronicles the journey from hardcoded credentials to a production-ready Vault integration, highlighting the security improvements and operational benefits achieved.

The Problem: Hardcoded Secrets Anti-Pattern

Initial State (The Bad)

# meta-dev.yml - Initial approach (NEVER DO THIS)
spec:
 containers:
 - name: wireless-exporter
 env:
 - name: WIRELESS_USER
 value: "postgres_user" # ❌ Hardcoded
 - name: WIRELESS_PASSWORD 
 value: "super_secret_password" # ❌ Exposed in Git
 - name: WIRELESS_HOST
 value: "replica-postgres-main-14.query.consul"

Problems with this approach: - Secrets exposed in version control - No rotation capabilities - Compliance violations - Security audit failures - Team members have access to production credentials

The Evolution: Step-by-Step Security Improvements

Phase 1: Environment Variables

# First improvement - Environment variables
spec:
 containers:
 - name: wireless-exporter
 env:
 - name: WIRELESS_USER
 value: "${WIRELESS_USER}" # Better, but still not ideal
 - name: WIRELESS_PASSWORD
 value: "${WIRELESS_PASSWORD}"

Issues remaining: - Environment variables visible in process lists - Still no proper rotation - Manual secret distribution

Phase 2: Kubernetes Secrets

# Better approach - Kubernetes secrets
spec:
 containers:
 - name: wireless-exporter
 env:
 - name: WIRELESS_USER
 valueFrom:
 secretKeyRef:
 name: wireless-secrets
 key: username
 - name: WIRELESS_PASSWORD
 valueFrom:
 secretKeyRef:
 name: wireless-secrets 
 key: password

Improvements: - Secrets not in application configuration - Base64 encoded (but not encrypted at rest) - RBAC controlled access

Phase 3: Vault Integration (The Solution)

# Production-ready approach
spec:
 containers:
 - name: wireless-exporter
 envFrom:
 - secretRef:
 name: wireless-secrets # Vault-managed secret

Vault Implementation Details

Vault Configuration Structure

vault-ha-prod.internal..com/
├── secret/
│ └── wireless-query-exporter/
│ ├── WIRELESS_USER
│ ├── WIRELESS_PASSWORD
│ ├── WIRELESS_BILLING_USER
│ └── WIRELESS_BILLING_PASSWORD

Application Integration

# main.py - Environment-based configuration
import os class HealthQueryExporterScript(QueryExporterScript): def __init__(self): super().__init__() self.config = self._load_config_from_env() def _load_config_from_env(self):
 """Load configuration from environment variables populated by Vault""" return { 'wireless_user': os.environ.get('WIRELESS_USER'), 'wireless_password': os.environ.get('WIRELESS_PASSWORD'), 'wireless_host': os.environ.get('WIRELESS_HOST', 'replica-postgres-main-14.query.consul'), 'wireless_port': int(os.environ.get('WIRELESS_PORT', '5432')), }

Deployment Configuration

# meta-prod.yml - Final production configuration
apiVersion: apps/v1
kind: Deployment
spec:
 template:
 spec:
 containers:
 - name: wireless-exporter
 envFrom:
 - secretRef:
 name: wireless-secrets # Managed by Vault Operator
 env:
 # Static configuration (non-sensitive)
 - name: WIRELESS_HOST
 value: "replica-postgres-main-14.query.consul"
 - name: WIRELESS_PORT
 value: "5432"
 - name: WIRELESS_DB
 value: "wireless_manager"

Security Benefits Achieved

1. Secret Rotation

# Vault CLI - Easy secret rotation
vault kv put secret/wireless-query-exporter \
 WIRELESS_USER="new_username" \
 WIRELESS_PASSWORD="new_rotated_password" # Application picks up new secrets on next deployment/restart
kubectl rollout restart deployment/wireless-exporter

2. Access Control

# Vault policy - wireless-query-exporter-policy.hcl
path "secret/data/wireless-query-exporter" {
 capabilities = ["read"]
} path "secret/metadata/wireless-query-exporter" {
 capabilities = ["list", "read"]
}

3. Audit Trail

{
 "time": "2024-01-15T10:30:00Z",
 "type": "request",
 "auth": {
 "client_token": "hmac-sha256:xxx",
 "accessor": "hmac-sha256:yyy"
 },
 "request": {
 "path": "secret/data/wireless-query-exporter",
 "operation": "read"
 }
}

Configuration Management Strategy

Static vs Secret Configuration

# Static configuration (safe to commit)
STATIC_CONFIG = { 'WIRELESS_HOST': 'replica-postgres-main-14.query.consul', 'WIRELESS_PORT': 5432, 'WIRELESS_DB': 'wireless_manager', 'WIRELESS_BILLING_HOST': 'replica-postgres-main-14.query.consul', 'WIRELESS_BILLING_PORT': 5432, 'WIRELESS_BILLING_DB': 'wireless_billing'
} # Secret configuration (from Vault)
SECRET_CONFIG = [ 'WIRELESS_USER', 'WIRELESS_PASSWORD', 'WIRELESS_BILLING_USER', 'WIRELESS_BILLING_PASSWORD'
]

Environment-Specific Deployment

# Different Vault paths for different environments
# Development
envFrom:
- secretRef:
 name: wireless-secrets-dev # Points to vault-ha-dev.internal # Production 
envFrom:
- secretRef:
 name: wireless-secrets # Points to vault-ha-prod.internal

Operational Improvements

1. Automated Secret Distribution

# Vault Operator automatically syncs secrets
apiVersion: v1
kind: Secret
metadata:
 name: wireless-secrets
 annotations:
 vault.security.banzaicloud.io/vault-addr: "https://vault-ha-prod.internal"
 vault.security.banzaicloud.io/vault-path: "secret/data/wireless-query-exporter"
 vault.security.banzaicloud.io/vault-role: "wireless-query-exporter"
type: Opaque

2. Local Development Setup

# .env.local (gitignored)
WIRELESS_USER=dev_user
WIRELESS_PASSWORD=dev_password
WIRELESS_BILLING_USER=billing_dev_user 
WIRELESS_BILLING_PASSWORD=billing_dev_password # docker-compose.yml 
services:
 wireless-exporter:
 env_file:
 - .env.local # Local development secrets

3. CI/CD Integration

// Jenkinsfile
pipeline {
 stages {
 stage('Deploy') {
 steps {
 script {
 // Vault authentication in CI
 sh """
 vault auth -method=aws
 vault read -field=status secret/wireless-query-exporter
 kubectl apply -f meta-prod.yml
 """
 }
 }
 }
 }
}

Best Practices Implemented

1. Principle of Least Privilege

  • Each environment has separate Vault paths
  • Service accounts have minimal required permissions
  • Regular access reviews and cleanup

2. Secret Hygiene

  • No secrets in git history (verified with tools like git-secrets)
  • Regular secret rotation schedule
  • Temporary secrets for development

3. Monitoring & Alerting

# Prometheus alert for secret access
groups:
- name: vault.rules
 rules:
 - alert: VaultSecretAccessFailure
 expr: increase(vault_audit_log_request_failure_total[5m]) > 0
 labels:
 severity: warning
 annotations:
 summary: "Failed Vault secret access detected"

Migration Checklist

For teams implementing similar improvements:

Pre-Migration

  • [ ] Audit current secret usage - Identify all hardcoded credentials
  • [ ] Set up Vault infrastructure - Install and configure Vault
  • [ ] Create secret policies - Define access control
  • [ ] Test in development - Validate the integration

Migration Steps

  • [ ] Create Vault secrets - Migrate credentials to Vault
  • [ ] Update application code - Environment variable integration
  • [ ] Deploy Vault operator - Automate secret synchronization
  • [ ] Update deployment configs - Remove hardcoded values
  • [ ] Verify functionality - Test secret rotation

Post-Migration

  • [ ] Document procedures - Secret rotation and access
  • [ ] Set up monitoring - Alert on access failures
  • [ ] Train team members - Vault usage and best practices
  • [ ] Regular audits - Review access and rotate secrets

Lessons Learned

1. Start Simple, Iterate

Don't try to implement perfect security on day one. Incremental improvements are better than analysis paralysis.

2. Environment Parity

Ensure development and production use the same secret management approach to catch issues early.

3. Documentation is Critical

Clear procedures for secret access, rotation, and emergency recovery are essential.

4. Monitoring Matters

Failed secret access should trigger alerts - it often indicates misconfigurations or security issues.

Security Metrics

Before Vault Integration:

  • ❌ 100% of secrets hardcoded
  • ❌ 0% secret rotation capability
  • ❌ No access audit trail
  • ❌ Secrets visible in git history

After Vault Integration:

  • ✅ 0% hardcoded secrets
  • ✅ 100% secrets support rotation
  • ✅ Complete audit trail
  • ✅ Zero secrets in version control

Future Enhancements

  1. Dynamic Secrets: Generate database credentials on-demand
  2. Secret Scanning: Automated detection of exposed secrets
  3. Rotation Automation: Automated secret rotation schedules
  4. Emergency Procedures: Break-glass access for incident response

Conclusion

Migrating from hardcoded secrets to Vault integration represents a fundamental security improvement that pays dividends in:

  • Compliance: Meeting security audit requirements
  • Operations: Simplified secret rotation and distribution
  • Security: Proper access control and audit trails
  • Team Confidence: Developers can deploy without security concerns

The key lesson: Security improvements can be implemented incrementally without disrupting development velocity. Start with identifying the most critical secrets and migrate them first.

Remember: The best security practices are the ones your team will actually follow consistently.

This security migration eliminated 14 lines of hardcoded secrets and established a foundation for enterprise-grade secret management practices.