Secure Environment Manager: Designing a Self-Hosted Secrets Platform

The Problem Statement

Every engineering team has the same problem: secrets proliferate. API keys, database credentials, JWT tokens, encryption keys. They live in .env files, Slack messages, email threads, and Notion docs. Nobody knows which services use which keys. Nobody knows when a key was rotated. Nobody knows if a key has been compromised.

The cloud-native solutions - AWS Secrets Manager, HashiCorp Vault, Doppler - work well but have three problems for small teams: cost, operational complexity, and third-party dependency.

I needed something I could self-host, understand completely, and extend without friction.

This is the design story of what I built.

Threat Model

Before I designed anything, I wrote down what I was protecting against.

External threats:

• Database credentials leaked through version control (the most common failure mode)
• API keys compromised via employee departure or phishing
• Unauthorized access to production credentials
• Audit trail requirements for compliance

Internal threats:

• Engineer accidentally exposing secrets in logs or error messages
• Service-to-service authentication without shared secrets
• Key rotation without service restart

Out of scope for v1:

• Secrets injection at runtime (Vault's Agent approach)
• Secret leasing and renewal
• Multi-tenancy

The threat model shaped every architectural decision that followed.

Architecture Overview

Why this layering:

The API Gateway handles rate limiting, request validation, and auth token verification before any request reaches the business logic. This is standard defense in depth - each layer is responsible for its own security domain.

The Encryption Service is a separate module that handles all cryptographic operations. This means if I need to change encryption algorithms or key rotation strategies, I change one module without touching the API layer.

The Audit Logger writes to PostgreSQL synchronously - I don't want audit logs that can be lost in a crash. Every secret access, rotation, and permission change is logged, because you can't observe what you haven't measured. I wrote about why observability comes before optimization in the context of AI platforms, but the principle applies here too: you can't detect a breach you haven't logged for.

Authentication Flow

Design decisions:

I chose short-lived JWT access tokens (15 minutes) and longer-lived refresh tokens (7 days). The access token is stateless JWT - the API verifies the signature without calling the auth service. The refresh token is stored in PostgreSQL and can be revoked.

Why not long-lived tokens? If a token is stolen, the attacker has 15 minutes of access, not 7 days.

Why not stateless refresh tokens? Because I needed revocation capability. If a service is compromised, I need to be able to invalidate its refresh token immediately.

Secret Storage Architecture

Secrets are encrypted at rest using AES-256-GCM. The encryption key hierarchy:

Why a key hierarchy:

If I need to rotate the master key, I only re-encrypt the data keys. I don't need to decrypt and re-encrypt every secret in the database.

Data keys are generated per-service. Service A's data key can't decrypt Service B's secrets. If Service A's data key is compromised, only Service A's secrets need rotation.

The schema:

CREATE TABLE data_keys (
    id UUID PRIMARY KEY,
    service_id TEXT NOT NULL,
    encrypted_key BYTEA NOT NULL,  -- Data key encrypted with master key
    iv BYTEA NOT NULL,
    created_at TIMESTAMPTZ NOT NULL,
    rotated_at TIMESTAMPTZ,
    active BOOLEAN DEFAULT true
);
 
CREATE TABLE secrets (
    id UUID PRIMARY KEY,
    data_key_id UUID REFERENCES data_keys(id),
    namespace TEXT NOT NULL,  -- e.g., "production", "staging"
    secret_name TEXT NOT NULL,
    encrypted_value BYTEA NOT NULL,
    iv BYTEA NOT NULL,
    version INTEGER NOT NULL DEFAULT 1,
    created_at TIMESTAMPTZ NOT NULL,
    updated_at TIMESTAMPTZ NOT NULL,
    UNIQUE(namespace, secret_name, version)
);

Each secret has a version. When you rotate a secret, you create a new version. The "current" secret is the one with the highest version number for that (namespace, secret_name) tuple.

API Design

The API follows REST conventions with predictable naming:

GET    /api/v1/secrets                    # List all secrets (names only, not values)
GET    /api/v1/secrets/{namespace}/{name}  # Get current value
POST   /api/v1/secrets/{namespace}/{name}  # Create new secret
PUT    /api/v1/secrets/{namespace}/{name}  # Update (creates new version)
DELETE /api/v1/secrets/{namespace}/{name}  # Soft delete (mark inactive)
POST   /api/v1/secrets/{namespace}/{name}/rotate  # Rotate: new version, notify subscribers
GET    /api/v1/audit                      # Audit log

What I got wrong in v1:

The initial API returned decrypted secret values by default. I changed this after realizing that logging middleware was capturing response bodies - meaning decrypted secrets were being written to log files.

The fix: secrets are decrypted in memory only for the specific API response, not stored or cached in decrypted form.

Real-Time Updates with Redis Pub/Sub

When a secret rotates, running services need to know immediately - otherwise they keep using the old secret until they restart.

The subscription model:

Services connect to a WebSocket endpoint authenticated with their service JWT. The Manager process publishes to a Redis channel when any secret changes. Redis fans out to all connected WebSocket clients.

Services receive a notification containing the namespace and secret name (not the value - the value is still fetched via REST). Services then fetch the new value and update their in-memory state.

What I got wrong:

I initially used Redis keyspace notifications for this. Keyspace notifications tell you that a key changed, not what the new value is. I switched to explicit pub/sub after realizing services had to re-fetch anyway - they weren't getting any value from the keyspace notification approach.

Audit Logging

Every secret access is logged:

CREATE TABLE audit_log (
    id BIGSERIAL PRIMARY KEY,
    timestamp TIMESTAMPTZ NOT NULL DEFAULT now(),
    actor_id TEXT NOT NULL,           -- Service ID or user ID
    actor_type TEXT NOT NULL,         -- 'service' or 'user'
    action TEXT NOT NULL,             -- 'read', 'create', 'update', 'delete', 'rotate'
    namespace TEXT,
    secret_name TEXT,
    success BOOLEAN NOT NULL,
    ip_address INET,
    user_agent TEXT,
    error_message TEXT
);

What I log:

• Every secret read (what accessed what, when, from where)
• Every secret modification (who changed what to what)
• Failed authentication attempts
• Authorization failures

What I don't log:

• The actual secret values (that would defeat the purpose)

The audit query I use most:

SELECT
    timestamp,
    actor_id,
    action,
    namespace,
    secret_name,
    success
FROM audit_log
WHERE secret_name = 'api_key'
    AND timestamp > now() - interval '24 hours'
ORDER BY timestamp DESC;

This tells me every access to a specific secret in the last 24 hours. When something goes wrong, this is the first query I run.

Deployment Strategy

Single-node deployment for v1. The Docker Compose stack runs on a $10/month VPS. The architecture would need changes for multi-node (primarily around Redis pub/sub fan-out), but for a small team with a handful of services, single-node is fine.

The startup sequence:

1. PostgreSQL starts first, creates database if not exists
2. Redis starts, verifies connection
3. API service starts, runs migrations, waits for Redis
4. Web dashboard starts
5. Manager worker starts, subscribes to Redis channels

Health checks at each step. If PostgreSQL isn't ready, the API won't start. If Redis isn't ready, the API starts but reports unhealthy.

What I Underestimated

The revocation problem. When a service is compromised, I need to:

1. Revoke its JWT refresh token (done)
2. Rotate all secrets it had access to (hard - requires knowing which secrets it could access)
3. Notify dependent services (done via pub/sub)

I didn't build role-based access control in v1. Every authenticated service can read every secret. The "rotation" command rotates the secret and notifies subscribers - but a compromised service that hasn't restarted yet still has the old secret in memory.

The fix for v2 is per-service secret bindings: each service can only access secrets it's explicitly granted access to.

The secret injection problem. This system manages secrets. It doesn't inject them into running services. Services still need to poll for updates or restart to pick up new secrets. The "correct" solution (Vault's agent sidecar approach) is significantly more complex.

What I'd Do Differently

Start with RBAC. The system started without access control because "all services can read all secrets" was simpler to implement. But this means a compromised service is a complete compromise. RBAC from day one would have been the right call.

Use Server-Sent Events instead of WebSockets. For one-way notification (secret changed → services notified), SSE is simpler and more appropriate than WebSockets. I used WebSockets because I was familiar with them, not because they were the right tool.

Separate the secret store from the notification system. I coupled these in Redis. A separate notification system (even a simple webhook dispatcher) would be easier to reason about and debug.

The Result

The system has been running for eight months. It manages secrets for three production services and a half-dozen development environments. It has never had a security incident (that I know of).

The rotation workflow has been used 23 times. The audit log has been useful exactly twice - both times when I was investigating anomalous behavior and needed to trace which service accessed what.

The most valuable thing I built wasn't the encryption or the audit logging or the real-time notifications. It was the discipline of knowing exactly which services used which secrets, and being able to reason about what would happen if any single credential were compromised.

That's the thing about security infrastructure: you don't know if it worked until the moment it had to.