Feature Flags: Safe Rollouts, Kill Switches, and the Dark Launch Pattern

Most engineers encounter feature flags through A/B testing frameworks and think of them primarily as experimentation tools. This undersells what flags actually provide: the ability to decouple code deployment from feature release.

Before feature flags, deploying code meant releasing the feature. The deploy was the release. This created enormous pressure on every deploy: if the feature had a bug, the only mitigation was a rollback of the entire deploy, which might include other changes, bug fixes, or infrastructure updates you did not want to roll back.

Feature flags break this coupling. You deploy the code with the feature behind a flag in the off state. The code is running in production, but users see no behavior change. You then enable the flag for 1% of users, watch the metrics, enable it for 10%, watch again, and gradually ramp to 100%. If anything goes wrong at any stage, you flip the flag to off and user behavior reverts instantly — no rollback, no redeploy.

This is why high-deployment-frequency organizations (GitHub deploys ~80 times per day, Netflix hundreds) are not reckless — they are disciplined about progressive delivery. Feature flags make every deploy reversible in seconds.

The second major use case that most resources skip: the kill switch. A kill switch is a flag specifically designed to be flipped off in production when a dependency or subsystem is degrading. "If the recommendation service latency exceeds 500ms, disable personalized recommendations and serve the popular items fallback." This is circuit breaker logic implemented as a feature flag, and it makes systems dramatically more resilient to dependency failures.

Gradual rollout reduces risk by limiting the blast radius of a bug to the fraction of users in the treatment group. But the ramp strategy matters — a naive percentage ramp can produce inconsistent user experiences and invalid metrics.

The consistent hashing requirement. The most important property of a rollout is that the same user sees the same experience on repeated visits. This requires consistent hashing: hash(user_id + flag_id) % 100 < rollout_percentage. This ensures:

• A user in the 10% rollout sees the new feature on every page load, not randomly
• When you ramp from 10% to 20%, the original 10% retain the new feature (they are not randomly re-assigned)
• The control group in metrics analysis is stable — the same users are always in control

Never use random assignment per-request (Math.random() < 0.1). This makes the same user see the new feature sometimes and the old feature other times, which produces confusing UX and invalid A/B test metrics (the same user appears in both treatment and control).

The recommended ramp schedule for a typical feature:

• 1% (QA validation, 24 hours): Verify the feature works correctly in production. Check error rates, latency, and business metrics for the treatment group vs. control. Fix bugs before ramping.
• 10% (initial validation, 48–72 hours): Check that metrics are consistent across the 1–10% range. Watch for P99 latency impact. Verify the feature behaves correctly for all user segments (mobile vs. web, logged-in vs. guest).
• 50% (broad validation, 48–72 hours): Watch for metrics that only appear at scale — connection pool exhaustion, cache invalidation patterns, rate limiting from third-party APIs.
• 100% (full launch): Remove the flag from the code, deprecate the flag in the system, create the cleanup ticket.

Ring deployments (used by Microsoft, AWS) are an alternative to percentage rollouts: instead of a random 10% of users, you define rings — internal employees, beta users, specific geographic regions, small customers, large customers. Each ring has well-understood risk tolerance. This is more complex to implement but gives more precise control over who sees the feature and when.

Every service that depends on an external or slow dependency should have a kill switch. This is the pattern that makes the difference between a graceful degradation and a cascading failure.

Kill switch design principles:

(1) Define the fallback behavior before the flag. A kill switch is useless if the code does not have a defined fallback for the disabled case. Before you can disable personalized recommendations, you need to know: what do users see instead? (Popular items? Cached last recommendations? Empty state?) The fallback must be implemented and tested before the kill switch is meaningful.

(2) Kill switches should default to on. The flag should be "disable_feature: false" rather than "enable_feature: true." This means when the flag evaluation system itself has a failure, the feature remains enabled — which is the safer default for most features. Exception: features that have known scaling problems under certain conditions may default to disabled.

(3) They must propagate quickly. A kill switch that takes 5 minutes to propagate to all serving instances is not useful during a cascading failure. Kill switch propagation should be via streaming push (SSE or WebSocket), not polling. Target: flag change visible in all instances within 30 seconds. LaunchDarkly's streaming system achieves this in under 500ms.

(4) Test them regularly. Kill switches that are never exercised tend to be broken when you need them. Run a quarterly kill switch exercise: flip the switch in a staging environment, verify the fallback behavior is correct, flip it back. This is the equivalent of fire drills for production resilience.

Kill switch anatomy for a recommendation service:

def get_recommendations(user_id: str) -> list[Product]:
    if flag_client.is_enabled("ops_disable_personalized_recs", user_id):
        # Kill switch is active — return the fallback immediately,
        # do not call the recommendation service
        return get_popular_items(limit=10)

    try:
        return recommendation_service.get_for_user(user_id, timeout=0.5)
    except (TimeoutError, ServiceUnavailableError):
        # Also fall back if the service is slow/down
        # Kill switch lets us do this preemptively without a code change
        return get_popular_items(limit=10)

The dark launch pattern: send real production traffic to new code or a new service, but do not show the results to users. Compare the new behavior against the old behavior in the background.

This is the safest possible way to validate that new code is correct under production traffic patterns before any user sees a different experience.

Dark launch use cases:

• Validating a new search ranking model without changing search results users see
• Testing a new database query that should return the same results as the old one
• Load testing a new service version under real traffic before cutover
• Validating that a new payment gateway produces the same charge amounts as the old one

Implementation pattern (shadow traffic):

def handle_search_request(query: str, user_id: str) -> SearchResults:
    # Primary path — always runs, results returned to user
    primary_results = current_search_engine.search(query, user_id)

    # Dark launch path — runs in background if flag is enabled
    if flag_client.is_enabled("dark_launch_new_search", user_id):
        asyncio.create_task(
            shadow_search_and_compare(query, user_id, primary_results)
        )

    return primary_results

async def shadow_search_and_compare(query, user_id, primary_results):
    try:
        shadow_results = new_search_engine.search(query, user_id)
        # Log divergence for analysis — do not affect the user's response
        if shadow_results != primary_results:
            metrics.increment("search.shadow.divergence",
                              tags={"query_type": classify_query(query)})
    except Exception as e:
        # Shadow failures are non-fatal — never let them affect the primary response
        logger.warning("Shadow search failed", error=str(e))

The key constraint: the shadow path must be fire-and-forget. It must not block the primary response, must not add to the primary response latency, and must not propagate exceptions. Shadow failures are expected and acceptable — you are validating new code.