Feature Stores: Online/Offline Architecture & Training-Serving Consistency

Without a feature store, every team that needs a feature (user's 30-day purchase history, item click-through rate) recomputes it independently. Data scientists compute it one way for training notebooks. The serving team reimplements it differently in the API. The result: training-serving skew — the #1 cause of models that pass offline evaluation but degrade silently in production.

A feature store solves this by establishing a single feature registry where features are defined once and computed consistently across all three contexts: training dataset generation, batch inference, and real-time serving. The definition is the contract. The store enforces it.

Production feature stores power the ML platforms at virtually every major tech company: Uber's Michelangelo, Meta's FBLearner Feature Store, Airbnb's Zipline, LinkedIn's Frame, and Twitter's (X's) Feature Engineering Platform all follow the same two-tier architecture.

Every production feature store has exactly two tiers:

Offline store (batch/historical): Parquet files on S3, BigQuery tables, or Snowflake. Stores historical feature values keyed by (entity_id, timestamp). Used for: generating training datasets with point-in-time correct joins, batch scoring jobs, historical analysis. Query latency: seconds to minutes. Updated by Spark or dbt batch pipelines on an hourly or daily cadence. Scale: petabytes.

Online store (low-latency serving): Redis, DynamoDB, Bigtable, or RonDB (Hopsworks). Stores only the latest feature value per entity (user_id, item_id). Used for: real-time model serving. Query latency: p99 < 10ms (Tecton SLA), typically 1–5ms on Redis. Updated by Kafka consumer pipelines or scheduled batch refresh from the offline store. Scale: millions of entities, but only the most recent values.

The critical constraint: both tiers use the same feature transformation code. Feast and Tecton enforce this through a feature registry — Python or SQL functions that define the transformation. Run that function over historical data → offline store. Run it over the streaming event feed → online store. Same function → same features → no skew.

The most underappreciated feature store capability. When generating a training dataset, you need to join feature values to labels — but you must only use feature values that were available at the time the label was generated.

Example: A fraud model trained on transactions from January. Each transaction has a label (fraud/not-fraud determined via chargeback, known 30 days later). If you naively join using the latest feature values, you might include 'user_fraud_rate_last_30d' computed in February — which includes the January fraud signal itself. This is future leakage. The model learns from information it won't have at serving time.

Point-in-time correct join: for each (entity_id, event_timestamp) in your label dataset, look up the feature value as of event_timestamp - epsilon. Not the latest value. The value that existed at that moment in time.

-- Point-in-time correct join in BigQuery
SELECT
  t.transaction_id,
  t.label,
  t.event_timestamp,
  f.user_fraud_rate_30d  -- value as of event_timestamp, not today
FROM transactions t
INNER JOIN LATERAL (
  SELECT user_fraud_rate_30d
  FROM feature_history
  WHERE entity_id = t.user_id
    AND feature_timestamp <= t.event_timestamp
  ORDER BY feature_timestamp DESC
  LIMIT 1
) f ON TRUE

Feast and Tecton both expose get_historical_features(entity_df) which handles point-in-time joins automatically. Without a feature store, teams implement this manually — and frequently get it wrong.

Stale online features: The materialization job that pushes from offline to online store fails silently. The online store serves last week's values. Solution: monitor feature freshness — track the age of the latest value per feature view. Alert if any feature is stale beyond 1.5× its expected update interval.

Timezone bugs: Offline training data uses UTC timestamps. The online store ingests events in local timezone from a mobile SDK. Features computed from these have systematically different distributions. Fix: standardize all timestamps to UTC at the event capture layer, not the feature layer.

Too many feature views per request: Fetching 15 separate feature views in a single inference request serially pushes p99 latency to 100ms+. Solution: bundle features into Feature Services (Feast concept) so they're fetched in a single batch call. Or pre-join features into a single wide entity in the online store.

Online store cost at scale: Redis storing 100M user entities × 500 features × 8 bytes = 400GB RAM. At ~$5/GB/month for ElastiCache, that's $2,000/month just for one feature set. Solution: be selective — only materialize features that are actually used at serving time. Features used only for training stay in the offline store.