Production ML systems: When to transform data?
Stay organized with collections
Save and categorize content based on your preferences.
Raw data must be feature engineered (transformed). When should you transform
data? Broadly speaking, you can perform feature engineering during either of
the following two periods:
- Before training the model.
- While training the model.
In this approach, you follow two steps:
- Write code or use specialized tools
to transform the raw data.
- Store the transformed data somewhere that the model can ingest, such
as on disk.
Advantages
- The system transforms raw data only once.
- The system can analyze the entire dataset to determine the best
transformation strategy.
Disadvantages
Training-serving skew is more dangerous when your system performs dynamic
(online) inference.
On a system that uses dynamic inference, the software that transforms
the raw dataset usually differs from the software that serves predictions,
which can cause training-serving skew.
In contrast, systems that use static (offline) inference can sometimes
use the same software.
In this approach, the transformation is part of the model code. The model
ingests raw data and transforms it.
Advantages
- You can still use the same raw data files if you change the transformations.
- You're ensured the same transformations at training and prediction time.
Disadvantages
- Complicated transforms can increase model latency.
- Transformations occur for each and every batch.
Transforming the data per batch can be tricky. For example, suppose you want to
use Z-score normalization
to transform raw numerical data. Z-score normalization requires the mean and
standard deviation of the feature.
However, transformations per batch mean you'll only have access to
one batch of data, not the full dataset. So, if the batches are highly
variant, a Z-score of, say, -2.5 in one batch won't have the same meaning
as -2.5 in another batch.
As a workaround, your system can precompute the mean and standard deviation
across the entire dataset and then use them as constants in the model.
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2025-08-25 UTC.
[null,null,["Last updated 2025-08-25 UTC."],[[["\u003cp\u003eFeature engineering can be performed before or during model training, each with its own advantages and disadvantages.\u003c/p\u003e\n"],["\u003cp\u003eTransforming data before training allows for a one-time transformation of the entire dataset but requires careful recreation of transformations during prediction to avoid training-serving skew.\u003c/p\u003e\n"],["\u003cp\u003eTransforming data during training ensures consistency between training and prediction but can increase model latency and complicate batch processing.\u003c/p\u003e\n"],["\u003cp\u003eWhen transforming data during training, considerations such as Z-score normalization across batches with varying distributions need to be addressed.\u003c/p\u003e\n"]]],[],null,["# Production ML systems: When to transform data?\n\nRaw data must be feature engineered (transformed). When should you transform\ndata? Broadly speaking, you can perform feature engineering during either of\nthe following two periods:\n\n- *Before* training the model.\n- *While* training the model.\n\nTransforming data before training\n---------------------------------\n\nIn this approach, you follow two steps:\n\n1. Write code or use specialized tools to transform the raw data.\n2. Store the transformed data somewhere that the model can ingest, such as on disk.\n\nAdvantages\n\n- The system transforms raw data only once.\n- The system can analyze the entire dataset to determine the best transformation strategy.\n\nDisadvantages\n\n- You must recreate the transformations at prediction time. Beware of [**training-serving skew**](/machine-learning/glossary#training-serving-skew)!\n\nTraining-serving skew is more dangerous when your system performs dynamic\n(online) inference.\nOn a system that uses dynamic inference, the software that transforms\nthe raw dataset usually differs from the software that serves predictions,\nwhich can cause training-serving skew.\nIn contrast, systems that use static (offline) inference can sometimes\nuse the same software.\n\nTransforming data while training\n--------------------------------\n\nIn this approach, the transformation is part of the model code. The model\ningests raw data and transforms it.\n\nAdvantages\n\n- You can still use the same raw data files if you change the transformations.\n- You're ensured the same transformations at training and prediction time.\n\nDisadvantages\n\n- Complicated transforms can increase model latency.\n- Transformations occur for each and every batch.\n\nTransforming the data per batch can be tricky. For example, suppose you want to\nuse [**Z-score normalization**](/machine-learning/glossary#z-score-normalization)\nto transform raw numerical data. Z-score normalization requires the mean and\nstandard deviation of the feature.\nHowever, transformations per batch mean you'll only have access to\n*one batch of data*, not the full dataset. So, if the batches are highly\nvariant, a Z-score of, say, -2.5 in one batch won't have the same meaning\nas -2.5 in another batch.\nAs a workaround, your system can precompute the mean and standard deviation\nacross the entire dataset and then use them as constants in the model.\n| **Key terms:**\n|\n| - [Training-serving skew](/machine-learning/glossary#training-serving-skew)\n- [Z-score normalization](/machine-learning/glossary#z-score-normalization) \n[Help Center](https://support.google.com/machinelearningeducation)"]]