# Normalising deployment data **URL:** \ **Category:** KNIME Analytics Platform\ **Created:** [January 28, 2025, 7:16am UTC](https://forum.knime.com/t/normalising-deployment-data/86003 "2025-01-28T07:16:41Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![lixingxing](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/l/96bed5/32.png) [@lixingxing](https://forum.knime.com/u/lixingxing)\ **Post date:** [January 28, 2025, 7:16am UTC](https://forum.knime.com/t/normalising-deployment-data/86003/1 "2025-01-28T07:16:42Z")
after training a dataset with random forest predictor, I want to deploy with a new dataset. (not from the 30% partitioning). I used the normalisation node for my trained data before partitioning. When I deploy my new dataset and connect it to the random forest predictor, do I need to normalise the new dataset as well? Note; I used smote to train my data after partitioning it. ---
**Author:** ![ActionAndi](https://forum-cdn.knime.com/user_avatar/forum.knime.com/actionandi/32/76198_2.png) [@ActionAndi](https://forum.knime.com/u/ActionAndi)\ **Post date:** [January 28, 2025, 8:27am UTC](https://forum.knime.com/t/normalising-deployment-data/86003/2 "2025-01-28T08:27:24Z")
Hi, if you train you model on normalized data you need to normalize the test data as well. It’s a bit tricky as you need to apply the same normallization model to the test data. Therefore use the blue output of the “normalizer node” with “normalizer (apply)” ---
**Author:** ![lixingxing](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/l/96bed5/32.png) [@lixingxing](https://forum.knime.com/u/lixingxing)\ **Post date:** [January 28, 2025, 8:39am UTC](https://forum.knime.com/t/normalising-deployment-data/86003/3 "2025-01-28T08:39:45Z")
I created a simple workflow as an example, could you explain whether option 1 or option 2 is the correct way to do it? And I am connecting a new dataset to the random forest predictor instead of using the test data to make prediction. Option 1: ![image](https://forum-cdn.knime.com/uploads/default/original/3X/9/4/945a7cfb926c4d81e55ad4909c7b839d620bea6d.png) Option 2: ![image](https://forum-cdn.knime.com/uploads/default/original/3X/b/5/b52e31b12fd1ad13c628fd67ea3a7bddcabf1f46.png) ---
**Author:** ![qqilihq](https://forum-cdn.knime.com/user_avatar/forum.knime.com/qqilihq/32/50_2.png) [@qqilihq](https://forum.knime.com/u/qqilihq)\ **Post date:** [January 28, 2025, 8:41am UTC](https://forum.knime.com/t/normalising-deployment-data/86003/4 "2025-01-28T08:41:47Z")
Why do you normalize RF training data at all? Did it give any benefit during evaluation? (I’d doubt that, but you never know). ---
**Author:** ![lixingxing](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/l/96bed5/32.png) [@lixingxing](https://forum.knime.com/u/lixingxing)\ **Post date:** [January 28, 2025, 8:44am UTC](https://forum.knime.com/t/normalising-deployment-data/86003/5 "2025-01-28T08:44:45Z")
i normalised the dataset to use Smote to balance it out. Originally, the target variable was imbalance. ---
**Author:** ![ActionAndi](https://forum-cdn.knime.com/user_avatar/forum.knime.com/actionandi/32/76198_2.png) [@ActionAndi](https://forum.knime.com/u/ActionAndi)\ **Post date:** [January 28, 2025, 9:06am UTC](https://forum.knime.com/t/normalising-deployment-data/86003/6 "2025-01-28T09:06:00Z")
If you read new data which is not normalized you need to apply the “normalizer model” from the normalizer node: ![image](https://forum-cdn.knime.com/uploads/default/original/3X/a/a/aabfbbc29148cebf084a74b5c94fc082e86b70af.png) The random forest model is trained on normalized data and can therefore be only applied to normalized data. ---
**Author:** ![nan](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/n/cc9497/32.png) [@nan](https://forum.knime.com/u/nan)\ **Post date:** [January 28, 2025, 9:18am UTC](https://forum.knime.com/t/normalising-deployment-data/86003/7 "2025-01-28T09:18:51Z")
Hi @lixingxing , in any case it is important to partition the data BEFORE you normalize. Otherwise you leak information from you test set into your training data. See this basic example: > **[SVM on iris dataset – knime](https://hub.knime.com/knime/spaces/Educators%20Alliance/Guide%20to%20Intelligent%20Data%20Science/Example%20Workflows/Chapter9/03_SVM~DTfbNITUngKQVF8v/current-state)** > > This workflow solves a classification problem on the iris dataset using Support Vector Machines (SVM). ---
**Author:** ![lixingxing](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/l/96bed5/32.png) [@lixingxing](https://forum.knime.com/u/lixingxing)\ **Post date:** [January 28, 2025, 9:22am UTC](https://forum.knime.com/t/normalising-deployment-data/86003/8 "2025-01-28T09:22:06Z")
But after doing so, do i have to normalise my new dataset as well? (its not the test data from partitioning, it a new one I use to make prediction) ---
**Author:** ![qqilihq](https://forum-cdn.knime.com/user_avatar/forum.knime.com/qqilihq/32/50_2.png) [@qqilihq](https://forum.knime.com/u/qqilihq)\ **Post date:** [January 28, 2025, 9:26am UTC](https://forum.knime.com/t/normalising-deployment-data/86003/9 "2025-01-28T09:26:29Z")
> [@lixingxing](#): > > i normalised the dataset to use Smote to balance it out. Originally, the target variable was imbalance. Got it - if it’s just the sampler which requires normalized data, how about denormalizing the data before training the Random Forest? This avoids the normalization step on the classification data altogether (incl. all other potential footguns, additional processing steps, dependence of normalized data of the learned model). > **[Denormalizer — NodePit](https://nodepit.com/node/org.knime.base.node.preproc.normalize.NormalizerDeNodeFactory)** > > Denormalizes the attributes of a table according to a model. ---
**Author:** ![lixingxing](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/l/96bed5/32.png) [@lixingxing](https://forum.knime.com/u/lixingxing)\ **Post date:** [January 28, 2025, 9:31am UTC](https://forum.knime.com/t/normalising-deployment-data/86003/10 "2025-01-28T09:31:32Z")
Hi, I might be misunderstanding because i am very new. If i denormalise the data before training, does it mean feeding denormalised data to smote? Which i suppose should be wrong to do so ---
**Author:** ![lixingxing](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/l/96bed5/32.png) [@lixingxing](https://forum.knime.com/u/lixingxing)\ **Post date:** [January 28, 2025, 9:38am UTC](https://forum.knime.com/t/normalising-deployment-data/86003/11 "2025-01-28T09:38:07Z")
Hi, it seems like my data range is different, would it work the same way if I use option 1 and set the normalization as min-mix for both normalizers? ![image](https://forum-cdn.knime.com/uploads/default/original/3X/f/7/f7df5f431545c5808d96c96f182385761c9d5b44.png) ---
**Author:** ![qqilihq](https://forum-cdn.knime.com/user_avatar/forum.knime.com/qqilihq/32/50_2.png) [@qqilihq](https://forum.knime.com/u/qqilihq)\ **Post date:** [January 28, 2025, 9:45am UTC](https://forum.knime.com/t/normalising-deployment-data/86003/12 "2025-01-28T09:45:21Z")
> Hi, I might be misunderstanding because i am very new. If i denormalise the data before training, does it mean feeding denormalised data to smote? Which i suppose should be wrong to do so I would try the following sequence: 1. [Normalizer](https://nodepit.com/node/org.knime.base.node.preproc.normalize3.Normalizer3NodeFactory) 2. Sampling (i.e. [SMOTE](https://nodepit.com/node/org.knime.base.node.mine.smote.SmoteNodeFactory)) 3. [Denormalizer](https://nodepit.com/node/org.knime.base.node.preproc.normalize.NormalizerDeNodeFactory) (which is the reverse of the normalization you performed before) 4. [Random Forest Learner](https://nodepit.com/node/org.knime.base.node.mine.treeensemble2.node.randomforest.learner.classification.RandomForestClassificationLearnerNodeFactory2) The classification data will not need to be normalized this way as the model was trained on the “original domain” data. ---
**Author:** ![lixingxing](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/l/96bed5/32.png) [@lixingxing](https://forum.knime.com/u/lixingxing)\ **Post date:** [January 28, 2025, 6:06pm UTC](https://forum.knime.com/t/normalising-deployment-data/86003/13 "2025-01-28T18:06:52Z")
Hi! Thank you, this solve my issue. Just want to ask if it is a must to normalize my data before Smote? I got a higher accuracy without normalizing it ---
**Author:** ![ActionAndi](https://forum-cdn.knime.com/user_avatar/forum.knime.com/actionandi/32/76198_2.png) [@ActionAndi](https://forum.knime.com/u/ActionAndi)\ **Post date:** [January 28, 2025, 7:06pm UTC](https://forum.knime.com/t/normalising-deployment-data/86003/14 "2025-01-28T19:06:45Z")
Hi @nan , are you sure? I think if the dataset is splitted before the normalization/standardization you can run into the problem, that the model must extrapolate or has to deal with features which are not in range of the training dataset. And with maybe not a large dataset the split can lead to a change of the mean or standard deviation of a feature. But I’m not an expert, maybe I’m wrong 🙂 ---
**Author:** ![nan](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/n/cc9497/32.png) [@nan](https://forum.knime.com/u/nan)\ **Post date:** [January 29, 2025, 8:40am UTC](https://forum.knime.com/t/normalising-deployment-data/86003/15 "2025-01-29T08:40:56Z")
Hi, you basically exactly describe the problem. If you normalize before partitioning, in your example the mean is influenced by the data in the test set. Therefore the training knows something about the test set, which means that the test is not independent of the training anymore, which again means that it is not a good test. In practice, if you sample randomly (and do not have bad luck) the mean should be similar in both, training and test set. However, as you also described each outlier/extreme value can only be in one of the two sets, which means the model needs to extrapolate. But that is also likely to happen later on when the model is deployed. If you know that certain features can only appear in a certain range – say 1 through 10 – you can also use that information to normalize instead of “learning” the normalization from the training data. That’s how I understand it, though I do not have a good source at hand right now. Cheers, nan ---
**Author:** ![system](https://forum-cdn.knime.com/uploads/default/original/1X/ab3ccf34482a0329361734a18199390177204f15.png) [@system](https://forum.knime.com/u/system)\ **Post date:** [February 5, 2025, 8:41am UTC](https://forum.knime.com/t/normalising-deployment-data/86003/16 "2025-02-05T08:41:16Z")
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.