# t-SNE - how to use in production pipeline? **URL:** \ **Category:** KNIME Extensions\ **Created:** [November 25, 2022, 11:31am UTC](https://forum.knime.com/t/t-sne-how-to-use-in-production-pipeline/49365 "2022-11-25T11:31:23Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![hermyknime](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/h/3da27b/32.png) [@hermyknime](https://forum.knime.com/u/hermyknime)\ **Post date:** [November 25, 2022, 11:31am UTC](https://forum.knime.com/t/t-sne-how-to-use-in-production-pipeline/49365/1 "2022-11-25T11:31:23Z")
Hi All, not sure if this section is the right section to post a question about the Knime Labs Statistics Node package I’m trying out the t-SNE package, but i’m having some troubles wrapping my head around how to use it in a production setting, where i want to apply the transformation on fresh incoming data > **[t-SNE (L. Jonsson)](https://hub.knime.com/knime/extensions/org.knime.features.stats2/latest/org.knime.ext.tsne.node.TsneNodeFactory)** > > t-SNE is a manifold learning technique, which learns low dimensional embeddings for high dimensional data. It is most often used for visualization purposes bec… > [@Feature Suggestion -\> t-distributed stochastic neighbor embedding (T-SNE)](https://forum.knime.com/t/feature-suggestion-t-distributed-stochastic-neighbor-embedding-t-sne/15055): > > T-SNE is rapidly becoming a key algorithm for dimension reduction in many fields including ours, rivaling the use of PCA. Unfortunately, T-SNE in Knime requires either the use of R/Python or a very alpha community node that doesn’t match r/python results. It would be helpful to have a built-in T-SNE node. > [@t-SNE implementation in Knime](https://forum.knime.com/t/t-sne-implementation-in-knime/5151): > > Hello, Did anyone successfully created or attempted to use t-SNE in Knime? Is t-SNE in Knime's future plans? Thanks! I was actually looking for a similar approach as the PCA Compute / Apply setup; where you can compute the transformation, and then export PMML and import it into the Production pipeline for the Apply step ![image](https://forum-cdn.knime.com/uploads/default/original/3X/d/8/d84b0ffc4f778d279c59b54943faeaa1efbec39d.png) is a similar approach possible? or how to go about applying the transformation on new data? – e.g. outside of a train-test workbench Thx a bunch! Herman ---
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [November 25, 2022, 12:36pm UTC](https://forum.knime.com/t/t-sne-how-to-use-in-production-pipeline/49365/2 "2022-11-25T12:36:58Z")
I do not think this is easily possible. T-SNE is mostly there to explore data structures and get new ideas but does not work in a traditionel model and deploy sense. One suggestion ([machine learning - python tsne.transform does not exist? - Stack Overflow](https://stackoverflow.com/a/59215713)) is to train the new data together with the old one and use the result and one more approach by the developer himself (see FAQ). But I do not think it is to be implemented in the near future in a node: > FAQ ([t-SNE – Laurens van der Maaten](https://lvdmaaten.github.io/tsne/)) > > **Once I have a t-SNE map, how can I embed incoming test points in that map?** > > t-SNE learns a non-parametric mapping, which means that it does not learn an explicit function that maps data from the input space to the map. Therefore, it is not possible to embed test points in an existing map (although you could re-run t-SNE on the full dataset). A potential approach to deal with this would be to train a multivariate regressor to predict the map location from the input data. Alternatively, you could also make such a regressor minimize the t-SNE loss directly, which is what I did in [this paper](https://lvdmaaten.github.io/publications/papers/AISTATS_2009.pdf). > **[t-SNE](https://lvdmaaten.github.io/tsne/)** > > t-SNE What you can do is use t-sne to select relevant groups and check them out. I have an older workflow somewhere still using the R t-sne implementation. ---
**Author:** ![Artem](https://forum-cdn.knime.com/user_avatar/forum.knime.com/artem/32/47971_2.png) [@Artem](https://forum.knime.com/u/Artem)\ **Post date:** [November 25, 2022, 12:36pm UTC](https://forum.knime.com/t/t-sne-how-to-use-in-production-pipeline/49365/3 "2022-11-25T12:36:59Z")
Hello @hermyknime, As long as t-SNE is a non-linear algorithm, so it cannot be reverted, and it always depends on the data and even initial random distributions, so you cannot use the same way as PCA. PCA - is a linear algorithm, this means you can always have the same matrix of transformations that you can apply to a new data set with the same structure. This way, I believe you can still use t-SNE in your production pipeline, the only difference is that the vector basis will always be different, as well as meaning of t-SNE projections. So let’s say if you are using t-SNE only for dimension reduction for clustering, ML or manual analysis based on visualization then it is completely fine. However if you are trying to use t-SNE projections for explaining the models, then t-SNE cannot help here. I hope my explanation will be useful for you. ---
**Author:** ![hermyknime](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/h/3da27b/32.png) [@hermyknime](https://forum.knime.com/u/hermyknime)\ **Post date:** [November 25, 2022, 2:12pm UTC](https://forum.knime.com/t/t-sne-how-to-use-in-production-pipeline/49365/4 "2022-11-25T14:12:21Z")
Hey @Artem , @mlauber71 Thank you very much for the info and the suggestions. It does make sense to use T-SNE in this case as a way of fine-tuning the selection of meaningful data cases to then do the ML training on, via the actual features. I was hoping to use T-SNE as a feature-selection / feature reduction approach, but seems it is not that straightforward. Thank you very much for enlightening this to me! (and hopefully others) 😄 Herman ---
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [November 25, 2022, 2:36pm UTC](https://forum.knime.com/t/t-sne-how-to-use-in-production-pipeline/49365/5 "2022-11-25T14:36:29Z")
> [@hermyknime](#): > > feature-selection / feature reduction approach If you are interested in that you could check out the preparation section of my machine learning collection. About tools like vtreat ([vtreat for KNIME! – Win Vector LLC](https://win-vector.com/2020/06/28/vtreat-for-knime/)) for example. Also the relevant KNIME workflow ([Techniques for Dimensionality Reduction – KNIME Community Hub](https://kni.me/w/7PBv1kGifxCng2qo)) which has a section about t-sne but I do not know if the methods are being stored in a way to reproduce them. The linked paper goes to a broken page … > **[Machine Learning Meta Collection (with KNIME) – mlauber71](https://hub.knime.com/mlauber71/spaces/Public/_machine_learning_meta_collection~--zu7353LMafrYQ_/current-state)** > > Machine Learning Meta Collection (with KNIME) This meta collection is about machine learning. It contains links to some examples demonstrating several types of… Also there is a (quite complete) set called “the poor man’s ML Ops” that I have built ([s\_601 - Sparkling predictions and encoded labels - "the poor man's ML Ops" (on a Big Data System) – KNIME Community Hub](https://kni.me/w/BwPL1ZF0AHxei8Q6)) and sort of explained in a video (in german), slides in english: [H2O.ai AutoML in KNIME for classification problems - #11 by mlauber71](https://forum.knime.com/t/h2o-ai-automl-in-knime-for-classification-problems/20923/11). This is specific to a big data environment but the priciples would apply. If you can do it on a laptop or small server vtreat or something similar might be more powerful. About dimension reduction there is also [featuretools](https://www.featuretools.com/) which I plan on building a KNIME workflow around for some time now … ---
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [November 29, 2022, 10:47am UTC](https://forum.knime.com/t/t-sne-how-to-use-in-production-pipeline/49365/6 "2022-11-29T10:47:19Z")
As an addition these articles from the KNIME low-code blog on Medium: Seven Techniques for Data Dimensionality Reduction > **[Seven Techniques for Data Dimensionality Reduction](https://medium.com/low-code-for-advanced-data-science/seven-techniques-for-data-dimensionality-reduction-1fe81d6174da)** > > A codeless KNIME solution to work with datasets with thousands of columns Three More Techniques for Data Dimensionality Reduction in ML > **[Three More Techniques for Data Dimensionality Reduction in ML](https://medium.com/low-code-for-advanced-data-science/three-new-techniques-for-data-dimensionality-reduction-in-ml-1378a4fe31a9)** > > Implementing LDA, Neural Autoencoder, and t-SNE in a codeless fashion ---
**Author:** ![system](https://forum-cdn.knime.com/uploads/default/original/1X/ab3ccf34482a0329361734a18199390177204f15.png) [@system](https://forum.knime.com/u/system)\ **Post date:** [February 27, 2023, 10:48am UTC](https://forum.knime.com/t/t-sne-how-to-use-in-production-pipeline/49365/7 "2023-02-27T10:48:13Z")
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.