# Big Data join

**URL:** <https://forum.knime.com/t/big-data-join/32912>\
**Category:** KNIME Analytics Platform\
**Created:** [May 13, 2021, 8:12am UTC](https://forum.knime.com/t/big-data-join/32912 "2021-05-13T08:12:31Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\
**Post date:** [May 13, 2021, 10:35am UTC](https://forum.knime.com/t/big-data-join/32912/3 "2021-05-13T10:35:31Z")

</div>

@alexandruradu welcome to the KNIME forum.

A few remarks:

- the local big data environment in my opinion is more for educational purposes than for real production tasks
- then you might want to try [Hive](https://hub.knime.com/mlauber71/spaces/Public/latest/kn_example_hive_school_of~1q-JwD0cwmuEkWpy) instead of Spark ([Being Lazy is Useful — Lazy Evaluation in Spark | by Lakshay Arora | Analytics Vidhya | Medium](https://medium.com/analytics-vidhya/being-lazy-is-useful-lazy-evaluation-in-spark-1f04072a3648)) since Spark will just try to load everything into RAM and process it there while Hive has the ability to use the hard drive, that might work better
- you might want to try the (new) KNIME [Joiner node](https://hub.knime.com/knime/extensions/org.knime.features.base/latest/org.knime.base.node.preproc.joiner3.Joiner3NodeFactory) that might also be suitable for this tasks
- You could try and combine this with the (new) columnar storage ([1](https://www.knime.com/blog/improved-performance-with-new-table-backend)|[2](https://forum.knime.com/t/best-memory-optimization-practices/32775/9)) feature along with other [performance enhancement measures](https://forum.knime.com/t/best-memory-optimization-practices/32775/6).

If you could use some sort of loop very much depends on the nature of your data. If the table that has to be joined is significantly smaller that the left a (chunk) [loop](https://hub.knime.com/mlauber71/spaces/Public/latest/_loops_and_flow_variables_knime~CET0X2z2VY9oKlYN#externalresources) might be an option. But this depends on the data. What kind of [duplicates](https://forum.knime.com/t/school-of-duplicates-and-how-to-deal-with-them/24164) would you expect to occur and how would you handle them?

---

_[View the full topic](https://forum.knime.com/t/big-data-join/32912)._