Hello, everybody! Today we have a new Just KNIME It! challenge on data profiling.
Your company is planning to upgrade its office setup with new chairs, desks, monitors, and other essentials. To help with the decision, someone scraped product details and reviews to start an analysis, but before trusting this data you’ve been asked to assess its quality. Your task is to evaluate two datasets (product details and product reviews) and create a data quality profile for each. Can you trust this scraped data?
Here is the challenge. Let’s use this thread to post our solutions to it, which should be uploaded to your public KNIME Hub spaces with tag JKISeason4-28 .
Need help with tags? To add tag JKISeason4-28 to your workflow, go to the description panel in KNIME Analytics Platform, click the pencil to edit it, and you will see the option for adding tags right there. Let us know if you have any problems!
Just to confirm for Challenge 28, is there only one dataset file available right now? Because the task says we’re supposed to evaluate two datasets (product details and product reviews).
Do the measures of Conformity, Duplicate and Completeness per data. I wanted to avoid doing the calculation on the full data as the product will appears many times once joined with the reviews. Had to process the data and do some pivot to get the percent,
Filter the data based on these criteria and keep “good data only”
Join the products and reviews to have a final full table.
Visualize the results
This is a simple visu, only showing number in tables. It could be improved.
I decided for this one to show the stats per dataset - knowing the quality of each dataset allows the data scientist to improve the part that is not perfect. And when possible, showing the measure per column - again, only for some of them for the exercise. It could be extended way more.
Hello, everybody! Today we have a new Just KNIME It! challenge on data profiling.
Your company is planning to upgrade its office setup with new chairs, desks, monitors, and other essentials. To help with the decision, someone scraped product details and reviews to start an analysis, but before trusting this data you’ve been asked to assess its quality. Your task is to evaluate two datasets (product details and product reviews) and create a data quality profile for each. 
Can you trust this scraped data?
Need help with tags? To add tag JKISeason4-28 to your workflow, go to the description panel in KNIME Analytics Platform, click the pencil to edit it, and you will see the option for adding tags right there. 
Let us know if you have any problems!
