# Scraping subcategories from main categories

**URL:** <https://forum.knime.com/t/scraping-subcategories-from-main-categories/19897>\
**Category:** Community Extensions\
**Tags:** palladian-selenium\
**Created:** [December 6, 2019, 8:23pm UTC](https://forum.knime.com/t/scraping-subcategories-from-main-categories/19897 "2019-12-06T20:23:32Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![V-P](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/v/ed655f/32.png) [@V-P](https://forum.knime.com/u/V-P)\
**Post date:** [December 6, 2019, 8:23pm UTC](https://forum.knime.com/t/scraping-subcategories-from-main-categories/19897/1 "2019-12-06T20:23:32Z")

</div>

Hi community,

I’m a newbie currently trying to scrape website ([https://dusseldorf.shopdutyfree.com/en/](https://dusseldorf.shopdutyfree.com/en/)) subcategories from its top categories, trying to do it dynamically as some categories don’t have any subcategories and also the number of existing subcategories varies per category.

The workflow is almost done, but I don’t get correct results, which I think has to do with the wrong loop node. For instance, I get subcategories skincare, make-up and fragrance listed across all categories (new, beauty, special offers etc), although they actually belong only to 1 category (beauty).

![image](https://forum-cdn.knime.com/uploads/default/original/2X/4/4161b9ba7114af5c90d97440ec3d2db133fed9de.png)

Does someone have an idea as to which loop node I should use or if there is another problem with the workflow? Thanks a lot.

 ![image](https://forum-cdn.knime.com/uploads/default/original/2X/e/e6a36eba5168d10081af8dae1087e5a56e6ace8b.png)

---

<div class="post-metadata">

**Author:** ![armingrudd](https://forum-cdn.knime.com/user_avatar/forum.knime.com/armingrudd/32/84633_2.png) [@armingrudd](https://forum.knime.com/u/armingrudd)\
**Post date:** [December 7, 2019, 7:25am UTC](https://forum.knime.com/t/scraping-subcategories-from-main-categories/19897/2 "2019-12-07T07:25:39Z")

</div>

Hi @V-P and welcome to the KNIME Forum,

You do not need loops at all.

Here is the workflow to extract the sub-categories for each category using the new [Webpage Retriever](https://nodepit.com/node/org.knime.rest.nodes.webpageretriever.WebpageRetrieverNodeFactory) node (you can also use the HTTP Retriever and HTML Parser nodes instead but in that case you need to add the namespace):

 ![webpage](https://forum-cdn.knime.com/uploads/default/original/2X/4/45e93ae9a9af6d6cb69f14c898ed39579bfc2c94.jpeg)

[webpage\_retriever.knwf](https://forum-cdn.knime.com/uploads/default/original/2X/c/cadff59570c7f30c3f32669e948aa1c525c82cbc.knwf) (486.1 KB)

😊

---

<div class="post-metadata">

**Author:** ![V-P](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/v/ed655f/32.png) [@V-P](https://forum.knime.com/u/V-P)\
**Post date:** [December 8, 2019, 11:04am UTC](https://forum.knime.com/t/scraping-subcategories-from-main-categories/19897/3 "2019-12-08T11:04:17Z")

</div>

Hi @armingrudd,

thank you very much! That is very helpful, works just as I wanted. Amazing how much shorter and more effective this new workflow ist.🙂

I have only 1 question left: could you please explain (maybe show in a screenshot) where you found the xpath? I think I found the first one for all top-categories

 ![image](https://forum-cdn.knime.com/uploads/default/original/2X/9/9233e148ce598d6cf8faf5a912a0b11b24de3c80.png) , at least I got the same results, but can’t find the second one. That would help a lot to build similar workflows.

---

<div class="post-metadata">

**Author:** ![armingrudd](https://forum-cdn.knime.com/user_avatar/forum.knime.com/armingrudd/32/84633_2.png) [@armingrudd](https://forum.knime.com/u/armingrudd)\
**Post date:** [December 8, 2019, 11:30am UTC](https://forum.knime.com/t/scraping-subcategories-from-main-categories/19897/4 "2019-12-08T11:30:14Z")

</div>

[https://blog.statinfer.com/how-to-get-the-content-of-a-web-page-in-knime/](https://blog.statinfer.com/how-to-get-the-content-of-a-web-page-in-knime/)

---

<div class="post-metadata">

**Author:** ![system](https://forum-cdn.knime.com/uploads/default/original/1X/ab3ccf34482a0329361734a18199390177204f15.png) [@system](https://forum.knime.com/u/system)\
**Post date:** [December 15, 2019, 11:30am UTC](https://forum.knime.com/t/scraping-subcategories-from-main-categories/19897/5 "2019-12-15T11:30:25Z")

</div>

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.