After encountering Cloudflare protection, I decided to try using Cloudscraper in KNIME. I installed Conda and realized I needed to install both BeautifulSoup and Cloudscraper. For BeautifulSoup, I was able to use a Conda command to install it successfully, but for Cloudscraper, there was no option other than installing it via pip. I read that I might need to create a new environment to use additional Python packages effectively. I’m unsure how to set up and import Cloudscraper for web scraping. Can you provide any assistance?
Hey there,
I have no clue about Cloudscraper, but I got you covered with how to create your own Python environment. I wrote an article about it a while back, but the concept has not changed:
When initially setting up your environment make sure to copy the correct command from the docs for the latest version (5.4):
https://docs.knime.com/latest/python_installation_guide/index.html#metapackages
After setting up the conda env, you can activate it and also install any packages via pip - you will be able to use them as well.
3 Likes
Wow! Thank you, Martin, for the great help. I learned how to create a new environment, and although Conda couldn’t install the cloudscraper package in the terminal after activating Conda, I realized I needed to activate the newly created environment first. Then, I was able to install the package using the pip command. The issue is now resolved. Thanks again!
1 Like
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.