imacros settings for Mac

[mcbenly]

Hi,

I wanted to automate some process from one specific URL.

I came across this post here to get the idea of imacros:

How to crawl the webpage which needs to login first using httpretriver

Hi all, I am trying httpretriver to download the content from a webpage, but what I got is the html which ask for login. I have already registed to this website. How should I do to get the correct content not the login requirement? Best regards

iMacros seems to be working with firefox. However, I am unable to configure in the Knime workflow (I am mac user). I am using the “External Tool” extension as per post suggest.

This seems to be the setting for Windows according to the post.

Could you please help me with the configuration? If you could include screenshot of your working flow, that will be great.

I am open to any alternative solution to run this marco.

Thanks,

[Kathrin]

Hi @mcbenly,

I haven’t use iMacros, but I have a small example for how to setup the External Tool node on Mac.

Below a screenshot for the settings for the external tool node on Mac for a small example workflow, which is available on the KNIME Hub.

The important settings are:

Input File - Where to write incoming data (from the Input Port)
Path to Executable- Path to the file to execute.
Execute in Directory- Where to run the executable (here you probably want to select the firefox application)
Command Line Arguments- Arguments to run the script with. (here I would copy the settings from the other post)
Output File from External Tool- Where the data from the executable is written

Was this helpful to find the correct settings to run your script?

Cheers
Kathrin

[mcbenly]

Thank you @Kathrin. Your response is really helpful.
However, I am still having issue with iMacros settings

Thanks,

[Kathrin]

Let’s try to find identify the error and find a solution together

Can you execute the iMacro script outside of KNIME?
What kind of error message do you get in KNIME Analytics Platform? (you can see the error message in the console on the lower right)

Cheers
Kathrin

[mcbenly]

I think the overall settings isn’t correct.
This is the first error for Firefox
“failed to apply settings: The specified external executable (’/Users/talpurb/Documents/Firefox.app’) must no be a directory.”

Perhaps try yourself running iMacros in Knime if you have mac? and share your configuration

Thanks,

[Kathrin]

Thanks for sharing your error message. I’m not familiar with iMacros and haven’t tried that, but I have an idea for the setting

If you right click on Firefox in the Finder and you select “Show package content” you can see the content of the folder Firefox.app. There should be something like Contents/MacOS/firefox

Can you please try to use this path for the setting option “Execute in directory”?

Cheers
Kathrin

[mcbenly]

Thanks @Kathrin
Now getting this error:
“failed to apply settings: The specified external executable (‘firefox’) doesn’t exist”
Is there any other way you can think of automating this process in Knime? other than imacros maybe?

My requirement is to go to the specific page and download the file periodically.

Thanks,

[Kathrin]

Hi @mcbenly,

hm, I have one more idea
Is it a public page? If yes, can you maybe share the link to the page?

If not - is it a page where you need to provide some credentials before you can download the file?

Cheers
Kathrin

[mcbenly]

Hi @Kathrin,

It is not a public page.

Yes it requires few steps before the file is downloaded. It’s a web-based CRM system. Inside this CRM, there is a SQL query page (within web). The SQL query page is static!

Here is the screenshot:

Just need to click the ‘Export Query As CSV’ to download.

But here are steps for macro that I recorded.
Step1: Go to the page (Password was saved in the browser and it didn’t ask me to login)
Step2: Click on the ‘Export Query As CSV’ to download.

Thanks,

[Kathrin]

Hi @mcbenly,

1. So in case the csv file is already there (and is not created when clicking on the button) you might be lucky and you can access the file via web crawling. Here you can find an example how to do so: Demonstrates how to download CSV data from a list of URLs – KNIME Hub

2. Can you let me know which web-based CRM system you are using?
   Maybe it has already an API, which would make it easier to access the dataset

3. If the CRM system doesn’t have an API and 1. doesn’t work, you might want to inspect the link behind the button. Here we you would see whether the button executes the SQL statement, or whether the csv file is already available.

Cheers
Kathrin

[qqilihq]

Hi there,

Wow. That original thread you posted above is from 2014 Quoting my former self from 2014:

Hi zwang,

we’re talking about form authentication here? We cannot handle this with our nodes currently.

Best,
Philipp

Many things have changed and this is fortunately no longer true You can do form encoded submits with the HTTP Retriever from Palladian just fine now – there’s a dedicated node to help you with that:

Form Encoded HTTP Entity Creator — NodePit

In the HTTP Retriever’s documentation you’ll find some details how to do form encoded requests.

For what you’re trying to do, the Selenium Nodes might be helpful too. They allow you to automate your browser from a KNIME workflow (and interact with a website just as a human being). As far as I understood, this is exactly what you’re trying to do here.

For any questions regarding Palladian or the Selenium Nodes just drop by in the appropriate subforum!

-Philipp

[mcbenly]

@qqilihq Thanks Philipp. Selenium is great!
Thanks again.

[qqilihq]

@mcbenly Happy to hear – don’t hesitate to get in touch if you have any questions