How to get content of a webpage that requires authentication Selenium

Community Extensions Palladian & Selenium
#1

Hi;
do you have a sample workflow with selenium yada knime in a website with authentication? I was a beginner, so I could not figure it out.

I am very happy if you send a sample workflow.

I am trying to login to a web site with username and password, then transfer the data from the json table to the csv file.

login with both username and password, I need to transfer the csv to the next table with the option I chose.

A simple example that will provide insight is very useful.

Thank you

#2

Hi,

please have a look at the “Facebook” workflow on the examples page:

Best,
Philipp

#3

image
image936×664 60.5 KB

Hello there;
I do not understand where I am doing wrong. I want to print the username and password for login but it does not. XPath is wrong, if you are a help

#4

Hi,

please post your WF if you want me to have a look.

– Philipp

#6

Selenium_TEST.knwf (28.3 KB)

#7

The reason is, that the content in the login window is shown within an <iframe>, and Find Elements will not operate into the <iframe>:

02

What you’ll have to do after opening the login window:

  1. Get the <iframe> using a Find Elements node (use e.g. an ID query iframe_login, or XPath //*[@id='iframe_login'])

  2. Use the Frame node, check ‘WebElement’ and select the <iframe> column from previous step

  3. Continue with your existing workflow

In case you need to ‘escape’ from the <iframe> later, you can use another Frame node with the option ‘Back to parent frame’ or ‘Default content’.

Does that help?

with selenium nodes upload files from pc to web browser
Change user agent in chrome through selenium nodes
Selenium Node - Help with Find Element Node
#8

Could you send me a sample workflow. I’ve done different experiments for the same url, but I’m a beginner and I can not figure out the exact way. A sample flow will help me a lot.

#9

Here’s the outlined workflow with the <iframe> handling as described.

– Philipp

Selenium_iframe_procurementmap.intracen.org.knwf (15.8 KB)

#10

@umutcankurt Any feedback?

#11

Hello there;
I had a little bit of work and I had a chance to try new. Unfortunately, the password and username are not entered and only remain on the login screen.

How many times have I tried it? I do not understand where the problem is

#12

We can provide password and username to google chorme page which comes to us with automatic form filling. ? Would not it be easier to login. Do we simplify the work steps?

#13

Hi there!

I do not really understand this one. Can you please go in detail? What do you mean with “are not entered and only remain on the login screen”? With the version of the WF posted above, your browser window should show the following after execution:

38
38862×668 78.2 KB

It should also enter YOUR_PASSWORD in the “Password” field, but this will be cleared when clicking the submit button. Now you should simply be able to update the flow variables with your valid credentials: Right click on the WF in the “KNIME Explorer”, open menu “Workflow variable administration”):

48

With correct credentials, the login should work. (I didn’t try this, b/c I obviously have no account on the mentioned site)

No. Auto-completion will not be available in the Chrome instance launched via Selenium.

#15
  1. Are the “Send Keys” node configured correctly?
  2. Are the text inputs in “Send Keys” actually populated from the flow variables? (see “Variables” tab in the node’s configuration)
  3. What happens if you enter the username/password directly in the “Send Keys” node? (instead of supplying it through variables)
  4. Do you see warnings and/or log output when executing the workflow?
  5. Is your browser up-to-date?
  6. Have you tried a different browser?
#16

  1. I think I have solved the problem of not working but it does not work in the meantime waiting on the password input screen and not stopping. Sometimes it does not work sometimes (it is strange, I do not understand that website is related to the website)

  2. the second topic to solve is to select the country from the menu after login and open the page of the selected countryTEST_Selenium_iframe_procurementmap.intracen.org.knwf (20.2 KB)
    image

#17

The procedure is quite similar to the one with the login 
:

  1. Locate the input field with a Find Elements node
  2. Enter text into the field with a Send Keys node
  3. In case the website requires that you select an entry from the menu, use a Find Elements node again to locate the corresponding menu entry
  4. Click on the entry using the Click node

As this is probably involves asynchronous logic, you’ll probably have to use the Find Element’s “Wait” functionality, or add an explicit Wait node in between.

– Philipp

#18

Thank you very much for all your support. Wait for the solution by adding the node. but compared to an unpredictable change.

XPath is now the correct path for the tentative information, but because the tentative numbers are constantly changing, XPath is now the correct path, but the tender-dialog-43 
 in the XPath is changed

Sample
// * [@ id = “tender-dialog-436349804”] / div [2] / div / div [1] / p / a

// * [@ id = “tender-dialog-436209111”] / div [2] / div / div [1] / p / a

I think I’m going to find a solution to this

#19

I’d recommend to use XPath’s contains function to select the dialog. This might do the trick:

//*[contains(@id, 'tender-dialog-')]

Above is a very simplified solution, and might also match too much, depending on the page structure. For more sophisticated strategies, have a look at this Stack Overflow question, e.g.

– Philipp

#20

Thank you, Philipp; I work exactly as I want for all the contributions and supports. :wink:

1 Like