How to get Google's / YouTube's Autocomplete list with Selenium?

Hey Phil,

during a long :train2: ride I had some time to play with the workflow.

Generally: Really a cool idea scraping these search suggestions, approximately 1,238,236 ideas what to do with that came to my mind :blush:

Some specific recommendations regarding the workflow:

  • I noticed that when executing the workflow, sometimes the suggestions would show, sometimes not. Likely it’s due to the fact, that automaton “types” too fast, and thus the suggestions are not always triggered. There seems to be an easy fix for that though: Pushing the cursor down button explicitly triggers the menu. You can simply achieve this by adding another Send Keys node after the one which enters the query and which sends the key combination \uE015 (this code denotes the :arrow_down_small: button – see the node docs)

  • You had a Wait node to ensure that the suggestions are opened before taking the next step. “Dumb” waits have the disadvantage that you either wait for too long (b/c the results are there earlier), or not long enough (slow Wifi network, hello Deutsche Bahn). You can make this smarter by remove the Wait node and setting a conditional wait in the following node:

    image

    This will resume execution immediately, no matter if the suggestions appear :turtle: or :rabbit2: style.

  • I changed the CSS selector in the Extract Text to grab the suggestions from the auto-generated one to just ul[role='listbox'] which might be a bit more robust if the page structure should ever change.

  • I definitely like the idea of using the HTML Parser just to extract the results “statically”.

Here’s my modified workflow:

Did you have any success with the counterpart for Google, @kowisoft ?

–Philipp

3 Likes