[Selenium]: Wait Node - Add option to wait for element

[mwiegand]

Hi @qqilihq,

when UIs are slow I resort to using the find node to search for an element in conjunction with the wait option. Though, that necessitates to remove the added column of the found element.

The use case really just is to wait until an element is present indicating the website has fully loaded.

I’d like to suggest extending the regular wait node by some of the options from the find node like:

1. Selector Option: CSS, XPath etc.
2. Wait Options: Duration and check interval

Best
Mike

[ArjenEX]

If my memory serves me well this feature is currently on their roadmap but +1 regardless.

[qqilihq]

Makes sense. Next update. Word!

[mwiegand]

*CLICK* Nice

[mwiegand]

PS: It just flashed up in my mind to add an option to invert the result, i.e. wait until an element is not found. This correlates to some degree with the other feature suggestion of mine:

[Selenium] Click Node: Element not clickable at point, other element would receive the click

Hi @qqilihq, since recently I face the below error. I assume something in the interface changed to the worse: 2024-08-09 21:34:43,492 : ERROR : KNIME-Worker-2801-Click 3:1126:0:972:0:1156:0:1155:0:1140 : : Node : Click : 3:1126:0:972:0:1156:0:1155:0:1140 : Execute failed: element click intercepted: Element <a>...</a> is not clickable at point (732, 583). Other element would receive the click: <div class="current">...</div> ws.palladian.nodes.selenium.SeleniumNodeExecutionException: element cl…

Though, I don’t want to ask for too much … just a suggestion

[qqilihq]

@mwiegand

Though, I don’t want to ask for too much … just a suggestion

(stay tuned)

-Philipp

[mwiegand]

FYI and maybe for some sort of inspiration. I have put this JavaScript into place to leverage the Web Animations API.

So far my tests were positive but I did not manage to actually retrieve the callback value true, false or error.

/**
 * Add your JavaScript code below:
 */
const callback = arguments[arguments.length - 1]; // last argument is callback

try {
  // Select the element with the id "master"
  const elementMasterOverlay = document.querySelector('#master');

  // Function to handle transition end event for the "width" property
  function handleTransitionEnd(event) {
    // Check if the transitioned property is 'width'
    if (event.propertyName === 'width') {
      // Log the completion of the width transition
      console.log('Width transition finished!', event);

      // Remove the event listener to prevent memory leaks
      elementMasterOverlay.removeEventListener('transitionend', handleTransitionEnd);

      // Resolve the callback with "true" since the width transition is complete
      callback(null, 'true');
    } else {
      // Resolve the callback with "false" if it's not the width transition
      callback(null, 'false');
    }
  }

  // Set up the transitionend event listener specifically for the width property
  if (elementMasterOverlay) {
    elementMasterOverlay.addEventListener('transitionend', (event) => {
      if (event.propertyName === 'width') {
        handleTransitionEnd(event);
      }
    });
  } else {
    // If the element is not found, throw an error to be caught
    throw new Error('Element with ID "master" not found');
  }
} catch (error) {
  // Catch any errors and resolve the callback with "error"
  console.error(error);
  callback(null, 'error');
}

[qqilihq]

Hi Mike,

cannot check the code in detail at the moment, but here’s some advice which might help:

1. The last entry in the Arguments list called [asyncCallbackMethod] must be enabled.

2. The callback allows one single argument, so callback(null, true) will not give you the second argument in the result table - if you need more than one result value, use the List or JSON type. You could e.g. do like this and use the JSON return:

   callback({ result: true });
   callback({ result: false  });
   
   // can usually not serialize entire Error object, 
   // so specify your properties manually, e.g. message
   callback({ error: error.message  });
   

Hope that helps!

-Philipp

[mwiegand]

Thanks for your feedback. asyncCallbackMethod is enabled. The callback is only triggered during either if, else statement or the catch statement. Though, I noticed a warning which, despite me slowing down the scraping to human levels, could cause some issues which led me to submit this feature request:

Throttling navigation to prevent the browser from hanging. See . Command line switch --disable-ipc-flooding-protection can be used to

[qqilihq]

@mwiegand, @arjenex, and others with who I have been in touch with:

In Selenium Nodes version 5.9 which we released a few days ago there’s a new section Conditions which provides nodes for all kind of conditional wait constructs.

Also tagging you here @arjenex as I’m not sure if my recent email made it through: It addresses your use case of letting a workflow wait until an element on the page disappears using the Number of Elements and Wait for Condition node.

Or another typical use case: Keep paginating a result list and click the “Next” link until the end using a conditional loop, case switch and flow variables:

Currently there’s several conditions available:

• Alert is Present
• Attribute Matches
• Number of Elements
• Text Matches
• URL Matches

… and you can “and”, “or“ or “not” combine them using the corresponding nodes:

I hope this will be useful - let me know if you have feedback or ideas for additional conditions!

Links to the download sites:

• KNIME 4.7: https://download.seleniumnodes.com/4.7
• KNIME 5.2: https://download.seleniumnodes.com/5.2
• KNIME 5.3: https://download.seleniumnodes.com/5.3

-Philipp

[mwiegand]

Hi @qqilihq,

that are some phantastic news. Once I am out of the hostial with my daughter, I will take some time incorporating these to replace the JavaScript based fallback.

Thanks again so much for your dedication!

Cheers
Mike

[qqilihq]

… and my best get-well wishes!

-Philipp

[ArjenEX]

Thanks for the heads-up and feature! That email has indeed not reached me since I switched jobs in between.