Extract XHR Request Results

[nxfxcom]

Hello,

I know I asked this before but we did not get anywhere. I am trying to scrape a subscription site. The site uses multiple XHR requests with requests to populate tables. I can see the POST requests in the inspector. Is there a way to extract the JSON reply from these requests? RIght now I am looking at potentially using Fiddler or so to save them all while selenium is browing which is insane

Thank you SO MUCH

[qqilihq]

Hi nxfxcom,

do I understand correctly, that you’d like to just “record” all the request content while interacting with a page, in order to later parse it separately?

I once had actually a roughly similar idea, but never really pursued implementing it. Let me give it another thought.

In the meantime any input how that feature would look like from your perspective (i.e. from a workflow perspective) will be appreciated.

–PHilipp

[nxfxcom]

Hello,
I am sorry I missed this reply and still spending hours a day trying to solve this. So here is the ideal scenario that would be a game changer:

1. Open Selenium Navigate and Login to Instagram
2. Navigate to profile
3. When you do that it loads an XHR: https://www.instagram.com/api/v1/feed/user/11813420769/?count=12&max_id=3037418347411889284_11813420769
4. The XHR below response is JSON list of 12 posts
   
   
5. Scroll down the page which triggers a new call that gets a paginated results set
6. Trying to recreate these calls with changing headers etc is overkill, it would be amazing if there was a way to intercept the XHR response.
7. I found some options: Access Chrome's network tab (e.g. XHR requests) with Selenium · GitHub

Thank you

[qqilihq]

Understand - thank you. We’ll evaluate if this is something which can be integrated into the nodes. I’ll keep this post updated if there’s any news!

-Philipp

[nxfxcom]

Hello Phillip,

Wanted to follow up and see if you had any thoughts on this. Any workarounds?

Thank you

[qqilihq]

Hi nxfxcom,

I’ve built an internal prototype which is promising, and agree that this idea makes sense. However it will require some fundamental changes in the plugin’s internals. Thus we will schedule this for the next major 5.x release.

At the moment my main focus is on some other hot projects, thus this will take some time - feel free to drop me a reminder email to mail@seleniumnodes.com and we might be able to arrange some pre-release access, once the upcoming Selenium Nodes v5 has matured a bit more.

–Philipp

[qqilihq]

If you’re willing to give it a try, I feel that this is ready for testing now (see email address above)

[nxfxcom]

This is amazing! Absolutely fantastic feature! Will share some feedback soon, but a total game changer!

[qqilihq]

Thanks, I’ve also been using it heavily already myself, so thank you for the feature request