Connecting to an external website to lookup certifications

We get reports of ID’s that have to be checked against an external website, the National Insurance Producer Registry. I am looking to see if any kind of automation could be added to this process via some kind of loop? I don’t think this will be possible but it’s always worth asking.

https://sbs.naic.org/solar-external-lookup/#_ga=2.104400579.250946743.1520454565-1652745966.1520454565

It requires the choosing of the ‘jurisdiction’, and then a ‘search type’

Then the ‘Entity type’ will be an individual, followed by the lookup of the license number.

Finally an ‘agreeing to terms and conditions’ selection, followed by search.

Thanks

Hi @Page0727 , it looks like you can submit all these values as GET.

For example, you can paste the following URL in a browser:
https://sbs.naic.org/solar-external-lookup/lookup?jurisdiction=IA&searchType=Licensee&entityType=IND&licenseType=PRO&lastName=test&firstName=test&licenseNumber=3443&npn=4344&residentLicense=Yes&licenseStatus=A&loaType=D&fein=te-&dbaName=test&businessCity=test&businessState=FL&businessZip=12345&mailingCounty=Buena%20Vista

It will do the search for you.

The checkbox for agreeing terms is just to enable the submit button, so it is assumed that you have agreed to the terms and conditions if you go to that url.

You can just replace the values of the variables in that URL for automation.

For example, you can set a template URL such as:
https://sbs.naic.org/solar-external-lookup/lookup?jurisdiction=##JURISDICTION##&searchType=##SEARCHTYPE##&entityType=##ENTITYTYPE##&licenseType=##LICENSETYPE##&...etc...&mailingCounty=##MAILINGCOUNTRY##

And replace these place holders (##JURISDICTION##, ##SEARCHTYPE##, etc) by the values that you want, and submit via a GET Request.

4 Likes

Hi @Page0727 , it looks like the URL is just the front-end of it, it’s not the “processor”.

I could not test properly before because I did not have any sample values to test with, but I tested with a much simpler search, basically do the 3 required entries, and just search for lastname “Smith”.

This generates the URL of:
https://sbs.naic.org/solar-external-lookup/lookup?jurisdiction=AL&searchType=Licensee&entityType=IND&lastName=Smith

And returns quite some data (more than 1 page).

However, if you do go to that URL directly, it will not actually give you the results, unless you click on Search. This means that the event of clicking the Search button triggers an asynchronous client-server-client call, sort of an AJAX call to fetch the data.

Looking at the webpage, it’s designed to make it difficult to check for that call as it’s all in unformatted javascript. My days of going through all of these are over, and it’s not something I enjoy. You may require a web integrator’s help to get that info for you. Once you see what’s being submitted, you can simulate the request with Knime.

I believe the answer lies in this file:
https://sbs.naic.org/solar-external-lookup/ruxitagentjs_ICA27SVfghjqrux_10223210811140219.js

4 Likes

I really appreciate you looking into this for me. Thanks!

No problem @Page0727 . An alternative could be using Selenium. The Palladian extension allows you to use Selenium with Knime. If you are not familiar with Selenium, please read about it first. It’s used for Web Automation. This is used a lot for QA testing of web applications.

3 Likes

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.