Sitemap Finder: Fallback to sitemap.xml if no robots.txt was found

Hi @qqilihq,

if the sitemap finder, which uses a robots.txt only, does not find one, I’d like to propose trying “/sitemap.xml” as a fallback.

Also, I happen to notice regularly that servers are configured poorly, responding with a non 404 so you might consider adding a check if the retrieved document is of the XML-type and maybe the status quo too.

The url I provided as a string didn’t had a protocol present. You might consider being more strict about URLs.

Overall however, a really nice quality of life improvement! Thanks for the treat :slight_smile:

Best
Mike

1 Like

:+1:

That totally makes sense- thank you for the feedback!

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.