Question Help creating custom search for website

[Marc TF]

Hi everyone

I’ve come across a problem and I was hoping someone here might be able to recommend a solution.

I need to create a custom search for a website and I need the search to look for results from multiple domains.

Eg. Someone searches for “Keyboard” and search function returns results from many different domains where the keyword “Keyboard” was found. Of course, I want to specify the list of domains to search.

My problem is that most of the options I am finding have hard limits on the number of domains I can search through.


I’ve already created all this for free using Google CSE - but when domains increased over 100, I found that 100 is the limit. After 100, domains, the search results get messed up.

I then created this again using a paid subscription for Bing Custom Search API but again.. Once my number of domains requirement exceeded 400 - I found that there’s a hard limit of 400 domains.

If someone could suggest any other cost effective methods on how to get this done, I would really appreciate it! Thanks


Cheers,
Marc

 

[mxnerd]

Googled.

Google Custom Search: Retrieving More Than 100 Results for Comprehensive Searches

Discover how to expand the limit of Google Custom Search results beyond 100 for thorough and extensive searches. Learn the techniques and methods to configure your Google Custom Search setup to retrieve a higher number of relevant results, enabling deeper exploration and analysis. Enhance your...

blog.expertrec.com

https://stackoverflow.com/questions/34031557/how-to-get-more-than-100-results-from-google-custom-search-api

404 Page

www.xspdf.com

 

[Marc TF]

Googled.

Google Custom Search: Retrieving More Than 100 Results for Comprehensive Searches

Discover how to expand the limit of Google Custom Search results beyond 100 for thorough and extensive searches. Learn the techniques and methods to configure your Google Custom Search setup to retrieve a higher number of relevant results, enabling deeper exploration and analysis. Enhance your...

blog.expertrec.com

https://stackoverflow.com/questions/34031557/how-to-get-more-than-100-results-from-google-custom-search-api

404 Page

www.xspdf.com

Hi @mxnerd
Thanks for your answer.

https://stackoverflow.com/questions/34031557/how-to-get-more-than-100-results-from-google-custom-search-api

This link seems to confirm what I said in my original post - which is that Google CSE had a limit of 100 domains: "(b) cannot be solved at all, it seems to be a feature of the CSE product "

404 Page

www.xspdf.com

This one doesn't seem to be a solution at all. It addresses the issue of Google CSE limit of only offering 100 results.. which is different to the issue I have which is that I need to search through more domains.

Google Custom Search: Retrieving More Than 100 Results for Comprehensive Searches

Discover how to expand the limit of Google Custom Search results beyond 100 for thorough and extensive searches. Learn the techniques and methods to configure your Google Custom Search setup to retrieve a higher number of relevant results, enabling deeper exploration and analysis. Enhance your...

blog.expertrec.com

This I will look into. I was actually aware that Google didn't used to have a 100 domain limit in the earlier days and in fact introduced it later. I suspect that this solution gives access to an old version of the Google software through a 3rd party app. I'll look into this further and update. Thanks!

 

[Marc TF]

Hi @mxnerd
Thanks for your answer.

https://stackoverflow.com/questions/34031557/how-to-get-more-than-100-results-from-google-custom-search-api

This link seems to confirm what I said in my original post - which is that Google CSE had a limit of 100 domains: "(b) cannot be solved at all, it seems to be a feature of the CSE product "

404 Page

www.xspdf.com

This one doesn't seem to be a solution at all. It addresses the issue of Google CSE limit of only offering 100 results.. which is different to the issue I have which is that I need to search through more domains.

Google Custom Search: Retrieving More Than 100 Results for Comprehensive Searches

Discover how to expand the limit of Google Custom Search results beyond 100 for thorough and extensive searches. Learn the techniques and methods to configure your Google Custom Search setup to retrieve a higher number of relevant results, enabling deeper exploration and analysis. Enhance your...

blog.expertrec.com

This I will look into. I was actually aware that Google didn't used to have a 100 domain limit in the earlier days and in fact introduced it later. I suspect that this solution gives access to an old version of the Google software through a 3rd party app. I'll look into this further and update. Thanks!

Google Custom Search: Retrieving More Than 100 Results for Comprehensive Searches

Discover how to expand the limit of Google Custom Search results beyond 100 for thorough and extensive searches. Learn the techniques and methods to configure your Google Custom Search setup to retrieve a higher number of relevant results, enabling deeper exploration and analysis. Enhance your...

blog.expertrec.com

I had a look at this option and looks like it's an option that would work - but it's a bit pricey @ 100$ per month (currently paying <10$ / month for the 400 domains with Bing.

So I'm still looking for more options if anyone knows of any

Thanks

 

[mxnerd]

Use offline website downloader to download the 400 websites (update periodically) to your machine, then use html indexing software or write your own website crawler software, or use some full text search engine?

15 Free Full-Text Search Engine Solutions for developers

Full-Text Search is a technical term referred to advanced linguistic text query for a database or text documents. The search engine examines all the words stored in a document as it tries to match certain search criteria giving by the user. Many web websites depend on Full-text search to perform

medevel.com

 

[Marc TF]

Use offline website downloader to download the 400 websites (update periodically) to your machine, then use html indexing software or write your own website crawler software, or use some full text search engine?

15 Free Full-Text Search Engine Solutions for developers

Full-Text Search is a technical term referred to advanced linguistic text query for a database or text documents. The search engine examines all the words stored in a document as it tries to match certain search criteria giving by the user. Many web websites depend on Full-text search to perform

medevel.com

Thanks a million @mxnerd. I'm going to seriously look into this option!