How to Search Multiple Websites at Once

Are you someone looking to search multiple sites on a regular basis? You could be a deal hunter looking to search multiple websites for the best price deals or a broker trying to find out the best house deals in your area. It would be practically impossible to visit each of these websites every day and find the best price. Creating a custom search engine could solve this issue. In this article, we will see various methods of creating a custom search engine that can help you search multiple websites.

[lwptoc]

Signup for ExpertRec

Using Expertrec custom search engine ($9 per month)

Expertrec adds a Google-like search box to your website using which you can search for multiple websites… The biggest advantage is that it doesn’t contain Ads.

Note: If your CMS is WordPress, Thinkific, or Shopify, then install the respective apps listed below:

For WordPress – WP Fastest Site Search
For Shopify – ExpertRec Smart Search Bar
For Thinkific – ExpertRec Smart Search

Before starting to make a custom search engine using Expertrec, make sure that the sites you are trying to search are live and not behind a login.

1. Go to https://cse.expertrec.com/?platform=cse.
2. Login with your Gmail id. 
3. Enter your website URL. (enter your website URL in the format https://www.yourwebsite.com).
4. Enter your sitemap URL. ( if you don’t have a sitemap, you can skip this step).
5. Go to Crawl->What to crawl->URLs to add more URLs.
6. Here you can add as many URLs as you like.
7. Initiate a website crawl. (Now Expertrec’s crawler will start crawling your website. This could take some time depending on the size and number of pages on your website).
8. Once the crawl is completed, you can change your search engine’s look and feel and the search engine’s ranking algorithm by using the control panel. If in case you want certain websites to rank higher than others, you can use this option. There are more advanced features like search weights, promotions, synonyms, and more.
   • the code contains two portions ( one within <script></script> and another piece of code <ci-search></ci-search>).
   • The <script></script> is just the JavaScript component and can be added using Google tag manager or using the code editor.
   • The <ci-search></ci-search> code adds the search box to your website. The place where you add this code determines the location of the search box. Go to the code section and add it to your website. You can skip this step if you just want to have the search demo and not add the search engine to any website. 
9. It is advisable to make the code changes in a test mode or development website before taking it live on your live/ production site.
10. As a final step, remove any website cache that you might have.
11. If you want to crawl the list of sites more frequently, you can go to recrawl frequency and adjust the crawl rate according to your needs. Please note that faster crawl speeds are available only in the premium plans.

Advantages of Using Expertrec for searching multiple sites

1. Control over which pages are crawled– You have very fine-tuned control over what is crawled. For example, you could only crawl pages that have /products/ in the URL.
2. No Ads. Compared to Google’s custom search, Expertrec shows zero ads in their search bar.
3. Affordable pricing plan.
4. Real-time support to help you make adjustments to your search algorithm.

Signup for ExpertRec

Using Google custom search

Using Google Custom Search, you will be able to search multiple websites. However, the number(of websites to search from) is limited to 10. If you are required to search more than ten websites, have a look at searching the entire web using Google custom search or Expertrec custom search engine (understandable, since If Google allowed everyone to search unlimited websites, it would be easier to launch a Google alternative).

1. Sign in to your Google custom search Control Panel using your Google Account (get an account if you don’t have one).
2. In the Sites to Search section, add the pages you want to include in your search engine. You can include any sites you want, not just the sites you own. You can include whole site URLs or individual page URLs. You can also use URL patterns. Here go to Add and click on “Add sites in bulk.
3. Enter a list of sites one per line.
   
4. The name of your search engine will be automatically generated based on the URLs you select. You can change this name at any time.
5. Select the language of your search engine. This defines the language of the buttons and other design elements of your search engine but doesn’t affect the actual search results.
6. Click Create.

Your basic search engine that searches through multiple sites is ready to use! To see more configuration options, go to the Control Panel. The control panel also has a preview window that allows you to test and fine-tune your search results.

Using the OR operator inside Google search

To search on Google and limit the results to some websites follow the steps below:

1. Go to Google and type your search as usual.
2. Before firing the search, add site:site1 OR site:site2, etc as many sites as you want.
3. Now do the search and see that the results have now been limited to site1 and site2.
4. You can also use the pip symbol “|” to separate the websites. 

Using Duckduckgo

Did you know that Duckduckgo has a custom search engine similar to Google Custom Search? 

To search multiple websites using this, go to https://duckduckgo.com/search_box and enter the site URLs separated by a comma under the site search section. If you have a list of URLs in an Excel sheet, you can use the concatenate function to append a comma at the end of each URL.

Under site search, enter the website URLs separated by commas. If you have the list of websites in an Excel sheet in separate rows, use the concatenate function to append a comma to every URL. This will save a whole lot of time.

Using the Yandex site search

In the first step, you define the search area (i.e., the set of pages covered by your search), search names, and other general properties. The only disadvantage of this method is that you won’t be able to enter more than 50 sites to search. If you are looking for more than 50 sites, try the Expertrec custom search engine.

Here are steps to follow to search multiple sites using Yandex:

1. Go to the Yandex site search creator.
2. Enter the technical name of your search.
3. Enter Search name in the Open search standard –This is the name your visitors can use to add your search as a browser search engine.
4. Add a family filter To exclude pages containing obscenities and pornography from the search area, set up a family filter. The filter settings are similar to the corresponding big search settings.
5. Add a search area Specify the search area, i.e., a list of sites to include in the search. You can specify up to 50 addresses of sites or their sections.
6. Click Add Sites and Exclusions List the addresses of sites or their subsections space-separated.
   

Signup for ExpertRec