What are real search engine APIs like? (As an alternative)
You’re absolutely right about “directly calling search engine APIs,” but they’re often more complex and restrictive.
- Google Custom Search JSON API
This is an official API provided by Google, but it’s not a fully open web search API.
How it works: You first create a “programmable search engine” in the Google backend and specify the scope of websites you want to search (for example, you could add Xinhuanet, People’s Daily Online, 36Kr, and dozens of other websites). The API then searches only within this specified scope.
Pros:
Leverages Google’s powerful search capabilities.
Search scopes can be highly customized, resulting in more precise results.
Cons:
Limited free quota: Only 100 queries per day.
Complex configuration: Requires creating a project, enabling the API, obtaining a key, and setting a search engine ID, which is a tedious process.
Non-global search: Can’t search websites you don’t specify.
Conclusion: Suitable for searching content only on a specific set of websites you trust.
- Bing Web Search API
This is an API provided by Microsoft, and it’s a truly universal web search API.
How it works: It gives you results similar to searching in the Bing search box.
Pros:
Provides real, global web search results.
You can filter by time (for example, only see results from the past 24 hours).
Cons:
Fewer free calls: Usually included in the Azure free tier, you get a monthly quota of 1,000 calls, but with a rate limit of 3 calls per second. For individual developers, applying for and setting up an Azure account may be more complex than using the News API.
Mixed results: The returned results are generic web pages, and you’ll need to do a lot of cleaning to distinguish between news, blogs, forums, etc.
- Third-party commercial search APIs (such as SerpApi, Scale SERP)
These companies specialize in scraping results from search engines like Google and Bing and returning them to you in a simple JSON format.
Pros:
Extremely easy to use: You avoid all the hassle of scraping and anti-scraping yourself.
The results are stable and reliable.
Disadvantages:
Commercial and expensive: The free quota is very small (e.g., 100 queries per month), which is basically only enough for testing and cannot support a daily application.