OpenWebSearch: Why the EU wants to build a public web index by 2025
If you are looking for something on the web, you google it in the vast majority of cases, and rarely consult Microsoft’s search engine Bing. Some smaller search engines generate their result lists to a large extent using the web indexes from Google (Startpage) or from Bing (DuckDuckGo, Ecosia, MetaGer, Neeva, Qwant, You.com). After Yahoo merged into Bing, there are only two other offers of a comparable size in addition to the two American search indexes. These are Yandex from Russia and Baidu from China, and both have a much lower relevance in the western world.
Microsoft has just shown the search engine operators how big the dependency on Google and Bing is now. In February 2023, the group made its subscription models for the Bing API significantly more expensive, depending on the usage model by three to ten times the previous costs. In addition, scientists such as the American psychologist Robert Epstein have been warning of the Search Engine Manipulation Effect (PDF) since 2015, according to which dominant search engines are able to influence the formation of opinion in a democracy through their result ranking.
But what could you do with a large web index if it were freely available to the public? One could set up alternative search engines or specialized search services for selected topics. Users would have a free choice and could better protect their private user profiles. Linguists could use the data pool of a large web index to track how our language is evolving, and sociologists could observe how we engage with each other on social media. Web services could use it to look for indications of incipient pandemics or other disasters and thus set up an early warning system.
