SE::Startpage - startpage.com search results scraper
Overview of the scraper
The Startpage search results scraper. allows you to get large databases of links, ready for further use. You can use queries in the same format you enter them into the Startpage search bar, including search operators (site, inurl etc.).
A-Parser's functionality allows you to save the parsing settings for the Startpage scraper for future use (presets), ), set a parsing schedule, and much more. You can use automatic query multiplication, substitution of subqueries from files, iteration of alphanumeric combinations and lists to get the maximum possible number of results.
Results can be saved in the format and structure you need, thanks to the built-in powerful templating engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL and CSV.
Collected data
- Links, anchors, and snippets from the search results
Capabilities
- Supports selecting country, search language and page language
- Ability to output anchors, links, snippets together or separately
- Ability to specify the number of results to output
- Specifying the output size (10 or 20 results)
Use cases
- Collecting databases of links, anchors and snippets
- Getting a list of sites most frequently cited in search engines
- Any other use cases for getting information
Queries
Queries are words, phrases, and word combinations in the same format they are entered into the search engine. Example:
test
site:http://test.ru
red roses
Query substitutions
You can use built-in macros for query multiplication. For example, if we want to get a very large database of forums, we can specify a few main queries in different languages:
forum
forum
foro
论坛
In the query format, we specify iterating characters from a to zzzz, . This method allows for maximum rotation of search results and retrieval of many new unique results:
$query {az:a:zzzz}
This macro will create 475254 additional queries for each initial search query, which totals 4 x 475254 = 1901016 search queries. The number is impressive, but it's not a problem for A-Parser at all. At a speed of 2000 queries per minute, such a task will be processed in just 16 hours.
Using operators
You can use search operators in the query format, so it will be automatically added to every query in your list:
site:$query
Output results examples
A-Parser supports flexible results formatting thanks to the built-in templating engine Template Toolkit, which allows it to output results in any form, as well as in a structured format, for example CSV or JSON
Exporting a list of links
Links + anchors + snippets with position output
Outputting links, anchors and snippets into a CSV table
Saving in SQL format
Dump results to JSON
Results processing
A-Parser allows you to process results directly during scraping, and in this section we have provided the most popular use cases for the Startpage scraper
Link deduplication
Link deduplication by domain
Extracting domains
Removing tags from anchors and snippets
Filtering links by inclusion
Possible settings
| Parameter Name | Default Value | Description |
|---|---|---|
| Pages count | 5 | Number of pages to scrape (from 1 to 50) |
| Family filter | Filter depending on search | Select filtering level (Filter all results / Filter depending on search / Do not filter my results) |
| Period | Any time | Select output period (Any time / Past 24 hours / Past week / Past month / Past year) |
| Links per page | 10 | Output size (10 / 20) |
| Results language | English | Select language for results |
| Page language | English | Select page language |
| Search country | All | Select country for search origin |