SE::Startpage - Startpage.com search results parser
Overview
Startpage search engine results parser. Thanks to the Startpage parser, you can obtain large databases of links ready for further use. You can use queries in the same format as you enter them in the Startpage search bar, including search operators (site, inurl, etc.).
A-Parser functionality allows you to save Startpage parser settings for future use (presets), set parsing schedules, and much more. You can use automatic query multiplication, substitution of subqueries from files, brute-forcing of alphanumeric combinations and lists to obtain the maximum possible number of results.
Saving results is possible in the form and structure you need, thanks to the built-in powerful Template Toolkit template engine, which allows you to apply additional logic to results and output data in various formats, including JSON, SQL, and CSV.
Collected data
- Links, anchors, and snippets from the search results
Capabilities
- Supports selection of country, search language, and page language
- Ability to output anchors, links, and snippets together or separately
- Ability to specify the number of results to output
- Specification of result size (10 or 20 results)
Use cases
- Collecting databases of links, anchors, and snippets
- Obtaining a list of sites most frequently mentioned in search engines
- Any other use cases for obtaining information
Queries
Words, phrases, and word combinations are specified as queries in the same form as they are entered into the search engine. Example:
test
site:http://test.ru
red roses
Query substitutions
You can use built-in macros for query multiplication; for example, if we want to get a very large database of forums, we specify several main queries in different languages:
forum
forum
foro
论坛
In the query format, we specify a character brute-force from a to zzzz; this method allows for maximum rotation of search results and obtaining many new unique results:
$query {az:a:zzzz}
This macro will create 475254 additional queries for each original search query, which in total will give 4 x 475254 = 1901016 search queries—an impressive figure, but not a problem for A-Parser at all. At a speed of 2000 queries per minute, such a task will be processed in just 16 hours.
Using operators
You can use search operators in the query format, so they will be automatically added to each query from your list:
site:$query
Output results examples
A-Parser supports flexible result formatting thanks to the built-in Template Toolkit template engine, which allows it to output results in an arbitrary form, as well as in a structured one, such as CSV or JSON
Exporting a list of links
Links + anchors + snippets with position output
Outputting links, anchors, and snippets to a CSV table
Saving in SQL format
Dumping results to JSON
Results processing
A-Parser allows you to process results directly during parsing; in this section, we have provided the most popular cases for the Startpage parser
Link deduplication
Link deduplication by domain
Extracting domains
Removing tags from anchors and snippets
Filtering links by inclusion
Possible settings
| Parameter name | Default value | Description |
|---|---|---|
| Pages count | 5 | Number of pages to parse (from 1 to 50) |
| Family filter | Filter depending on search | Selection of filtering level (Filter all results / Filter depending on search / Do not filter my results) |
| Period | Any time | Selection of result period (Any time / Past 24 hours / Past week / Past month / Past year) |
| Links per page | 10 | Result size (10 / 20) |
| Results language | English | Selection of result language |
| Page language | English | Selection of page language |
| Search country | All | Selection of the country from which the search is performed |