SE::AOL::Suggest - AOL Search Suggestions Scraper
Scraper Overview
Scraper for search suggestions by keywords in AOL. Thanks to the SE::AOL::Suggest scraper, you can automatically collect keyword databases from AOL search engine suggestions by query. Using the SE::AOL::Suggest scraper, you can easily and quickly scrape AOL suggestions by query.
Thanks to the multithreading work of A-Parser, the speed of processing requests can reach 4700 requests per minute, which on average allows obtaining up to 13500-14500 results per minute.
You can use automatic query multiplication, substitution of subqueries from files, iteration of alphanumeric combinations and lists to get the maximum possible number of results. Using result filtering, you can immediately clean up the result by removing all unnecessary junk (using negative keywords).
A-Parser functionality allows you to save scraping settings of the SE::AOL::Suggest scraper for further use (presets), set a scraping schedule, and much more.
Saving results is possible in the form and structure that you need, thanks to the built-in powerful template engine Template Toolkit which allows applying additional logic to the results and outputting data in various formats, including JSON, SQL, and CSV.
Collected Data
- Number of results by query
- Suggestions for the query
Capabilities
- Support for all AOL search operators (site:, etc.)
- Scrapes the maximum number of results provided by AOL - 100 pages of 20 items each in the search results
Use Cases
- Collection of keyword databases
Queries
As queries, you need to specify search phrases, for example:
write essay
Football
Speak in english
forex
cheap essay
Query Substitutions
You can use built-in macros for automatic substitution of subqueries from files, for example, we want to add some list of other words to each query, let's specify several main queries:
essay
article
thesis
In the query format, we will specify a macro for substituting additional words from the file Keywords.txt, this method allows to increase the variability of queries many times over:
{subs:Keywords} $query
This macro will create as many additional queries as there are in the file for each original search query, which in total will give [number of original queries (domains)] x [number of queries in the Keywords file] = [total number of queries] as a result of the macro's work.
For example, if the file Keywords.txt contains:
buy
cheap
As a result, the substitution macro will turn 3 main queries into 6:
buy essay
cheap essay
buy article
cheap article
buy thesis
cheap thesis
Output Results Examples
A-Parser supports flexible result formatting thanks to the built-in template engine Template Toolkit, which allows it to output results in an arbitrary form, as well as in a structured one, for example, CSV or JSON
Exporting a List of Suggestions
Similar to SE::Google::Suggest.
Output in a CSV Table
Similar to SE::Google::Suggest.
Saving in SQL Format
Result format:
[% FOREACH results;
"INSERT INTO serp VALUES('" _ query _ "', '"; suggest _ "')\n";
END %]
Example of result:
INSERT INTO serp VALUES('write essay', 'write essay for me')
INSERT INTO serp VALUES('write essay', 'write essay')
INSERT INTO serp VALUES('write essay', 'write essay online')
INSERT INTO serp VALUES('write essay', 'write essay for you')
INSERT INTO serp VALUES('write essay', 'write essay free')
INSERT INTO serp VALUES('write essay', 'write essay conclusion')
INSERT INTO serp VALUES('write essay', 'write essay today')
INSERT INTO serp VALUES('write essay', 'write essays for money')
...
Dumping results to JSON
Similarly as in SE::Google::Suggest.
Results processing
A-Parser allows processing results directly during scraping, in this section we have provided the most popular cases for the scraper SE::AOL::Suggest
Parse to level option
Similarly as in SE::Google::Suggest.
Filtering results (using negative keywords)
Similarly as in SE::Google::Suggest.