Shop::Wildberries::Suggest - Wildberries Search Suggestion Scraper
Overview of the scraper
A scraper for search suggestions by keywords in Wildberries. Thanks to the Shop::Wildberries::Suggest scraper, you can automatically collect keyword databases from Wildberries marketplace search bar suggestions based on a query. Using the Shop::Wildberries::Suggest scraper, you can quickly and easily scrape Wildberries suggestions based on a query.
The Wildberries suggestion scraper solves one of the main SEO tasks, namely, the quick automated acquisition of an expanded semantic core. Wildberries search suggestions allow covering the maximum number of phrases, especially when using the Parse to level function.
Thanks to the multi-threaded operation of A-Parser, the query processing speed can reach 10000 queries per minute, which on average allows receiving about 20000-50000 results per minute.
You can use automatic query multiplication, substitution of subqueries from files, iteration of alphanumeric combinations and lists to obtain the maximum possible number of results. Using result filtering you can immediately clean the result, removing all unnecessary garbage (by using negative keywords or filtering by suggestion type).
A-Parser functionality allows saving the parsing settings of the Shop::Wildberries::Suggest scraper for further use (presets), setting a parsing schedule and much more.
Saving results is possible in the form and structure that you need, thanks to the built-in powerful templating engine Template Toolkit which allows applying additional logic to the results and outputting data in various formats, including JSON, SQL and CSV.
Collected data
- Number of results per query
- Suggestions for the query
- Suggestion type (suggest/category/brand)
Capabilities
- Ability to filter keywords during deep parsing
- Ability to use the Parse to level function
Use cases
- Collecting keyword databases
Queries
Search phrases should be specified as queries, for example:
adidas
T-shirts and tops
Xiaomi smartphone
Query substitutions
You can use built-in macros for automatic substitution of subqueries from files; for example, we want to add a list of other words to each query. Let's specify a few main queries:
discount
promotion
sale
In the query form, we will specify a macro for substituting additional words from the file Keywords.txt, this method allows increasing the variability of queries many times:
$query {subs:Keywords}
This macro will create as many additional queries as there are in the file for each initial search query, which in total will give [number of initial queries] x [number of queries in the Keywords file] = [total number of queries] as a result of the macro operation.
For example, if the file Keywords.txt contains the substitutions mentioned above, the result will be 9 queries:
adidas discount
T-shirts and tops discount
Xiaomi smartphone discount
adidas promotion
T-shirts and tops promotion
Xiaomi smartphone promotion
adidas sale
T-shirts and tops sale
Xiaomi smartphone sale
Examples of result output
A-Parser supports flexible result formatting thanks to the built-in templating engine Template Toolkit, which allows it to output results in an arbitrary form, as well as in a structured form, such as CSV or JSON
Exporting list of suggestions
Result format:
$results.format('$suggest\n')
Result example:
adidas sneakers
adidas for women
adidas men's sneakers
adidas terrex
adidas superstar
adidas originals
adidas terrex men's sneakers
Sneakers and trainers
Deodorants
adidas
Output of the query, suggestion for the query + suggestion type
Result format:
$query:\n$results.format('$suggest - $type\n')
Result example:
adidas sneakers - suggest
adidas for women - suggest
adidas men's sneakers - suggest
adidas terrex - suggest
adidas superstar - suggest
adidas originals - suggest
adidas terrex men's sneakers - suggest
Sneakers and trainers - category
Deodorants - category
adidas - brand
Output to CSV table
The built-in utility $tools.CSVLine allows creating correct tabular documents ready for import into Excel or Google Sheets.
Result format:
[% FOREACH i IN results;
tools.CSVline(i.suggest);
END %]
File name:
$datefile.format().csv
Initial text:
Suggestions
The Result Format uses the Template Toolkit templating engine to output elements of the $results array in a FOREACH loop.
In the result file name, you just need to change the file extension to csv.
For the "Initial text" option to be available in the Job Editor, you need to activate "More options". In the "Initial text," write the column names separated by commas and make the second line empty.
Saving in SQL format
Result format:
[% FOREACH results;
"INSERT INTO serp VALUES('" _ query _ "', '"; suggest _ "', '"; type _ "')\n";
END %]
Result example:
INSERT INTO serp VALUES('adidas', 'adidas sneakers', 'suggest')
INSERT INTO serp VALUES('adidas', 'adidas for women', 'suggest')
INSERT INTO serp VALUES('adidas', 'adidas men\'s sneakers', 'suggest')
INSERT INTO serp VALUES('adidas', 'adidas terrex', 'suggest')
INSERT INTO serp VALUES('adidas', 'adidas superstar', 'suggest')
INSERT INTO serp VALUES('adidas', 'adidas originals', 'suggest')
INSERT INTO serp VALUES('adidas', 'adidas terrex men\'s sneakers', 'suggest')
INSERT INTO serp VALUES('adidas', 'Sneakers and Trainers', 'category')
INSERT INTO serp VALUES('adidas', 'Deodorants', 'category')
INSERT INTO serp VALUES('adidas', 'adidas', 'brand')
Dump results to JSON
General result format:
[% result.${query} = {
totalcount = p1.totalcount
suggests = []
};
FOREACH i IN p1.results;
result.${query}.suggests.push(i.suggest);
END %]
Initial text:
[% result = {} %]
Final text:
[% result.json %]
Result example:
{
"adidas": {
"suggests": [
"adidas sneakers",
"adidas for women",
"adidas men's sneakers",
"adidas terrex",
"adidas superstar",
"adidas originals",
"adidas terrex men's sneakers",
"Sneakers and trainers",
"Deodorants",
"adidas"
],
"totalcount": 10
},
"T-shirts": {
"suggests": [
"women's summer T-shirts",
"men's summer T-shirts",
"children's T-shirts",
"sports T-shirts",
"T-shirts for boys",
"women's T-shirts",
"women's linen T-shirts turkey",
"T-shirts"
],
"totalcount": 8
}
}
For the "Initial text" and "Final text" options to be available in the Job Editor, you need to activate "More options".
Results processing
A-Parser allows processing results directly during parsing; in this section, we provide the most popular use cases for the Shop::Wildberries::Suggest scraper
Parse to level option
The option instructs the scraper to substitute the received results into the query queue in depth up to the specified level, for example:
- If level 1 is specified, the scraper will add all results received from the initial query to the queries
- If level 2 is specified, the scraper will add all results received from the initial query + all results received for queries from level 1
- etc.
Since the results are likely to contain duplicates, to prevent the scraper from doing unnecessary work (not scraping the same thing), it is recommended to enable query uniqueness (Unique queries)
Possible settings
| Parameter Name | Default Value | Description |
|---|---|---|
| Follow suggests | All | Selection of the type of suggestions to be substituted into queries when using Parse to level (All / Suggest / Tag / Brand), multiple selection supported |