Shop::Wildberries::ProductsList - scraper of Wildberries product list
Scraper Overview
Using the Wildberries product scraper, you can obtain product data from search results by keyword or from a list of products in a specific category or brand. It allows you to compile a database of product links, track price dynamics, changes in the number of reviews or ratings, collect product images, or information about available colors and sizes. Additionally, a separate list of sponsored ads is collected with the position of each in the organic search results. The scraper provides the ability to set the sorting of results and specify a pickup point.
The A-Parser functionality allows you to save scraping settings for future use (presets), set a scraping schedule, and much more. You can use automatic query multiplication, substitution of subqueries from files, permutation of alphanumeric combinations and lists to obtain the maximum possible number of results.
Collected Data
- Number of products
- Product name
- Product link
- Product image
- Brand
- Price and old price
- Rating and number of reviews
- Sizes and/or colors
- For sponsored ads - position in the product list
- List of keys from "People also search for" and "You may also like"
Use Cases
- Collecting product links
- Assessing product popularity
- Tracking price dynamics and product popularity
- Collecting an array of product images
- Determining positions of sponsored ads
Queries
You can use the following as queries:
Links to a brand's product list, for example:
https://www.wildberries.ru/brands/kristiano-fashion
https://www.wildberries.ru/brands/s-a-s
Links to a product list in a category, for example:
https://www.wildberries.ru/catalog/elektronika/razvlecheniya-i-gadzhety/igrovye-konsoli/playstation
https://www.wildberries.ru/catalog/avtotovary/shiny-i-diski/shiny
Keywords, just as if you were entering them directly into the Wildberries search form, for example:
xiaomi mi10
футболки и майки
магнитола
Query Substitutions
When using keywords as queries, you can use built-in macros for query multiplication, for example, if you need to get more results than the regular search output provides.
For example, for the aforementioned keywords in the query format, we will specify a character permutation from a to zzzz, this method allows you to maximally rotate the search output and obtain many new unique results:
$query {az:a:zzzz}
This macro will create 475254 additional queries for each original search query, which in total will give 4 x 475254 = 1901016 search queries, an impressive figure, but it's not a problem for A-Parser at all. With a speed of 2000 queries per minute, such a task will be processed in just 16 hours.
Output Results Examples
A-Parser supports flexible result formatting thanks to the built-in Template Toolkit, which allows it to output results in any form, as well as in structured formats, such as CSV or JSON
Output of brand, name, and price
Result format:
$items.format('$brand, $name, $price\n')
Example of result:
Xiaomi, Смартфон Poco X3 Pro / 6.67'' / 2400x1080 / IPS / 8 ГБ / 256 ГБ / 5160 мА*ч, 23751
Realme, Смартфон realme 8 / 6.4'' / 2400x1080 / Super AMOLED / 6 ГБ / 128 ГБ / 5000 mAh, 19911
Apple, Смартфон iPhone 11 128GB / 6.1'' / 1792x828 / Liquid Retina HD / 128 ГБ, 54990
Apple, Смартфон iPhone 12 128GB / 6.1'' / 2532x1170 / OLED / 128 ГБ, 69990
Samsung, Смартфон Galaxy A32 / 6.4'' / 2400x1080 / Super AMOLED / 4 ГБ / 128 ГБ / 5000 мАч, 19791
Output with variable names
Result format:
$items.format('Бренд: $brand, Название: $name, Цена: $price\n')
Example of result:
Бренд: Samsung, Название: Смартфон Galaxy A32 / 6.4'' / 2400x1080 / Super AMOLED / 4 ГБ / 128 ГБ / 5000 мАч, Цена: 19791
Бренд: Realme, Название: Смартфон realme 8 / 6.4'' / 2400x1080 / Super AMOLED / 6 ГБ / 128 ГБ / 5000 mAh, Цена: 19911
Бренд: Honor, Название: Смартфон Honor 50 / 6.57'' / 2340x1080 / OLED / 6 ГБ / 128 ГБ / 4300 мА*ч, Цена: 31490
Бренд: Apple, Название: Смартфон iPhone 13 256GB / 6.1'' / 2532x1170 / OLED / 256 ГБ, Цена: 89990
Бренд: Xiaomi, Название: Смартфон 6.53'' / 1600x720 / 2 ГБ / 32 ГБ / 5000 мА*ч, Цена: 8990
Output of the result with the parsing date in a CSV table
Result format:
[% USE d = date(format = '%Y %m %d %H:%M', locale = 'C');
FOREACH item IN items;
tools.CSVline(d.format(), item.name, item.price, item.link);
END %]
Example of result:
"2022 01 26 10:15","Смартфон 6.53'' / 1600x720 / 2 ГБ / 32 ГБ / 5000 мА*ч",8990,https://www.wildberries.ru/catalog/13615126/detail.aspx
"2022 01 26 10:15","Смартфон iPhone 13 128GB / 6.1'' / 2532x1170 / OLED / 128 ГБ",74390,https://www.wildberries.ru/catalog/40640907/detail.aspx
"2022 01 26 10:15","Смартфон Galaxy S21 256GB / 6.2'' / Dynamic AMOLED / 8 ГБ / 256 ГБ / 4000 мА*ч",64791,https://www.wildberries.ru/catalog/18592983/detail.aspx
"2022 01 26 10:15","Смартфон Galaxy M22 / 6.4'' / 720x1600 / Super AMOLED / 4 ГБ / 128 ГБ / 5000 мAh",17280,https://www.wildberries.ru/catalog/41501725/detail.aspx
Output of the name, price, and discount percentage of a product into a CSV table
Format of the result:
[% USE Math;
FOREACH item IN items;
discount = item.oldPrice ? (item.oldPrice - item.price) / item.oldPrice * 100 : 0;
tools.CSVline(item.name, item.price, Math.int(discount + 0.5) _ '%');
END %]
Example of result:
"Смартфон Poco X3 Pro / 6.67'' / 2400x1080 / IPS / 8 ГБ / 256 ГБ / 5160 мА*ч",23751,12%
"Смартфон realme 8 / 6.4'' / 2400x1080 / Super AMOLED / 6 ГБ / 128 ГБ / 5000 mAh",19911,17%
"Смартфон iPhone 12 128GB / 6.1'' / 2532x1170 / OLED / 128 ГБ",69990,0%
"Смартфон Galaxy A32 / 6.4'' / 2400x1080 / Super AMOLED / 4 ГБ / 128 ГБ / 5000 мАч",19791,10%
"Смартфон galaxy a52 / 6.5'' / 1080x2400 / Super AMOLED / 8 ГБ / 256 ГБ / 4500 мА*ч",28691,18%
Output into a CSV table
Format of the result:
[% FOREACH item IN items;
tools.CSVline(item.link, item.name, item.price, item.oldPrice, item.rating, item.reviews);
END %]
Initial text:
Ссылка на товар, Название товара, Цена, Старая цена, Рейтинг, Количество отзывов
Example of result:
Ссылка на товар, Название товара, Цена, Старая цена, Рейтинг, Количество отзывов
https://www.wildberries.ru/catalog/54067214/detail.aspx,"Шапка бини женская",703,1900,5,6
https://www.wildberries.ru/catalog/41415461/detail.aspx,"Шапка бини женская / бини с отворотом / Бини (beanie)",1487,1750,5,346
https://www.wildberries.ru/catalog/16782596/detail.aspx,"Шапка бини женская / бини с отворотом / Бини (beanie)",1487,1750,5,346
https://www.wildberries.ru/catalog/9468600/detail.aspx,"Шапка с шарфом / с перчатками / бини женская",2700,4500,5,137
https://www.wildberries.ru/catalog/4750212/detail.aspx,"Берет",765,1075,5,349
https://www.wildberries.ru/catalog/9793364/detail.aspx,"Берет с шарфом",3297,4710,5,307
Saving in SQL format
Format of the result:
[% FOREACH item IN items;
"INSERT INTO products VALUES('" _ item.name _ "', '"; item.link _ "', '"; item.price _ "', '"; item.brand _ "')\n";
END %]
Example of result:
INSERT INTO products VALUES('Смартфон 6.53'' / 1600x720 / 2 ГБ / 32 ГБ / 5000 мА*ч', 'https://www.wildberries.ru/catalog/13615125/detail.aspx', '8990', 'Xiaomi')
INSERT INTO products VALUES('Смартфон galaxy a52 / 6.5'' / 1080x2400 / Super AMOLED / 4 ГБ / 128 ГБ / 4500 мА*ч', 'https://www.wildberries.ru/catalog/23155682/detail.aspx', '24291', 'Samsung')
INSERT INTO products VALUES('Смартфон 6.53'' / 1600x720 / 2 ГБ / 32 ГБ / 5000 мА*ч', 'https://www.wildberries.ru/catalog/13615126/detail.aspx', '8990', 'Xiaomi')
Dumping results into JSON
Общий формат результата:
[% IF notFirst;
",\n";
ELSE;
notFirst = 1;
END;
obj = {};
obj.query = query;
obj.items = [];
FOREACH item IN p1.items;
obj.items.push({
link = item.link
name = item.name
price = item.price
});
END;
obj.json %]
Начальный текст:
[
Конечный текст:
]
Example of result:
[
{
"query": "смартфон",
"items": [
{
"link": "https://www.wildberries.ru/catalog/27379808/detail.aspx",
"name": "Смартфон realme 8 / 6.4'' / 2400x1080 / Super AMOLED / 6 ГБ / 128 ГБ / 5000 mAh",
"price": 19911
},
{
"link": "https://www.wildberries.ru/catalog/16023994/detail.aspx",
"name": "Смартфон iPhone 11 128GB / 6.1'' / 1792x828 / Liquid Retina HD / 128 ГБ",
"price": 54990
},
{
"link": "https://www.wildberries.ru/catalog/15875669/detail.aspx",
"name": "Смартфон iPhone 12 128GB / 6.1'' / 2532x1170 / OLED / 128 ГБ",
"price": 69990
},
...
]
},
...
]
To make the "Initial text" and "Final text" options available in the Task Editor, you need to activate "More options".
Possible settings
| Parameter | Default value | Description |
|---|---|---|
| Pages count | 5 | Number of pages to scrape |
| Sort by | Popularity | Sorting of results |
| Address | Pickup point address | |
| Longitude | Longitude of the pickup point | |
| Latitude | Latitude of the pickup point |
By default, Wildberries displays results for Moscow. The scraper has the ability to set a pickup point and the list of results will be tied to a specific location. To do this, you need to override 3 parameters: Address, Longitude, and Latitude. You can get the necessary values for these parameters in the browser:
1. Select the desired pickup point, having previously opened Developer Tools (in Chrome press F12)
2. In the Developer Tools on the Network tab, we look for the request saveprefereduserloc and in the request body, we see the necessary data
3. We copy them exactly into the scraper settings
