SE::DuckDuckGo::Images - scraper of images

Overview of the scraper

Scraper of DuckDuckGo search results images. Thanks to the SE::DuckDuckGo::Images scraper, you can obtain databases of image links or images ready for further use. You can use queries in the same way you enter them in the DuckDuckGo search bar.

The functionality of A-Parser allows you to save DuckDuckGo scraping settings for future use (presets), set a scraping schedule, and much more. You can use automatic query multiplication, substitution of subqueries from files, permutations of alphanumeric combinations and lists to obtain the maximum possible number of results.

Saving results is possible in the form and structure that you need, thanks to the built-in powerful template engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.

Go to DEMO Buy A-Parser Pro ($299)

Use cases for the scraper

Downloading images by link

A-Parser allows you to use a chain of tasks, upon completion of the first, the second will start, with the links from the first task being used as queries for the second one.

Download example

How to import an example into A-Parser

eJyNVk1T2zAQ/SuMJofQhsQcevGFCdC0dCihEE4hnVHjtSuQJSPJKYzJf+9KMv6q
E3rIjLXaL2nfe0pBDNWP+lqBBqNJuCxI5r5JSM7z9aP9fZEHLKUJ6INI/hFc0ggU
GZGMKg3KxizJ7ecwrN3D8ML5o1MEMc25IavViGBe/NQzqVJq8w+y43FZrNq8pRtY
SNyMGYfaPMPVFU3BRkXUgN0dxy7R8HBsnm0GGkXMMCko9xVsZ3XVO8GechuvjWIi
QX9cKgZ6pmSKZgMuiTW+vHW4JAO3Jpgmd/E/fAwJY8o1jIjGdmcUm4m6O8yAokaq
eWZ7QntBpJhyfgkb4LWby3+aM453qqcxBl2Ugf0u839ybKsjNkttQP1R2EOVxa1O
59/rqEheygRPHv3Cc3OWMoNrfSZzYYcToPERIKvu7UqiJZUKqjJG5VAVRwRlICJ0
rKc2zWpT6xStybSNaylilsyxf8UiePPMxQJhOhdnMs042GMRD7GD8yYkcw03NWCm
uhyKXVTtdlOduYL2HkqojoiRkutvt77xTDHE4yfbborX2uyhvNo15fzu5rLVXY0v
m1kmsJZMEOtrIJEIKTzXdlS0SHQFJgy/LhbXDd6gi4IEnjEL3owBYY7MSwbh+MOJ
4+RkmInk9SGDk+Q1YfGhu3z0X6ATxjDLC6XoS0keezi/s861kam/rWpiaP8N1A3j
zVQmijltsNRSw/LFHWFDuQOIkALq/MhQ42YiHVws8UDoZiVPOO9XVhHSwFNOOdk2
9aLmvpehiWflWOTp0aAov9FhO7aK4tP2IEbknO8Aeh+Ge+Xhf9CzF4RBzTsh3xWt
rhruQFoTVP0yi5pJe0hYEC1ztbZpvNBZ7Nvh2uskq1GFvclw+XOy+nh4fz8enoRt
yA16MFeCwIdvV6P6UemjbY9cdOQ26BOjipk9Khw0+deldEvFgh1C1H033LjeU/Vg
v6J3t1tqHmz7NCbY8yb1a+W+xyBoPgS2oJsB3vmxG5IXo+rPQLHrXQ8LDK3lCpe4
ftDXPt4ivfTBytqx/3j7F6rw6z8=

Capabilities

Selection of number of pages for scraping
Search by region
Selection of search results language
Selection of safe search
Indication of image size
Selection of image type
Layout selection
Color selection

Collected data

Links to images
Image anchors
Links to pages
Height and width
Links to previews

Use cases

Collecting images for filling your blogs, tubes, doorways...
Collecting avatar databases

Queries

As queries, you need to specify search phrases, for example:

Audi  
Box  
Byron  
hunting and fishing

Query substitutions

You can use built-in macros for query multiplication, for example, if we want to get a very large database of forums, we will specify several main queries in different languages:

forum
форум
foro
论坛

In the query format, we will specify a permutation of characters from a to zzzz, this method allows to maximally rotate the search results and obtain many new unique results:

$query {az:a:zzzz}

This macro will create 475254 additional queries for each original search query, which in total will give 4 x 475254 = 1901016 search queries, an impressive figure, but this is not a problem for A-Parser. At a speed of 2000 queries per minute, such a task will be processed in just 16 hours.

Output results examples

A-Parser supports flexible formatting of results thanks to the built-in template engine Template Toolkit, which allows it to output results in any form, as well as in structured ones, for example CSV or JSON

Default output

Result format:

$serp.format('$link\n')

Example of result:

https://viralcats.net/blog/wp-content/uploads/2017/12/Mean-looking-cat-Viral-Cats-03.jpg
http://mymodernmet.com/wp/wp-content/uploads/2017/03/gabrielius-khiterer-stray-cats-8.jpg
http://fishsubsidy.org/wp-content/uploads/2020/01/abyssinian-cats.jpg
https://cdn2.theweek.co.uk/sites/theweek/files/2017/11/131117-wd-cats.jpg
https://www.israelhayom.com/wp-content/uploads/2020/04/why-cats-are-best-pets-worshipped-animals-1559234295.jpg
https://s-i.huffpost.com/gen/964776/images/o-CATS-KILL-BILLIONS-facebook.jpg
https://external-preview.redd.it/gxbKXOj-OF1_RSHa7Ncp8Gs_OFFP5i6V7SU5DPT2t1E.jpg?auto=webp&s=b6e85ba0f1517dc629d21208a7d9db992d550ba9
http://www.zastavki.com/pictures/originals/2013/Animals_Cats_Sleeping_gray_kitten_036760_.jpg
http://mcdaniel.hu/wp-content/uploads/2015/01/6784063-cute-cats-hd.jpg
https://img.webmd.com/dtmcms/live/webmd/consumer_assets/site_images/article_thumbnails/reference_guide/why_cats_sneeze_ref_guide/1800x1200_why_cats_sneeze_ref_guide.jpg
http://www.zastavki.com/pictures/originals/2013/Animals___Cats_Silver_beautiful_Scottish_Fold_cat_045199_.jpg

Output in CSV table

Result format:

[% FOREACH item IN serp;
    tools.CSVline(query, item.link, item.width, item.height,  item.page, item.thumb);
END %]

Example of result:

cats,https://viralcats.net/blog/wp-content/uploads/2017/12/Mean-looking-cat-Viral-Cats-03.jpg,462,722,https://viralcats.net/blog/2017/12/30/10-kitties-that-you-dont-want-to-mess-with/,https://tse2.mm.bing.net/th?id=OIP.AdkhgipoWbJwiQBp9VIWpgAAAA&pid=Api
cats,http://mymodernmet.com/wp/wp-content/uploads/2017/03/gabrielius-khiterer-stray-cats-8.jpg,750,1028,https://mymodernmet.com/gabrielius-khiterer-stray-cat-photos/,https://tse2.mm.bing.net/th?id=OIP.ZjfS8JQc9sahsK0-w8dRFAHaKJ&pid=Api
cats,http://fishsubsidy.org/wp-content/uploads/2020/01/abyssinian-cats.jpg,1204,1445,http://fishsubsidy.org/category/cat/cat-breeds/,https://tse3.mm.bing.net/th?id=OIP.uHEu4-5TLJ6SSgDree6ahQHaI4&pid=Api
cats,https://cdn2.theweek.co.uk/sites/theweek/files/2017/11/131117-wd-cats.jpg,1400,788,https://www.theweek.co.uk/94877/why-are-so-many-australian-towns-introducing-cat-curfews,https://tse3.mm.bing.net/th?id=OIP.iYyPimFLj1_wgKEsTsggQgHaEK&pid=Api
cats,https://www.israelhayom.com/wp-content/uploads/2020/04/why-cats-are-best-pets-worshipped-animals-1559234295.jpg,2119,1415,https://www.israelhayom.com/2020/04/23/2-nyc-cats-test-positive-for-coronavirus-officials-recommend-pet-precautions/,https://tse1.mm.bing.net/th?id=OIP.U7274nc_llbuQTChXpKVNgHaE8&pid=Api
cats,https://s-i.huffpost.com/gen/964776/images/o-CATS-KILL-BILLIONS-facebook.jpg,1536,1536,https://www.huffingtonpost.com/2013/01/30/domestic-cats-kill-billions-mice-birds-annually-study_n_2575833.html,https://tse1.mm.bing.net/th?id=OIP.ETFxELWtgKQwMlcoccq-SAHaHa&pid=Api
cats,https://external-preview.redd.it/gxbKXOj-OF1_RSHa7Ncp8Gs_OFFP5i6V7SU5DPT2t1E.jpg?auto=webp&s=b6e85ba0f1517dc629d21208a7d9db992d550ba9,1920,2560,https://www.reddit.com/r/cats/comments/2k2pio/my_very_ugly_cat/,https://tse1.mm.bing.net/th?id=OIP.t2BxlpEwcGrXJJQSToWVBAHaJ4&pid=Api
cats,http://www.zastavki.com/pictures/originals/2013/Animals_Cats_Sleeping_gray_kitten_036760_.jpg,2560,1600,http://www.zastavki.com/eng/Animals/Cats/wallpaper-36760.htm,https://tse4.mm.bing.net/th?id=OIP.3c_ISLWidlMWXHfjqkpB2wHaEo&pid=Api
cats,http://mcdaniel.hu/wp-content/uploads/2015/01/6784063-cute-cats-hd.jpg,2560,1600,http://mcdaniel.hu/cat-adoption-101/,https://tse4.mm.bing.net/th?id=OIP.QdEkrZjd1c_VN_aUtleoFgHaEo&pid=Api

Saving in SQL format

Result format:

[% FOREACH serp;
    "INSERT INTO serp VALUES('" _ query _ "', '"; link _ "', '";  page _ "', '"; thumb _ "')\n"; 
END %]

Example of result:

INSERT INTO serp VALUES('cats', 'https://viralcats.net/blog/wp-content/uploads/2017/12/Mean-looking-cat-Viral-Cats-03.jpg', 'https://viralcats.net/blog/2017/12/30/10-kitties-that-you-dont-want-to-mess-with/', 'https://tse2.mm.bing.net/th?id=OIP.AdkhgipoWbJwiQBp9VIWpgAAAA&pid=Api')
INSERT INTO serp VALUES('cats', 'http://mymodernmet.com/wp/wp-content/uploads/2017/03/gabrielius-khiterer-stray-cats-8.jpg', 'https://mymodernmet.com/gabrielius-khiterer-stray-cat-photos/', 'https://tse2.mm.bing.net/th?id=OIP.ZjfS8JQc9sahsK0-w8dRFAHaKJ&pid=Api')
INSERT INTO serp VALUES('cats', 'http://fishsubsidy.org/wp-content/uploads/2020/01/abyssinian-cats.jpg', 'http://fishsubsidy.org/category/cat/cat-breeds/', 'https://tse3.mm.bing.net/th?id=OIP.uHEu4-5TLJ6SSgDree6ahQHaI4&pid=Api')
INSERT INTO serp VALUES('cats', 'https://cdn2.theweek.co.uk/sites/theweek/files/2017/11/131117-wd-cats.jpg', 'https://www.theweek.co.uk/94877/why-are-so-many-australian-towns-introducing-cat-curfews', 'https://tse3.mm.bing.net/th?id=OIP.iYyPimFLj1_wgKEsTsggQgHaEK&pid=Api')
INSERT INTO serp VALUES('cats', 'https://www.israelhayom.com/wp-content/uploads/2020/04/why-cats-are-best-pets-worshipped-animals-1559234295.jpg', 'https://www.israelhayom.com/2020/04/23/2-nyc-cats-test-positive-for-coronavirus-officials-recommend-pet-precautions/', 'https://tse1.mm.bing.net/th?id=OIP.U7274nc_llbuQTChXpKVNgHaE8&pid=Api')
INSERT INTO serp VALUES('cats', 'https://s-i.huffpost.com/gen/964776/images/o-CATS-KILL-BILLIONS-facebook.jpg', 'https://www.huffingtonpost.com/2013/01/30/domestic-cats-kill-billions-mice-birds-annually-study_n_2575833.html', 'https://tse1.mm.bing.net/th?id=OIP.ETFxELWtgKQwMlcoccq-SAHaHa&pid=Api')
INSERT INTO serp VALUES('cats', 'https://external-preview.redd.it/gxbKXOj-OF1_RSHa7Ncp8Gs_OFFP5i6V7SU5DPT2t1E.jpg?auto=webp&s=b6e85ba0f1517dc629d21208a7d9db992d550ba9', 'https://www.reddit.com/r/cats/comments/2k2pio/my_very_ugly_cat/', 'https://tse1.mm.bing.net/th?id=OIP.t2BxlpEwcGrXJJQSToWVBAHaJ4&pid=Api')
INSERT INTO serp VALUES('cats', 'http://www.zastavki.com/pictures/originals/2013/Animals_Cats_Sleeping_gray_kitten_036760_.jpg', 'http://www.zastavki.com/eng/Animals/Cats/wallpaper-36760.htm', 'https://tse4.mm.bing.net/th?id=OIP.3c_ISLWidlMWXHfjqkpB2wHaEo&pid=Api')
INSERT INTO serp VALUES('cats', 'http://mcdaniel.hu/wp-content/uploads/2015/01/6784063-cute-cats-hd.jpg', 'http://mcdaniel.hu/cat-adoption-101/', 'https://tse4.mm.bing.net/th?id=OIP.QdEkrZjd1c_VN_aUtleoFgHaEo&pid=Api')

Dumping results to JSON

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.images = [];

FOREACH item IN p1.serp;
    obj.images.push({
        width = item.width
        height = item.height
        link = item.link
        pagelink = item.pagelink
        thumb = item.thumb
    });
END;

obj.json %]

Начальный текст:

Конечный текст:

Example of result:

[{
    "images": [
        {
            "link": "https://viralcats.net/blog/wp-content/uploads/2017/12/Mean-looking-cat-Viral-Cats-03.jpg",
            "width": 462,
            "thumb": "https://tse2.mm.bing.net/th?id=OIP.AdkhgipoWbJwiQBp9VIWpgAAAA&pid=Api",
            "height": 722
        },
        {
            "link": "http://mymodernmet.com/wp/wp-content/uploads/2017/03/gabrielius-khiterer-stray-cats-8.jpg",
            "width": 750,
            "thumb": "https://tse2.mm.bing.net/th?id=OIP.ZjfS8JQc9sahsK0-w8dRFAHaKJ&pid=Api",
            "height": 1028
        },
        {
            "link": "http://fishsubsidy.org/wp-content/uploads/2020/01/abyssinian-cats.jpg",
            "width": 1204,
            "thumb": "https://tse3.mm.bing.net/th?id=OIP.uHEu4-5TLJ6SSgDree6ahQHaI4&pid=Api",
            "height": 1445
        },
    
    ],
    "query": "cats"
}]

tip

To make "Start text" and "End text" options available in Task Editor, you need to activate "More options".

Possible settings

note

Common settings for all scrapers

Parameter	Default value	Description
Pages count	`5`	Number of pages to scrape
Location	`United States`	Search by region
Language	`English of United States`	Language of the results
Safe search	`Moderate`	Safe search
Size	`All`	Image size
Type	All	Image type
Layout	`All`	Layout
Color	`All`	Color

Overview of the scraper​

Use cases for the scraper​

Downloading images by link​

Capabilities​

Collected data​

Use cases​

Queries​

Query substitutions​

Output results examples​

Default output​

Output in CSV table​

Saving in SQL format​

Dumping results to JSON​

Possible settings​

Overview of the scraper

Use cases for the scraper

Downloading images by link

Capabilities

Collected data

Use cases

Queries

Query substitutions

Output results examples

Default output

Output in CSV table

Saving in SQL format

Dumping results to JSON

Possible settings