Skip to main content

SE::Brave - Brave Search scraper

Brave

Scraper overview​

The Brave Search scraper allows you to collect data from a fast-growing, privacy-oriented search engine. With it, you can obtain link databases ready for further use. You can use queries in the same way you enter them in Brave, including search operators.

Thanks to A-Parser's multi-threaded operation, query processing speed can be very high, allowing you to process large volumes of data in a short time.

A-Parser functionality allows you to save scraping settings for further use (presets), set a scraping schedule, and much more. You can use query multiplication, substitution of subqueries from files, enumeration of alphanumeric combinations and lists to get the maximum possible number of results.

Saving results is possible in the form and structure you need, thanks to the built-in powerful template engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL and CSV.

Collected data​

  • Links, anchors and snippets from the search results, as well as the date from the snippet (if available)
  • Related keywords

Capabilities​

  • support for all search operators Brave (site:, inurl: etc.)
  • specifying the number of pages(from 1 to 10), on average Brave outputs about 10 results per page
  • specifying the search region and interface language of the retrieved page
  • ability to get accurate local results, support for determining location by IP or exact specification
  • ability to choose the type of results, besides the main organic results, the Brave scraper can collect news or video results
  • supports specifying the result time (any time or a specific interval from 24 hours to a year)
  • session support

Use cases​

  • Collecting link databases - for XRumer, AllSubmitter, GSA Ranker, etc.
  • Dumping the SERP, including links, anchors, and snippets, allows for deep analysis for SEO specialists and marketers.
  • Searching for inbound links and website mentions
  • Any other cases involving obtaining search results for an unlimited number of queries

Queries​

As queries, you should specify search phrases exactly as if you were entering them directly into the Brave search form, for example:

car purchase
windows in Kyiv
site:https://lenta.ru
inurl:guestbook

Query substitutions​

You can use built-in macros for query multiplication, for example, we want to get a very large database of forums, for this we will specify several main queries in different languages:

forum
forum
foro
论坛

In the query format, we will specify character enumeration from a to zzzz, this method allows for maximum rotation of search results and obtaining many new unique results:

$query {az:a:zzzz}

This macro will create 475254 additional queries for each original search query, which in total will give 4 x 475254 = 1901016 search queries, an impressive figure, but it's no problem at all for A-Parser. At a speed of 2000 queries per minute, such a task will be processed in just 16 hours.

Using operators​

You can use search operators in the query format, so they will be automatically added to each query from your list, for example:

inurl:$query

Output results examples​

A-Parser supports flexible result formatting thanks to the built-in template engine Template Toolkit, which allows it to output results in an arbitrary form, as well as in structured formats like CSV or JSON

Exporting a list of links​

Result format:

$serp.format('$link\n')

Example result:

https://www.mobile.de/ru
https://auto.drom.ru/
https://autospot.ru/brands/
https://www.destacar.de/
https://www.autoscout24.de/auto/gebrauchtwagen/bayern/falkenstein/
https://www-autoservice-falkenstein.skoda-auto.de/?e=7
https://auto.ru/
https://www.autohaus-schueler.de/
https://www.avtogermes.ru/sale/
https://falkenstein.seat.de/
...

Result format:

[% FOREACH item IN serp;
    loop.count _ ' - ' _ item.link _ ' - ' _ item.anchor _ ' - ' _ item.snippet _ "\n";
END %]

Example result:

1 - https://www.mobile.de/ru - mobile.de: used cars — ads for ... - mobile.de/ru is part of mobile.de, Germany's largest vehicle marketplace. mobile.de makes buying a car easy. With a simple search and just a few clicks, you get a complete overview of the market.
2 - https://auto.drom.ru/ - Selling cars in Russia. Use... - 775 361 ads for the sale of cars. Sale of new and used cars from 10,000 rubles. Car dealerships and private ads.
3 - https://www.destacar.de/ - Cars from Germany to order | ... - Destacar is always ready to help you in searching, selecting, inspecting and buying a car in Germany. We will help you buy a used car in Germany, do an inspection and test drive so that you are completely confident in the condition of your car.
4 - https://autospot.ru/brands/ - Buy a car in Moscow: prices for ... - ✸ Car selection by parameters in Moscow: prices in the Autospot car catalog. ✸ Catalog of cars with prices and technical specifications - sale from official dealers.
5 - https://www.autoscout24.de/auto/gebrauchtwagen/bayern/falkenstein/ - Gebrauchtwagen Falkenstein: Auto günstig kaufen - Gebrauchtwagen & Jahreswagen in Falkenstein kaufen und verkaufen bei AutoScout24.de - Europas großem Automarkt » Jetzt finden oder inserieren!
6 - https://www-autoservice-falkenstein.skoda-auto.de/?e=7 - ŠKODA Gebrauchtwagen in Falkenstein | Auto-Service Falkenstein GmbH - ŠKODA Gebrauchtwagen in Falkenstein | Auto-Service Falkenstein GmbH ✔ Günstig finanzieren ✔ Beste ŠKODA Vorführwagen und Jahreswagen ✔ ŠKODA gebraucht kaufen ✔ Probefahrt vereinbaren
7 - https://auto.ru/ - Auto.ru: buy, sell and exchange ... - This website operates in accordance with the legislation of the Russian Federation and is not intended for operation in your region. By continuing to use the website, ...
8 - https://www.autohaus-schueler.de/ - Ihr Audi und VW Autohaus Schüler in Falkenstein - Ihr Autohaus Schüler in Falkenstein, Oelsnitz, Plauen & Wilkau-Hasslau - Wir bieten ein großes Angebot an Neu- und Gebrauchtwagen, Audi & VW Service.
...

The built-in utility $tools.CSVLine allows you to create correct tabular documents ready for import into Excel or Google Sheets.

General result format:

[%  FOREACH item IN p1.serp;
    tools.CSVline(item.link, item.anchor, item.snippet);
END  %]

File name:

$datefile.format().csv

Initial text:

Link,Anchor,Snippet

Example result:

Link,Anchor,Snippet
https://www.mobile.de/ru,"mobile.de: used cars — ads for ...","mobile.de/ru is part of mobile.de, Germany's largest vehicle marketplace. mobile.de makes buying a car easy. With a simple search and just a few clicks, you get a complete overview of the market."
https://auto.drom.ru/,"Selling cars in Russia. Use...","775 361 ads for the sale of cars. Sale of new and used cars from 10,000 rubles. Car dealerships and private ads."
https://www.destacar.de/,"Cars from Germany to order | ...","Destacar is always ready to help you in searching, selecting, inspecting and buying a car in Germany. We will help you buy a used car in Germany, do an inspection and test drive so that you are completely confident in the condition of your car."
https://autospot.ru/brands/,"Buy a car in Moscow: prices for ...","✸ Car selection by parameters in Moscow: prices in the Autospot car catalog. ✸ Catalog of cars with prices and technical specifications - sale from official dealers."
https://www.autoscout24.de/auto/gebrauchtwagen/bayern/falkenstein/,"Gebrauchtwagen Falkenstein: Auto günstig kaufen","Gebrauchtwagen & Jahreswagen in Falkenstein kaufen und verkaufen bei AutoScout24.de - Europas großem Automarkt » Jetzt finden oder inserieren!"
https://www-autoservice-falkenstein.skoda-auto.de/?e=7,"ŠKODA Gebrauchtwagen in Falkenstein | Auto-Service Falkenstein GmbH","ŠKODA Gebrauchtwagen in Falkenstein | Auto-Service Falkenstein GmbH ✔ Günstig finanzieren ✔ Beste ŠKODA Vorführwagen und Jahreswagen ✔ ŠKODA gebraucht kaufen ✔ Probefahrt vereinbaren"
https://auto.ru/,"Auto.ru: buy, sell and exchange ...","This website operates in accordance with the legislation of the Russian Federation and is not intended for operation in your region. By continuing to use the website, ..."
https://www.autohaus-schueler.de/,"Ihr Audi und VW Autohaus Schüler in Falkenstein","Ihr Autohaus Schüler in Falkenstein, Oelsnitz, Plauen & Wilkau-Hasslau - Wir bieten ein großes Angebot an Neu- und Gebrauchtwagen, Audi & VW Service."
...
tip

In the Result Format section, the Template Toolkit is used to output the $serp array in a FOREACH loop.

In the result file name, you simply need to change the file extension to csv.

To make the "Initial text" option available in the Task Editor, you need to activate "More options". In "Initial text", write the column names separated by commas and make the second line empty.

Result format:

$related.format('$key\n')

Example result:

test speed
ookla speed test
speed test wifi
internet speed test
run speed test
fast speed test
mbps test
test 2
test match
google speed test
...

Saving in SQL format​

Result format:

[%  FOREACH serp;
    "INSERT INTO serp VALUES('" _ query _ "', '";
    link _ "', '";
    anchor _ "')\n";
END  %]

Example result:

    
INSERT INTO serp VALUES('test', 'https://www.speedtest.net/', 'Speedtest by Ookla - The Global Broadband Speed Test')
INSERT INTO serp VALUES('test', 'https://www.merriam-webster.com/dictionary/test', 'TEST Definition & Meaning - Merriam-Webster')
INSERT INTO serp VALUES('test', 'https://englishtest.duolingo.com/', 'Duolingo English Test')
INSERT INTO serp VALUES('test', 'https://link.springer.com/journal/11749', 'Home | TEST | Springer Nature Link')
INSERT INTO serp VALUES('test', 'https://test.io/', 'Home | Test IO')
INSERT INTO serp VALUES('test', 'https://www.spectrum.com/internet/speed-test', 'Spectrum Internet Speed Test: Broadband Internet Speed Check')
INSERT INTO serp VALUES('test', 'https://fast.com/', 'Internet Speed Test | Fast.com')
INSERT INTO serp VALUES('test', 'https://en.wikipedia.org/wiki/Test', 'Test - Wikipedia')
...

Dumping results to JSON​

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.results = [];

FOREACH item IN p1.serp;
    obj.results.push({
        link = item.link
        anchor = item.anchor
        snippet = item.snippet
    });
END;

obj.json %]

Начальный текст:

[

Конечный текст:

]

Example result:

{"results":[{"link":"https://www.speedtest.net/","snippet":"Use Speed<strong>test</strong> on all your devices with our free desktop and mobile apps.","anchor":"Speedtest by Ookla - The Global Broadband Speed Test"},{"link":"https://www.merriam-webster.com/dictionary/test","snippet":"The meaning of TEST is <strong>a means of testing</strong>. How to use test in a sentence.","anchor":"TEST Definition & Meaning - Merriam-Webster"},{"link":"https://englishtest.duolingo.com/","snippet":"The Duolingo English Test is a fast, affordable online English exam accepted by institutions worldwide. Take it anytime and get results in 2 days","anchor":"Duolingo English Test"},{"link":"https://link.springer.com/journal/11749","snippet":"Journal TEST <strong>focuses on cutting-edge research in all areas of Statistics and Probability with significant theoretical, methodological, and applied impact</strong>. ...","anchor":"Home | TEST | Springer Nature Link"},{"link":"https://test.io/","snippet":"Test IO delivers a full range of web, mobile, and IoT <strong>test</strong>ing, delivered as a service.","anchor":"Home | Test IO"},{"link":"https://www.spectrum.com/internet/speed-test","snippet":"Test your broadband Internet and WiFi speed with the Internet Speed Test. Check your upload and download speeds for your home Internet network.","anchor":"Spectrum Internet Speed Test: Broadband Internet Speed Check"},{"link":"https://fast.com/","snippet":"FAST.com will <strong>test</strong> Internet speed globally on any device (phone, laptop, or smart TV with browser).","anchor":"Internet Speed Test | Fast.com"},{"link":"https://en.wikipedia.org/wiki/Test","snippet":"Look up test, testing, Test, or TEST in Wiktionary, the free dictionary. ... Test (assessment), <strong>an educational assessment intended to measure the respondents&#x27; knowledge or other abilities</strong> ... Test., abbreviation for Testament, referring to the Old Testament and New Testament of the Christian Bible.","anchor":"Test - Wikipedia"},{"link":"https://speed.cloudflare.com/","snippet":"Test your Internet connection. Check your network performance with our Internet speed <strong>test</strong>. Powered by Cloudflare&#x27;s global edge network.","anchor":"Internet Speed Test - Measure Network Performance | Cloudflare"},{"link":"https://testmy.net/","snippet":"TestMy.net is a powerful broadband speed <strong>test</strong> that will <strong>test</strong> your Internet, calculate your transfer rate and output accurate, reliable and easy to understand results. TestMy.net is an independent third party and is not affiliated with your Internet service provider.","anchor":"Internet Speed Test"},{"link":"https://www.livechat.com/typing-speed-test/","snippet":"The faster you type, the faster you communicate with others. With our free typing speed <strong>test</strong>, you can check your WPM and CPM typing scores in a flash!","anchor":"Free Typing Test - Check Your Typing Speed in 60 Seconds | LiveChat ..."},{"link":"https://fiber.google.com/speedtest/","snippet":"Test your current internet speed, and find out how fast your broadband wi-fi handles uploads and downloads. See Google Fiber plan options for faster internet.","anchor":"Internet Speed Test | Check Broadband Speed | Google Fiber"},{"link":"https://zoom.us/test","snippet":"Preview your next Zoom Meeting using out <strong>test</strong> meeting tool to check your settings and display.","anchor":"Join a Test Meeting | Zoom"},{"link":"https://implicit.harvard.edu/implicit/takeatest.html","snippet":"On the next page, you&#x27;ll be asked to select an Implicit Association Test (IAT) from a list of possible topics. We&#x27;ll also ask you (optionally) to report your attitudes or beliefs about these topics and give you some information about yourself. We ask these questions because the IAT can be more valuable if you also describe your own self-understanding of the attitude or stereotype that the IAT measures.","anchor":"Take a Test"},{"link":"https://www.act.org/content/act/en/products-and-services/the-act.html","snippet":"The ACT <strong>test</strong> is a curriculum-based education and career planning tool for high school students that assesses the mastery of college readiness standards","anchor":"The ACT Test for Students | ACT"},{"link":"https://play.google.com/store/apps/details?id=org.zwanoo.android.speedtest&hl=en_US","snippet":"Test your download and upload speeds as well as three measures of latency to check a slow connection or use the app to make sure your network is ready for a gaming session. Use real-world data to find out where mobile network coverage is strong and weak for providers in your area with the Speed<strong>test</strong> Maps tab.","anchor":"Speedtest by Ookla - Apps on Google Play"},{"link":"https://speedtest.xfinity.com/","snippet":"Xfinity Speed Test – You need to enable JavaScript to run this app","anchor":"Xfinity Speed Test - Check Your Internet Speed"},{"link":"https://www.ada.org/education/testing/exams/dental-admission-test-dat","snippet":"Find out how to apply for, prepare for, take, and receive scores for the Dental Admission Test (DAT).","anchor":"Dental Admission Test (DAT) | American Dental Association"},{"link":"https://www.16personalities.com/free-personality-test","snippet":"Take our free personality <strong>test</strong> and discover your personality type in 10 minutes. Get a ‘freakishly accurate’ description of who you are and why you do things the way you do.","anchor":"Free personality test | 16Personalities"},{"link":"https://www.faa.gov/uas/recreational_flyers/knowledge_test_updates","snippet":"The law requires that all recreational flyers pass an aeronautical knowledge and safety <strong>test</strong> and provide proof of passage if asked by law enforcement or FAA personnel. The Recreational UAS Safety Test (TRUST) was developed to meet this requirement. TRUST provides education and <strong>test</strong>ing on important safety and regulatory information.","anchor":"The Recreational UAS Safety Test (TRUST) | Federal Aviation ..."}]}
tip

To make the "Initial text" and "End text" options available in the Task Editor, you need to activate "More options".

Possible settings​

note

Common settings for all scrapers

Parameter nameDefault valueDescription
Serp typeAllDetermines whether to scrape general results (All), news (News), or videos (Videos)
Pages count5Number of pages to scrape (from 1 to 10)
RegionAll regionsChoice of the country from which the search is carried out
Display languageEnglish (United States)Ability to select the interface language of the search results page, for maximum identity of results in the scraper and in the browser
Safe searchModerateAbility to enable "Safe Search"
Search timeAnySERP time (time-dependent search)
Local resultsOn (auto, from IP)Local search, possible values: Off - disabled, Auto - location is determined by the request IP, Manual - the location specified in Manual location is used
Manual locationField for entering an exact location, used only when Local results = Manual. The location must be specified as a pair of coordinates separated by a comma in the format latitude,longitude, for example: 55.780844,37.6572693