Skip to main content

Requests to Other Scrapers

await this.parser.request(parser, preset, overrideParams, query)​

await this.parser.request(parser, preset, overrideParams, query)

Retrieving results from another scraper (built-in or another JS scraper), the arguments are specified as follows:

  • parser - the name of the scraper (SE::Google, JS::Custom::Example)
  • preset - the preset of the settings for the called scraper
  • overrideParams - a hash with overrides for the settings of the called scraper
  • query - the query

Partially ignores the preset of the called scraper, which is set by the preset parameter. Namely, the settings useproxy, proxyChecker, proxybannedcleanup are taken from the scraper that calls await this.parser.request.

In overrideParams, you can override the parameters of the called scraper, and the following flags are also available:

overrideParams.resultArraysWithObjects​

resultArraysWithObjects: 0 - determines the format in which to return arrays of results from the called scraper:

  • if enabled (1) - arrays of objects will be returned
    [{link: 'link1', anchor: 'anchor1'}, {link: 'link2', anchor: 'anchor2'}, ...]
  • if disabled (0) - standard arrays of values will be returned
    ['link1', 'anchor1', 'link2', 'anchor2', ...]

Example:

import { BaseParser } from 'a-parser-types';

class JS_DocExample extends BaseParser {
    static defaultConf: typeof BaseParser.defaultConf = {
        results_format: "$links.format('$link\n')",
        results: {
            arrays: {
                links: ['Links', [
                    ['link', 'link']
                ]]
            }
        }
    }

    async parse(set, results) {
        let response = await this.parser.request('SE::Google', 'default', {
            resultArraysWithObjects: 1,
            pagecount: 1
        }, set.query)
        results.success = response.success;
        
        if(response.success) {
            response.serp.forEach(element => {
                results.links.push(element.link);
            });
        }

        return results;
    }
}

Example result:

https://www.speedtest.net/
https://www.investopedia.com/terms/t/t-test.asp
https://www.cdc.gov/coronavirus/2019-ncov/testing/diagnostic-testing.html
https://fast.com/
https://www.thinkwithgoogle.com/feature/testmysite/
https://projectstream.google.com/speedtest
https://www.nhs.uk/conditions/coronavirus-covid-19/testing/
https://www.fda.gov/consumers/consumer-updates/coronavirus-disease-2019-testing-basics
https://zoom.us/test
https://www.gov.uk/get-coronavirus-test
https://en.wikipedia.org/wiki/Test_(assessment)
...

overrideParams.needData​

needData: 1 - determines whether to include (1) or not (0) data/pages[] in the response, can be used for optimization

overrideParams.needResults​

needResults: [ ... ] - a list of results that need to be returned.

Example:

let response = await this.parser.request('SE::Bing', 'default', {
    needResults: [
        'totalcount'
    ]
}, set.query)

Result:

{"success":1,"info":{"success":1,"retries":0},"totalcount":"2130000000"}
tip

Works similarly for API::Server::RedisAPI::Server::Redis

overrideParams.skipProxySettingsInheritance​

skipProxySettingsInheritance: 0 - this option allows you to disable the inheritance of the useproxy parameter for the called scraper

Requests to Scrapers Supporting Batch Mode​

It is also possible to make requests to scrapers that operate in batch mode. For this, in query you need to pass an array of queries like ['key1', 'key2', ...].

The result processed by the called scraper will be contained in the bulkResults array, below is an example of such an array:

{
    "bulkResults": [
        {
            "success": 1,
            "someArrayResult": [...],
            "someFlatResult": '...',
            "query": "key1",
            "data": "..."
        },
        {
            "success": 1,
            "someArrayResult": [...],
            "someFlatResult": '...',
            "query": "key2",
            "data": "..."
        },
        {
            "success": 1,
            "someArrayResult": [...],
            "someFlatResult": '...',
            "query": "key3",
            "data": "..."
        }
    ],
    "success": 1,
    "info": {
        "success": 1,
        "retries": 1
    }
}

Example of Making a Request to Another Scraper in Batch Mode​

import { BaseParser } from 'a-parser-types';

export class JS_Example_BulkQueries extends BaseParser {
    static defaultConf: typeof BaseParser.defaultConf = {
        version: '1.0.0',
        results: {
            flat: [
                ['views', 'Views count per month']
            ]
        },
        results_format: "$query: $views\\n",
        SE_Yandex_Direct_Frequency_preset: 'default',
        bulkQueries: 10 // устанавливаем количество запросов в "пачке"
    };

    static editableConf: typeof BaseParser.editableConf = [
        ['SE_Yandex_Direct_Frequency_preset', ['combobox', 'SE::Yandex::Direct::Frequency preset']]
    ];

    async parse(set, results) {
        const { success, bulkResults } = await this.parser.request(
            'SE::Yandex::Direct::Frequency',
            this.conf.SE_Yandex_Direct_Frequency_preset,
            { useAccounts: 1 },
            set.bulkQueries.map((el) => el.query) // преобразуем массив обьектов set.bulkQueries в массив вида [query1, query2, ... query10]
        );
        if(success) {
            // заполняем результаты в results.bulkResults
            for(let query_number = 0; query_number < set.bulkQueries.length; query_number++) {
                results.bulkResults[query_number].views = bulkResults[query_number].views;
                results.bulkResults[query_number].success = bulkResults[query_number].success;
            }
        }

        return results;
    }
}
Last updated