Requests to other Scrapers

await this.parser.request(parser, preset, overrideParams, query)

await this.parser.request(parser, preset, overrideParams, query)

Getting results from another scraper (built-in or another JS scraper); the arguments are

• preset - predefined settings of the called scraper

• overrideParams - hash with settings overrides for the called scraper

• query - query resultArraysWithObjects: 0 - determines the format of result arrays returned by the called scraper:

• if enabled (1) - arrays of objects will be returned

• if disabled (0) - standard arrays of values will be returned

overrideParams.resultArraysWithObjects

resultArraysWithObjects: 0 - determines in what form to return the result arrays of the called scraper:

• if enabled (1) - arrays of objects will be returned

  [{link: 'link1', anchor: 'anchor1'}, {link: 'link2', anchor: 'anchor2'}, ...]
• if disabled (0) - standard arrays of values will be returned

  ['link1', 'anchor1', 'link2', 'anchor2', ...]

Example:

import { BaseParser } from 'a-parser-types';

class JS_DocExample extends BaseParser {
    static defaultConf: typeof BaseParser.defaultConf = {
        results_format: "$links.format('$link\n')",
        results: {
            arrays: {
                links: ['Links', [
                    ['link', 'link']
                ]]
            }
        }
    }

    async parse(set, results) {
        let response = await this.parser.request('SE::Google', 'default', {
            resultArraysWithObjects: 1,
            pagecount: 1
        }, set.query)
        results.success = response.success;
        
        if(response.success) {
            response.serp.forEach(element => {
                results.links.push(element.link);
            });
        }

        return results;
    }
}

Example result:

https://www.speedtest.net/
https://www.investopedia.com/terms/t/t-test.asp
https://www.cdc.gov/coronavirus/2019-ncov/testing/diagnostic-testing.html
https://fast.com/
https://www.thinkwithgoogle.com/feature/testmysite/
https://projectstream.google.com/speedtest
https://www.nhs.uk/conditions/coronavirus-covid-19/testing/
https://www.fda.gov/consumers/consumer-updates/coronavirus-disease-2019-testing-basics
https://zoom.us/test
https://www.gov.uk/get-coronavirus-test
https://en.wikipedia.org/wiki/Test_(assessment)
...

overrideParams.needData

needData: 1 - determines whether to pass (1) or not (0) data/pages[], in the response; can be used for optimization

overrideParams.needResults

needResults: [ ... ] - a list of results that need to be returned.

Example:

let response = await this.parser.request('SE::Bing', 'default', {
    needResults: [
        'totalcount'
    ]
}, set.query)

Result:

{"success":1,"info":{"success":1,"retries":0},"totalcount":"2130000000"}

tip

Works similarly for API::Server::Redis

overrideParams.skipProxySettingsInheritance

skipProxySettingsInheritance: 0 - this option allows you to disable the inheritance of the useproxy parameter for the called scraper

Requests to scrapers that support bulk mode

Calling scrapers operating in bulk mode. is also supported. To do this, in query you need to pass an array of queries of the form ['key1', 'key2', ...].

The result of the called scraper's processing will be contained in the array bulkResults, an example of such an array is below:

{
    "bulkResults": [
        {
            "success": 1,
            "someArrayResult": [...],
            "someFlatResult": '...',
            "query": "key1",
            "data": "..."
        },
        {
            "success": 1,
            "someArrayResult": [...],
            "someFlatResult": '...',
            "query": "key2",
            "data": "..."
        },
        {
            "success": 1,
            "someArrayResult": [...],
            "someFlatResult": '...',
            "query": "key3",
            "data": "..."
        }
    ],
    "success": 1,
    "info": {
        "success": 1,
        "retries": 1
    }
}

Example of calling another scraper in bulk mode

import { BaseParser } from 'a-parser-types';

export class JS_Example_BulkQueries extends BaseParser {
    static defaultConf: typeof BaseParser.defaultConf = {
        version: '1.0.0',
        results: {
            flat: [
                ['views', 'Views count per month']
            ]
        },
        results_format: "$query: $views\\n",
        SE_Yandex_Direct_Frequency_preset: 'default',
        bulkQueries: 10 // set the number of queries in a "batch"
    };

    static editableConf: typeof BaseParser.editableConf = [
        ['SE_Yandex_Direct_Frequency_preset', ['combobox', 'SE::Yandex::Direct::Frequency preset']]
    ];

    async parse(set, results) {
        const { success, bulkResults } = await this.parser.request(
            'SE::Yandex::Direct::Frequency',
            this.conf.SE_Yandex_Direct_Frequency_preset,
            { useAccounts: 1 },
            set.bulkQueries.map((el) => el.query) // convert the array of objects set.bulkQueries into an array of the form [query1, query2, ... query10]
        );
        if(success) {
            // fill the results into results.bulkResults
            for(let query_number = 0; query_number < set.bulkQueries.length; query_number++) {
                results.bulkResults[query_number].views = bulkResults[query_number].views;
                results.bulkResults[query_number].success = bulkResults[query_number].success;
            }
        }

        return results;
    }
}