Requests to Other Scrapers
await this.parser.request(parser, preset, overrideParams, query)
await this.parser.request(parser, preset, overrideParams, query)
Retrieving results from another scraper (built-in or another JS scraper), the arguments are specified as follows:
parser
- the name of the scraper (SE::Google, JS::Custom::Example)preset
- the preset of the settings for the called scraperoverrideParams
- a hash with overrides for the settings of the called scraperquery
- the query
Partially ignores the preset of the called scraper, which is set by the preset
parameter. Namely, the settings useproxy
, proxyChecker
, proxybannedcleanup
are taken from the scraper that calls await this.parser.request
.
In overrideParams
, you can override the parameters of the called scraper, and the following flags are also available:
overrideParams.resultArraysWithObjects
resultArraysWithObjects: 0
- determines the format in which to return arrays of results from the called scraper:
- if enabled (1) - arrays of objects will be returned
[{link: 'link1', anchor: 'anchor1'}, {link: 'link2', anchor: 'anchor2'}, ...]
- if disabled (0) - standard arrays of values will be returned
['link1', 'anchor1', 'link2', 'anchor2', ...]
Example:
import { BaseParser } from 'a-parser-types';
class JS_DocExample extends BaseParser {
static defaultConf: typeof BaseParser.defaultConf = {
results_format: "$links.format('$link\n')",
results: {
arrays: {
links: ['Links', [
['link', 'link']
]]
}
}
}
async parse(set, results) {
let response = await this.parser.request('SE::Google', 'default', {
resultArraysWithObjects: 1,
pagecount: 1
}, set.query)
results.success = response.success;
if(response.success) {
response.serp.forEach(element => {
results.links.push(element.link);
});
}
return results;
}
}
Example result:
https://www.speedtest.net/
https://www.investopedia.com/terms/t/t-test.asp
https://www.cdc.gov/coronavirus/2019-ncov/testing/diagnostic-testing.html
https://fast.com/
https://www.thinkwithgoogle.com/feature/testmysite/
https://projectstream.google.com/speedtest
https://www.nhs.uk/conditions/coronavirus-covid-19/testing/
https://www.fda.gov/consumers/consumer-updates/coronavirus-disease-2019-testing-basics
https://zoom.us/test
https://www.gov.uk/get-coronavirus-test
https://en.wikipedia.org/wiki/Test_(assessment)
...
overrideParams.needData
needData: 1
- determines whether to include (1) or not (0) data/pages[]
in the response, can be used for optimization
overrideParams.needResults
needResults: [ ... ]
- a list of results that need to be returned.
Example:
let response = await this.parser.request('SE::Bing', 'default', {
needResults: [
'totalcount'
]
}, set.query)
Result:
{"success":1,"info":{"success":1,"retries":0},"totalcount":"2130000000"}
Works similarly for API::Server::Redis
overrideParams.skipProxySettingsInheritance
skipProxySettingsInheritance: 0
- this option allows you to disable the inheritance of the useproxy
parameter for the called scraper
Requests to Scrapers Supporting Batch Mode
It is also possible to make requests to scrapers that operate in batch mode. For this, in query
you need to pass an array of queries like ['key1', 'key2', ...]
.
The result processed by the called scraper will be contained in the bulkResults
array, below is an example of such an array:
{
"bulkResults": [
{
"success": 1,
"someArrayResult": [...],
"someFlatResult": '...',
"query": "key1",
"data": "..."
},
{
"success": 1,
"someArrayResult": [...],
"someFlatResult": '...',
"query": "key2",
"data": "..."
},
{
"success": 1,
"someArrayResult": [...],
"someFlatResult": '...',
"query": "key3",
"data": "..."
}
],
"success": 1,
"info": {
"success": 1,
"retries": 1
}
}
Example of Making a Request to Another Scraper in Batch Mode
import { BaseParser } from 'a-parser-types';
export class JS_Example_BulkQueries extends BaseParser {
static defaultConf: typeof BaseParser.defaultConf = {
version: '1.0.0',
results: {
flat: [
['views', 'Views count per month']
]
},
results_format: "$query: $views\\n",
SE_Yandex_Direct_Frequency_preset: 'default',
bulkQueries: 10 // устанавливаем количество запросов в "пачке"
};
static editableConf: typeof BaseParser.editableConf = [
['SE_Yandex_Direct_Frequency_preset', ['combobox', 'SE::Yandex::Direct::Frequency preset']]
];
async parse(set, results) {
const { success, bulkResults } = await this.parser.request(
'SE::Yandex::Direct::Frequency',
this.conf.SE_Yandex_Direct_Frequency_preset,
{ useAccounts: 1 },
set.bulkQueries.map((el) => el.query) // преобразуем массив обьектов set.bulkQueries в массив вида [query1, query2, ... query10]
);
if(success) {
// заполняем результаты в results.bulkResults
for(let query_number = 0; query_number < set.bulkQueries.length; query_number++) {
results.bulkResults[query_number].views = bulkResults[query_number].views;
results.bulkResults[query_number].success = bulkResults[query_number].success;
}
}
return results;
}
}