Queries to other scrapers

await this.parser.request(parser, preset, overrideParams, query)

await this.parser.request(parser, preset, overrideParams, query)

Retrieving results from another scraper (built-in or another JS scraper), the following arguments are specified:

• parser - scraper name (SE::Google, JS::Custom::Example)
• preset - settings preset of the called scraper
• overrideParams - hash with setting overrides for the called scraper
• query - query

Partially ignores the preset of the called scraper specified by the preset parameter. Specifically, the useproxy, proxyChecker, and proxybannedcleanup settings are inherited from the scraper that calls await this.parser.request.

In overrideParams, you can override the parameters of the called scraper; the following flags are also available:

overrideParams.resultArraysWithObjects

resultArraysWithObjects: 0 - determines the format for returning result arrays of the called scraper:

• if enabled (1) - arrays of objects will be returned

  [{link: 'link1', anchor: 'anchor1'}, {link: 'link2', anchor: 'anchor2'}, ...]
• if disabled (0) - standard arrays of values will be returned

  ['link1', 'anchor1', 'link2', 'anchor2', ...]

Example:

import { BaseParser } from 'a-parser-types';

class JS_DocExample extends BaseParser {
    static defaultConf: typeof BaseParser.defaultConf = {
        results_format: "$links.format('$link\n')",
        results: {
            arrays: {
                links: ['Links', [
                    ['link', 'link']
                ]]
            }
        }
    }

    async parse(set, results) {
        let response = await this.parser.request('SE::Google', 'default', {
            resultArraysWithObjects: 1,
            pagecount: 1
        }, set.query)
        results.success = response.success;
        
        if(response.success) {
            response.serp.forEach(element => {
                results.links.push(element.link);
            });
        }

        return results;
    }
}

Example result:

https://www.speedtest.net/
https://www.investopedia.com/terms/t/t-test.asp
https://www.cdc.gov/coronavirus/2019-ncov/testing/diagnostic-testing.html
https://fast.com/
https://www.thinkwithgoogle.com/feature/testmysite/
https://projectstream.google.com/speedtest
https://www.nhs.uk/conditions/coronavirus-covid-19/testing/
https://www.fda.gov/consumers/consumer-updates/coronavirus-disease-2019-testing-basics
https://zoom.us/test
https://www.gov.uk/get-coronavirus-test
https://en.wikipedia.org/wiki/Test_(assessment)
...

overrideParams.needData

needData: 1 - determines whether to include (1) or not (0) data/pages[] in the response; can be used for optimization

overrideParams.needResults

needResults: [ ... ] - a list of results that need to be returned.

Example:

let response = await this.parser.request('SE::Bing', 'default', {
    needResults: [
        'totalcount'
    ]
}, set.query)

Result:

{"success":1,"info":{"success":1,"retries":0},"totalcount":"2130000000"}

tip

Works similarly for API::Server::Redis

overrideParams.skipProxySettingsInheritance

skipProxySettingsInheritance: 0 - this option allows disabling the inheritance of the useproxy parameter for the called scraper

Requests to scrapers supporting bulk mode

Requests to scrapers operating in bulk mode are also supported. To do this, an array of queries like ['key1', 'key2', ...] must be passed in query.

The processing result from the called scraper will be contained in the bulkResults array; an example of such an array is below:

{
    "bulkResults": [
        {
            "success": 1,
            "someArrayResult": [...],
            "someFlatResult": '...',
            "query": "key1",
            "data": "..."
        },
        {
            "success": 1,
            "someArrayResult": [...],
            "someFlatResult": '...',
            "query": "key2",
            "data": "..."
        },
        {
            "success": 1,
            "someArrayResult": [...],
            "someFlatResult": '...',
            "query": "key3",
            "data": "..."
        }
    ],
    "success": 1,
    "info": {
        "success": 1,
        "retries": 1
    }
}

Example of calling another scraper in bulk mode

import { BaseParser } from 'a-parser-types';

export class JS_Example_BulkQueries extends BaseParser {
    static defaultConf: typeof BaseParser.defaultConf = {
        version: '1.0.0',
        results: {
            flat: [
                ['views', 'Views count per month']
            ]
        },
        results_format: "$query: $views\\n",
        SE_Yandex_Direct_Frequency_preset: 'default',
        bulkQueries: 10 // sets the number of queries in a "batch"
    };

    static editableConf: typeof BaseParser.editableConf = [
        ['SE_Yandex_Direct_Frequency_preset', ['combobox', 'SE::Yandex::Direct::Frequency preset']]
    ];

    async parse(set, results) {
        const { success, bulkResults } = await this.parser.request(
            'SE::Yandex::Direct::Frequency',
            this.conf.SE_Yandex_Direct_Frequency_preset,
            { useAccounts: 1 },
            set.bulkQueries.map((el) => el.query) // transform the set.bulkQueries object array into an array like [query1, query2, ... query10]
        );
        if(success) {
            // filling results into results.bulkResults
            for(let query_number = 0; query_number < set.bulkQueries.length; query_number++) {
                results.bulkResults[query_number].views = bulkResults[query_number].views;
                results.bulkResults[query_number].success = bulkResults[query_number].success;
            }
        }

        return results;
    }
}