Skip to main content

Hook methods

These methods work on the hook principle. The implementation of these methods allows you to control the scraper's operation at different stages, from initialization to object destruction.

Implementation of all methods except parse is optional.

async parse(set, results)​

The parse method implements the main logic for processing a query and obtaining the scraping result; the following are passed as arguments:

  • set - an object with information about the query:
    • set.query - the query text string
    • set.lvl - the query level, defaults to 0
  • results - an object with results that need to be filled and returned from the parse() method
    • the scraper must check for the presence of each key in the results object and fill it only if present; this optimizes speed and scrapes only the data used in forming the result
    • results contains keys for the required flat variables with a value of none, which by default means no result was obtained, as well as keys for array variables (arrays) with a value in the form of an empty array ready to be filled
    • results.success must be set to 1 upon successful query processing; the default value is 0, meaning the query was processed with an error

Let's look at an example:

class JS_HTML_Tags extends BaseParser {
    static defaultConf = {
        results: {
            flat: [
                ['title', 'Title'],
            ],
            arrays: {
                h2: ['H2 Headers List', [
                    ['header', 'Header'],
                ]],
            }
        },
        ...
    };

    async parse(set, results) {
        // Getting the content of the HTML page whose address was passed in the query
        const {success, data, headers} = await this.request('GET', set.query);
    
        // Checking success and data type; when processing HTML pages correctly, we should receive type 'string', otherwise A-Parser returns a Buffer type object
        if (success && typeof data == 'string') {
            let matches;
            
            // Checking the need to collect the title and saving the value
            if (results.title && matches = data.match(/<title[^>]*>(.*?)<\/title>/))
                results.title = matches[1];

            // Checking the need to collect h2
            if (results.h2) {
                let count = 0;
                const re = /<h2[^>]*>(.*?)<\/h2>/g;
                while(matches = re.exec(data)) {
                    // Saving all found h2 tags in a loop
                    results.h2.push(matches[1]);
                }
            }

            // Notifying about scraping success
            results.success = 1;
        }

        // Returning the processed results
        return results;
    }
};

Note that you can create your own functions and methods for better code organization:

function Answer() {
    return 42;
}

class JS_HTML_Tags extends BaseParser {
    ...

    async parse(set, results) {
        results = await this.doWork(set, results);
        return results;
    }

    async doWork(set, results) {
        results.answer = Answer();
        return results;
    }
};

async processConf?(conf)​

This method is used to transform the config according to certain rules, for example, when using a captcha, we always need to use sessions:

async processConf(conf) {
    if (conf.useCaptcha)
        conf.useSessions = 1
}

async parse(set, results) {
    if (conf.useSessions)
        await this.login();
}

The existence of this method is due to the fact that A-Parser supports dynamic config fields and within a single task, the config can have different values; such a scenario is possible in two cases:

  • Using templates in configuration fields, for example [% tools.ua.random() %] for the User-Agent field
  • Using overrides when calling one scraper from another for this.parser.request

The processConf method is called once before init(). For the cases described above, processConf is additionally called before processing each query

Main rules for applying processConf:

  • Use only if config transformations have an effect on performance
  • Keep in mind that init is executed once, while processConf can be executed for each query; in this case, logic may be broken if init depends on changing config fields (see below)

async init?()​

The init method is called once during the initialization of the base scraper object and serves to perform one-time actions:

  • Starting the browser
  • Initializing the session manager using the this.sessionManager.init() method
  • Connecting to a database and creating tables in the DB
  • Reading static data
  • Etc.
caution

Since the method is called once, all configuration fields that init() depends on cannot be used in conjunction with configuration field templates or with overrides when calling this.parsers.request

async destroy?()​

The destroy method is called once upon completion of the task and is necessary for the correct destruction of open resources:

  • Closing the browser
  • Closing the DB connection
  • Etc.

async threadInit?()​

This method is launched during the initialization of each thread; each thread is a copy of the base scraper object with its own unique this.threadId, which starts from 0 and ends at threads_count - 1

Main use cases:

  • creating a browser page (tab) for each thread

async threadDestroy?()​

Executed when a thread finishes during the task completion process; serves to release resources allocated for this thread

async afterResultsProcessor?(results)​

This method is executed after the results have been processed by the Results Builder, filtering, and deduplication. The primary use case is adding queries to the queue using the this.query.add method after applying user filters; this is how link filtering for transitions (followlinks) is implemented for the HTML::LinkExtractorHTML::LinkExtractor scraper.