Skip to main content

Helper methods (utils, tools, sleep)

this.utils.*​

.updateResultsData(results, data)​

await this.utils.updateResultsData(results, data) - a method for automatically filling $pages.$i.data and $data, must be called to add content to the resulting page

.urlFromHTML(url, base)​

await this.utils.urlFromHTML(url, base) - processes a link obtained from HTML code - decodes entities (& etc.), optionally base can be passed - the base URL (e.g., the URL of the source page), thus a full link can be obtained

.url.extractDomain(url, removeDefaultSubdomain)​

await this.utils.url.extractDomain(url, removeDefaultSubdomain) - the method takes a link as the first parameter and returns the domain from this link. The second optional parameter determines whether to trim the www subdomain from the domain. The default is 0 - meaning do not trim.

.url.extractTopDomain(url)​

await this.utils.url.extractTopDomain(url) - the method takes a link as the first parameter and returns the domain from this link, without subdomains.

.url.extractTopDomainByZone(url)​

await this.utils.url.extractTopDomainByZone(url) - the method takes a link as the first parameter and returns the domain from this link, including without subdomains. Works with all regional zones

.url.extractMaxPath(url)​

await this.utils.url.extractMaxPath(url) - the method takes a string and extracts a URL from it

.url.extractWOParams(url)​

await this.utils.url.extractWOParams(url)- the method takes a link and returns the same link trimmed to the parameter string. That is, it will return the URL up to ?

.removeHtml(string)​

await this.utils.removeHtml(string) - the method takes a string and returns it cleared of HTML tags

.removeNoDigit(string)​

await this.utils.removeNoDigit(string) - the method takes a string, removes everything except digits from it, and returns the result

.removeComma(string)​

await this.utils.removeComma(string) - the method takes a string, removes characters such as .,\r\n from it, and returns the result

.getAllBlocks(html, regexp, opts?)​

await this.utils.getAllBlocks(html, regexp, opts?) - obtaining all blocks on a page with corresponding closing tags, the method takes an HTML string and a regular expression that indicates the start of the block (any blocks that have paired closing tags, for example <div>...</div>), the result returns an array of all found blocks

Options opts:

  • searchStartIndex - specifies the index in the string from which to start the search, default is 0 
const blocks = this.utils.getAllBlocks(html, /<div [^>]*?class="results"/)

.getAllBlocksByAttr(html, tag, attrName, attrRegExp, opts?)​

await this.utils.getAllBlocksByAttr(html, tag, attrName, attrRegExp, opts?) - a method similar to .getAllBlocks, instead of a regular expression to find the start of a block, the tag name, the attribute name to search by (e.g., id, class), and a regular expression to be applied to the value of the specified attribute are specified

const blocks = this.utils.getAllBlocksByAttr(html, 'div', 'class', /results/)

await tools.*​

Global object tools, allows access to built-in A-Parser functions

Analogue of template engine tools $tools.*

note

tools.query is unavailable, this.query must be used

await tools.createTemplate(string)​

Allows using the Template Toolkit template engine inside a JavaScript scraper.

let template = await tools.createTemplate("Hello [% content %]!")
template = typeof template == 'function' ? await template({content: 'World'}) : template
this.logger.put(template) // Output: Hello World!

Example of use "Scraper for translating Markdown markup to HTML"

await this.sleep(sec)​

await this.sleep(sec)

Sets a delay in the thread for a number of seconds (sec), can be fractional.

await this.mutex.*​

A mutex for synchronization between threads, allows locking a code section for one thread

.lock()​

Waiting for a lock, execution will continue for the first thread that acquired the lock, other threads will wait for the lock to be released

.unlock()​

Releasing the lock, the next thread will continue execution if it was waiting for the lock - .lock()

results.<array>.addElement()​

The results.<array>.addElement() method allows for more convenient filling of arrays in results. When using it, there is no need to remember the sequence of variables in the array and list them manually.

results.serp.addElement({
    link: 'https://google.com',
    anchor: 'Google',
    snippet: 'Loreps ipsum...',
});

this.isContextAlive()​

This method is necessary for long-lived threads that process queries in a loop, allowing for correct termination when a task is stopped or deleted

while (this.isContextAlive()) {
    await this.request(...)
}