Selective processing of JavaScript: decoding phones numbers

[Support]

From version 1.1.318 to A-Parser adds support for JS. Therefore, in this article we examine an example of this functionality, as well as will be shown, both directly during the task, you can add and execute JS from the sites.

Task. Necessary parse all phone numbers from a private ads about selling used cars on the site http://ru.auto.soov.ee/

Open and analyze site. In the search we set all the necessary parameters and obtain a link to the results. The first step is the parsing of links to ads. In the second step, we will parse the numbers from each advertisement.

1) Parsing links:

Spoiler: Code for import

eyJwcmVzZXQiOiJKU190ZWxzIDEiLCJ2YWx1ZSI6eyJwcmVzZXQiOiJKU190ZWxz
IDEiLCJwYXJzZXJzIjpbWyJIVE1MOjpMaW5rRXh0cmFjdG9yIiwiZGVmYXVsdCIs
eyJ0eXBlIjoiZmlsdGVyIiwicmVzdWx0IjpbImludGxpbmtzIiwibGluayJdLCJm
aWx0ZXJUeXBlIjoicmVtYXRjaCIsInZhbHVlIjoib2JqYXZsZW5pamEvLis/XFwu
aHRtbCQiLCJvcHRpb24iOiIifSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoiZm9y
bWF0cmVzdWx0IiwidmFsdWUiOiIkaW50bGlua3MuZm9ybWF0KCckbGlua1xcbicp
In1dXSwicmVzdWx0c0Zvcm1hdCI6IiRwMS5wcmVzZXQiLCJyZXN1bHRzU2F2ZVRv
IjoiZmlsZSIsInJlc3VsdHNGaWxlTmFtZSI6IiRkYXRlZmlsZS5mb3JtYXQoKS50
eHQiLCJhZGRpdGlvbmFsRm9ybWF0cyI6W10sInJlc3VsdHNVbmlxdWUiOiJzdHJp
bmciLCJxdWVyeUZvcm1hdCI6WyIkcXVlcnkiXSwidW5pcXVlUXVlcmllcyI6ZmFs
c2UsInNhdmVGYWlsZWRRdWVyaWVzIjpmYWxzZSwiaXRlcmF0b3JPcHRpb25zIjp7
Im9uQWxsTGV2ZWxzIjpmYWxzZSwicXVlcnlCdWlsZGVyc0FmdGVySXRlcmF0b3Ii
OmZhbHNlLCJxdWVyeUJ1aWxkZXJzT25BbGxMZXZlbHMiOmZhbHNlfSwicmVzdWx0
c09wdGlvbnMiOnsib3ZlcndyaXRlIjpmYWxzZX0sImRvTG9nIjoibm8iLCJrZWVw
VW5pcXVlIjoiTm8iLCJtb3JlT3B0aW9ucyI6ZmFsc2UsInJlc3VsdHNQcmVwZW5k
IjoiIiwicmVzdWx0c0FwcGVuZCI6IiIsInF1ZXJ5QnVpbGRlcnMiOltdLCJyZXN1
bHRzQnVpbGRlcnMiOltdLCJjb25maWdPdmVycmlkZXMiOltdfX0=

• In the query uses a macro {num:1:1000} to loop through all the pages.

2) Given a list of links to ads, we begin to parsing phone numbers. But if you look at the code of any page with the ad, you may notice, that the phone number where encrypted by script:

Thanks tools.js we are able to run this script and get a result. For this we need parse the script and substitute it in tools.js.

Spoiler: Code for import

eyJwcmVzZXQiOiJKU190ZWxzIDIiLCJ2YWx1ZSI6eyJwcmVzZXQiOiJKU190ZWxz
IDIiLCJwYXJzZXJzIjpbWyJOZXQ6OkhUVFAiLCJkZWZhdWx0Iix7InR5cGUiOiJv
dmVycmlkZSIsImlkIjoiZ29vZENvZGUiLCJ2YWx1ZSI6MjAwfSx7InR5cGUiOiJj
dXN0b21SZXN1bHQiLCJyZXN1bHQiOiJkYXRhIiwicmVnZXgiOiI8dGg+XHUwNDFh
XHUwNDNlXHUwNDNkXHUwNDQyXHUwNDMwXHUwNDNhXHUwNDQyXHUwNDNkXHUwNDRi
XHUwNDM5IFx1MDQ0Mlx1MDQzNVx1MDQzYlx1MDQzNVx1MDQ0NFx1MDQzZVx1MDQz
ZDo8XFwvdGg+PHRkPjxzdHJvbmc+PHNjcmlwdCB0eXBlPVwidGV4dFxcL2phdmFz
Y3JpcHRcIj4oLis/KTxcXC9zY3JpcHQ+IiwicmVnZXhUeXBlIjoicyIsInJlc3Vs
dFR5cGUiOiJmbGF0IiwiYXJyYXlOYW1lIjoiIiwicmVzdWx0cyI6WyJzdGVsIl19
LHsidHlwZSI6Im92ZXJyaWRlIiwiaWQiOiJmb3JtYXRyZXN1bHQiLCJ2YWx1ZSI6
IlslIHRvb2xzLmpzLmNhdGNoV3JpdGUoc3RlbCkgJV0ifV1dLCJyZXN1bHRzRm9y
bWF0IjoiJHAxLnByZXNldCIsInJlc3VsdHNTYXZlVG8iOiJmaWxlIiwicmVzdWx0
c0ZpbGVOYW1lIjoiJGRhdGVmaWxlLmZvcm1hdCgpLnR4dCIsImFkZGl0aW9uYWxG
b3JtYXRzIjpbXSwicmVzdWx0c1VuaXF1ZSI6InN0cmluZyIsInF1ZXJ5Rm9ybWF0
IjpbIiRxdWVyeSJdLCJ1bmlxdWVRdWVyaWVzIjpmYWxzZSwic2F2ZUZhaWxlZFF1
ZXJpZXMiOmZhbHNlLCJpdGVyYXRvck9wdGlvbnMiOnsib25BbGxMZXZlbHMiOmZh
bHNlLCJxdWVyeUJ1aWxkZXJzQWZ0ZXJJdGVyYXRvciI6ZmFsc2UsInF1ZXJ5QnVp
bGRlcnNPbkFsbExldmVscyI6ZmFsc2V9LCJyZXN1bHRzT3B0aW9ucyI6eyJvdmVy
d3JpdGUiOmZhbHNlfSwiZG9Mb2ciOiJubyIsImtlZXBVbmlxdWUiOiJObyIsIm1v
cmVPcHRpb25zIjpmYWxzZSwicmVzdWx0c1ByZXBlbmQiOiIiLCJyZXN1bHRzQXBw
ZW5kIjoiIiwicXVlcnlCdWlsZGVycyI6W10sInJlc3VsdHNCdWlsZGVycyI6W10s
ImNvbmZpZ092ZXJyaWRlcyI6W119fQ==

• To use this preset, you must first register function catchWrite in tools.js. To do this, insert the following code into the JavaScript Editor and save:

Tools.prototype.catchWrite = function(script) {
var output = '';
    var document = {
        write: function(s) {
            output += s;
        }
    };

eval(script);

return output;
}

As a result we get a list of almost 13K unique phone numbers. The average speed of parsing on 300 threads was almost 4k queries per minute.

P.S. To use presets in this article is needed A-Parser version not below 1.1.318.