Today, on the example of site http://www.chegg.com/ we learn how to parse the information that is loaded by JS-script.
1) In this link, as described above, the content really is loaded by JS script. But in the end it too somewhere is receiving data, and therefore make somewhere request. We just need to find this query, and then use it. For this we use Developer Tools (Ctrl + Shift + I, tab Network), which are available in the browser Chrome. Open the link above and analyze it. We will see a lot of different requests and responses. After reviewing all of the answers on the subject information that interests us (for example you can search by the first name: "Cengage Advantage Books ..."), we find the desired us query.
2) Check and analyze the founded query. For a start it will open in a new browser tab and see whether it gives what we need.
As you can see - yes, there is the name of the book and link to it (highlighted in green), as well as a lot of other information. If you look closely - you can see that this data in format JSON. You can verify this by inserting data into Template tester at A-parser and clicking Pretiffy JSON (highlighted green square).
3) As you can see, it was only the first page, in which only 10 results. But what if you need all the rest? Analyzing found earlier query:
If instead of the bold one put 2 and open the link in a browser - we see that the data out is what we need (2-page) in already familiar JSON. Well, now there are all loaded into the A-parser.
4) For parsing we will use
Net::HTTP. For obtaining the necessary data from all content using Parse custom result and regular expressions. By the way, the regular expression in this example, you must be very careful, because a result there are has "relatedItems" (something like similar books), and with them on the first page, you can get 18 results instead of 10.
Notes:
1) In this link, as described above, the content really is loaded by JS script. But in the end it too somewhere is receiving data, and therefore make somewhere request. We just need to find this query, and then use it. For this we use Developer Tools (Ctrl + Shift + I, tab Network), which are available in the browser Chrome. Open the link above and analyze it. We will see a lot of different requests and responses. After reviewing all of the answers on the subject information that interests us (for example you can search by the first name: "Cengage Advantage Books ..."), we find the desired us query.
2) Check and analyze the founded query. For a start it will open in a new browser tab and see whether it gives what we need.
As you can see - yes, there is the name of the book and link to it (highlighted in green), as well as a lot of other information. If you look closely - you can see that this data in format JSON. You can verify this by inserting data into Template tester at A-parser and clicking Pretiffy JSON (highlighted green square).
3) As you can see, it was only the first page, in which only 10 results. But what if you need all the rest? Analyzing found earlier query:
If instead of the bold one put 2 and open the link in a browser - we see that the data out is what we need (2-page) in already familiar JSON. Well, now there are all loaded into the A-parser.
4) For parsing we will use
Net::HTTP. For obtaining the necessary data from all content using Parse custom result and regular expressions. By the way, the regular expression in this example, you must be very careful, because a result there are has "relatedItems" (something like similar books), and with them on the first page, you can get 18 results instead of 10.
Notes:
- The input is previously found request, but instead numbers substitute macroes of busting numbers. In this example, all pages will be passed with the 1st of 1000-th.
- The result is a file with following information: Author - Title: Link to the page. Optionally you can change the format of the result as you like, changing the regular expression, and the data they collected, as well as changing himself Result format. Please note, there is still using the Result builder is replaced "\ /" to "/" because in JSON symbol "/" shielded.
Code:
eyJwcmVzZXQiOiJodHRwOi8vYS1wYXJzZXIuY29tL3RocmVhZHMvMTY5OSIsInZh
bHVlIjp7InByZXNldCI6Imh0dHA6Ly9hLXBhcnNlci5jb20vdGhyZWFkcy8xNjk5
IiwicGFyc2VycyI6W1siTmV0OjpIVFRQIiwiZGVmYXVsdCIseyJ0eXBlIjoib3Zl
cnJpZGUiLCJpZCI6ImZvcm1hdHJlc3VsdCIsInZhbHVlIjoiJGJvb2tzLmZvcm1h
dCgnJGF1dGhvciAtICR0aXRsZTogaHR0cDovL3d3dy5jaGVnZy5jb20vJGxpbmtc
XG4nKSJ9LHsidHlwZSI6ImN1c3RvbVJlc3VsdCIsInJlc3VsdCI6ImRhdGEiLCJy
ZWdleCI6IlssfFxcW117XCJ0eXBlXCI6Lis/XCJwcmltYXJ5QXV0aG9yXCI6XCIo
Lio/KVwiLis/XCJ0aXRsZVwiOlwiKC4rPylcIi4rP1wicGRwXCI6XCIoLis/KVwi
IiwicmVnZXhUeXBlIjoiZyIsInJlc3VsdFR5cGUiOiJhcnJheSIsImFycmF5TmFt
ZSI6ImJvb2tzIiwicmVzdWx0cyI6WyJhdXRob3IiLCJ0aXRsZSIsImxpbmsiXX1d
XSwicmVzdWx0c0Zvcm1hdCI6IiRwMS5wcmVzZXQiLCJyZXN1bHRzU2F2ZVRvIjoi
ZmlsZSIsInJlc3VsdHNGaWxlTmFtZSI6IiRkYXRlZmlsZS5mb3JtYXQoKS50eHQi
LCJhZGRpdGlvbmFsRm9ybWF0cyI6W10sInJlc3VsdHNVbmlxdWUiOiJubyIsInF1
ZXJ5Rm9ybWF0IjpbIiRxdWVyeSJdLCJ1bmlxdWVRdWVyaWVzIjpmYWxzZSwic2F2
ZUZhaWxlZFF1ZXJpZXMiOmZhbHNlLCJpdGVyYXRvck9wdGlvbnMiOnsib25BbGxM
ZXZlbHMiOmZhbHNlLCJxdWVyeUJ1aWxkZXJzQWZ0ZXJJdGVyYXRvciI6ZmFsc2V9
LCJyZXN1bHRzT3B0aW9ucyI6eyJvdmVyd3JpdGUiOmZhbHNlfSwiZG9Mb2ciOiJu
byIsImtlZXBVbmlxdWUiOiJObyIsIm1vcmVPcHRpb25zIjpmYWxzZSwicmVzdWx0
c1ByZXBlbmQiOiIiLCJyZXN1bHRzQXBwZW5kIjoiIiwicXVlcnlCdWlsZGVycyI6
W10sInJlc3VsdHNCdWlsZGVycyI6W3sic291cmNlIjpbMCxbImJvb2tzIiwibGlu
ayJdXSwidHlwZSI6InN0cmluZ1JlcGxhY2UiLCJhcnJheSI6ImJvb2tzIiwic2Vh
cmNoIjoiXFwvIiwicmVwbGFjZSI6Ii8iLCJ0byI6ImxpbmsifV0sImNvbmZpZ092
ZXJyaWRlcyI6W119fQ==