SE::AOL - search.aol.com search engine scraper
AOL scraper overview
AOL scraper is a scraper for AOL search results. Thanks to the AOL scraper, you will be able to obtain huge databases of links ready for further use. You can use queries in the same form as you enter them in the AOL search bar, including search operators (contains, intitle, language, site, etc.). More details on the official AOL Search FAQs page.
A-Parser functionality allows you to save AOL scraper settings for future use (presets), set scraping schedules, and much more. You can use automatic query multiplication, substitution of subqueries from files, brute-forcing of alphanumeric combinations and lists to get the maximum possible number of results.
Saving results is possible in the form and structure you need, thanks to the built-in powerful Template Toolkit template engine, which allows you to apply additional logic to results and output data in various formats, including JSON, SQL, and CSV.
Collected data
- Number of results for the query
- Links, anchors, and snippets from the search results
Features
The AOL search engine scraper has the following features and settings:
- Support for all AOL search operators (site:, prefer:, url:, feed:, etc.). More details on the official AOL Search FAQs page
- Scrapes the maximum number of results provided by AOL - 100 pages with 20 elements in the results of each page
- Support for result periods (Anytime, Day, Week, Month)
Use Cases
- Collecting link databases - for A-Poster, XRumer, AllSubmitter, etc.
- Assessing competition for keywords
- Scraping US Google search results
- Searching for backlinks (mentions) of sites
- Checking site indexing
- Searching for vulnerable sites
- Any other cases involving AOL scraping in one form or another
Queries
As queries, you should specify search phrases exactly as if they were entered directly into the AOL search form, for example:
test
buy house
site:www.realtor.com
feed:football
a-parser loc:ca
Query substitutions
You can use built-in macros to multiply queries; for example, if we want to get a very large database of forums, we specify several main queries in different languages:
forum
forum
foro
论坛
In the query format, we specify a character brute-force from a to zzzz; this method allows for maximum rotation of search results and obtaining many new unique results:
$query {az:a:zzzz}
This macro will create 475254 additional queries for each original search query, which in total will give 4 x 475254 = 1901016 search queries—an impressive figure, but no problem at all for A-Parser. At a speed of 2000 queries per minute, such a task will be processed in just 16 hours.
Using operators
You can use search operators in the query format, so they will be automatically added to each query from your list:
site:$query
Output results examples
A-Parser supports flexible result formatting thanks to the built-in Template Toolkit template engine, which allows it to output results in arbitrary form, as well as in structured formats like CSV or JSON
Exporting a list of links
Links + anchors + snippets with position output
Outputting links, anchors, and snippets to a CSV table
Saving related keywords
Keyword competition
Checking link indexing
Saving in SQL format
Dumping results to JSON
Results processing
A-Parser allows processing results directly during scraping; in this section, we have listed the most popular cases for the AOL scraper
Link deduplication
Link deduplication by domain
Extracting domains
Removing tags from anchors and snippets
Filtering links by inclusion
Possible settings
| Parameter name | Default value | Description |
|---|---|---|
| Pages count | 5 | Number of pages to scrape (from 1 to 100) |
| Period | Anytime | Result period (Anytime, Day, Week, Month) |
| AOL domain | search.aol.com | Choice of search results country (search.aol.com, search.aol.co.uk, recherche.aol.fr, suche.aol.de) |