Skip to main content

SE::Startpage - startpage.com search engine scraper

Startpage

Overview of the scraper​

Startpage search results scraper. Thanks to the Startpage scraper, you will be able to obtain large databases of links ready for further use. You can use queries in the same format as you enter them in the Startpage search bar, including search operators (site, inurl, etc.).

A-Parser functionality allows you to save Startpage scraper settings for future use (presets), set scraping schedules, and much more. You can use automatic query multiplication, substitution of subqueries from files, brute-forcing of alphanumeric combinations and lists to obtain the maximum possible number of results.

Saving results is possible in the form and structure you need, thanks to the built-in powerful Template Toolkit which allows applying additional logic to results and outputting data in various formats, including JSON, SQL, and CSV.

Collected data​

  • Links, anchors, and snippets from search results
Collected data

Features​

  • Supports selection of country, search language, and page language
  • Ability to output anchors, links, and snippets together or separately
  • Ability to specify the number of results to be output
  • Specification of result size (10 or 20 results)

Use Cases​

  • Collecting databases of links, anchors, and snippets
  • Obtaining a list of websites most frequently mentioned in search engines
  • Any other use cases for obtaining information

Queries​

Words, phrases, and word combinations are specified as queries in the same form as they are entered into the search engine. Example:

test   
site:http://test.ru  
red roses

Query substitutions​

You can use built-in macros to multiply queries; for example, if we want to get a very large database of forums, we specify several main queries in different languages:

forum
forum
foro
论坛

In the query format, we will specify a character brute-force from a to zzzz; this method allows for maximum rotation of search results and obtaining many new unique results:

$query {az:a:zzzz}

This macro will create 475254 additional queries for each original search query, which in total will give 4 x 475254 = 1901016 search queries; the figure is impressive, but it is not a problem at all for A-Parser. At a speed of 2000 queries per minute, such a task will be processed in just 16 hours.

Using operators​

You can use search operators in the query format, so it will be automatically added to each query from your list:

site:$query

Output results examples​

A-Parser supports flexible result formatting thanks to the built-in Template Toolkit, which allows it to output results in arbitrary form, as well as in structured form, such as CSV or JSON

Exporting a list of links​

Same as in SE::Google.

Same as in SE::Google.

Same as in SE::Google.

Saving in SQL format​

Same as in SE::Google.

Dumping results to JSON​

Same as in SE::Google.

Results processing​

A-Parser allows processing results directly during scraping; in this section, we have provided the most popular cases for the Startpage scraper

Same as in SE::Google.

Same as in SE::Google.

Extracting domains​

Same as in SE::Google.

Removing tags from anchors and snippets​

Same as in SE::Google.

Same as in SE::Google.

Possible settings​

note

Common settings for all scrapers

Parameter nameDefault valueDescription
Pages count5Number of pages to scrape (from 1 to 50)
Family filterFilter depending on searchSelection of filtering level (Filter all results / Filter depending on search / Do not filter my results)
PeriodAny timeSelection of result period (Any time / Past 24 hours / Past week / Past month / Past year)
Links per page10Result size (10 / 20)
Results languageEnglishSelection of result language
Page languageEnglishSelection of page language
Search countryAllSelection of the country from which the search is performed