Reddit::Posts - Reddit post scraper
Overview of the Reddit::Posts scraper
Reddit::Posts - Reddit post scraper.Collects a list of messages (posts) and a wealth of information for each of them from the service of the same name.
You can use automatic query expansion, substitution of subqueries from files, iteration of alphanumeric combinations, and lists to obtain the maximum possible number of results.
A-Parser functionality allows you to save Reddit::Posts scraper settings for future use (presets), set scraping schedules, and much more.
Saving results is possible in the form and structure you need, thanks to the built-in powerful Template Toolkit which allows applying additional logic to results and outputting data in various formats, including JSON, SQL, and CSV.
Collected data
Array of messages (posts):
- Link to the post
- Title and flair
- Rating, number of comments, and number of awards
- Creation date
- Community where the post was published
- Author and their flair
- Post content: text in markdown, link to media content, and link to an external resource
- Whether the post is promotional
Features
- Specifying the number of pages to scrape
- Specifying the result sorting method
- Selecting the time range for results
- Ability to scrape within a specific community
Use Cases
- Any scenarios where you need to obtain data about messages (posts) on Reddit
Queries
Several query options are supported:
Topic links
Example:
https://www.reddit.com/t/bitcoin/
https://www.reddit.com/t/kim_kardashian/
By default, the result will output a list of links to posts, for example:
https://www.reddit.com/r/Bitcoin/comments/14nbyy2/i_took_out_a_35000_loan_to_buy_bitcoin_1_year/
https://www.reddit.com/r/CryptoCurrency/comments/14guprs/bitcoin_is_up_75_since_jim_cramer_told_investors/
https://www.reddit.com/r/Bitcoin/comments/14opp2t/this_guy_was_paid_32_bitcoin_to_hold_up_this_sign/
https://www.reddit.com/r/CryptoCurrency/comments/14ivx43/nearly_69_of_all_bitcoin_supply_did_not_move_in/
https://www.reddit.com/r/CryptoCurrency/comments/149vy0o/bitcoin_dips_below_25k_for_the_first_time_in_3/
...
Community links
Parameters in links indicating time and sorting of results are also taken into account, while those specified in settings are ignored. Example:
https://www.reddit.com/r/nba/
https://www.reddit.com/r/OrlandoMagic/top/?t=month
By default, the result will output a list of links to posts, for example:
https://www.reddit.com/r/OrlandoMagic/comments/14a5br2/
https://www.reddit.com/r/OrlandoMagic/comments/14nqfk1/keep_mo_or_no_mo/
https://www.reddit.com/r/nba/comments/14nfzki/202324_nba_free_agent_tracker/
https://www.reddit.com/user/Grammarly/comments/14ghtld/verbessere_deine_schreibfertigkeit_auf_englisch/
https://www.reddit.com/r/nba/comments/14r4l4s/vernon_dillon_brooks_took_991_shots_last_year_he/
https://www.reddit.com/r/nba/comments/14ql1es/highlight_matt_devlin_inexplicably_yells_punjabi/
https://www.reddit.com/user/TelekomShop/comments/yqkina/der_highspeedhotspot_zum_mitnehmen_die_speedbox/
https://www.reddit.com/r/nba/comments/14qysvi/michael_jordan_with_the_spin_hanging_onehanded/
https://www.reddit.com/r/nba/comments/14qxrep/dwyane_wade_leads_the_redeem_team_with_27_points/
...
Keywords
Example:
wordpress features
parser
By default, the result will output a list of links to posts, for example:
https://www.reddit.com/r/ShitpostXIV/comments/14511em/i_am_a_proud_grey_parser/
https://www.reddit.com/r/opengl/comments/147sbjk/4_hours_of_my_obj_parser_so_far/
https://www.reddit.com/r/Compilers/comments/14pi9xh/demystifying_pratt_parsers/
https://www.reddit.com/r/ZETTAHOST/comments/11qdg99/how_to_change_the_wordpress_featured_image_size/
https://www.reddit.com/r/Wordpress/comments/14p1k2p/what_features_is_wordpress_missing_i_want_to_help/
https://www.reddit.com/r/Wordpress/comments/13q8g5x/is_it_possible_and_advisable_to_build_a_website/
...
Keywords and community links
The scraper supports searching by keyword within a specific community. To do this, you must specify the keyword and the community link separated by a space in the query. Example:
jesus https://www.reddit.com/r/atheism/
stage 3 https://www.reddit.com/r/Audi/
By default, the result will output a list of links to posts, for example:
https://www.reddit.com/r/Audi/comments/vi6cs5/thoughts_on_used_stage_3_2017_a3/
https://www.reddit.com/r/Audi/comments/lfvjuo/just_picked_up_this_beauty_stage_3_b5_s4/
https://www.reddit.com/r/Audi/comments/ssr8ui/anyone_else_track_their_audis_ttrs_stage_3_big/
https://www.reddit.com/r/atheism/comments/14lq0y6/heaven_and_hell_are_not_what_jesus_preached/
https://www.reddit.com/r/atheism/comments/13gxzj6/so_jesus_freaks_can_shove_their_religion_onto/
https://www.reddit.com/r/atheism/comments/13b8kl6/chris_pratt_compares_his_struggles_to_jesus/
https://www.reddit.com/r/atheism/comments/137k88b/artwork_of_jesus_surrounded_by_hot_leather/
...
Result output options
A-Parser supports flexible result formatting thanks to the built-in Template Toolkit, which allows it to output results in arbitrary forms, as well as structured formats like CSV or JSON.
Possible settings
| Parameter | Default value | Description |
|---|---|---|
| Pages count | 5 | Number of result pages |
| Sort | Relevance | Result sorting |
| Time | All time | Result time |
| Use HTTP/2 transport | ☑ | Determines whether to use HTTP/2 instead of HTTP/1.1 |