I'm relatively new to Python and I'd like to learn how to read data from websites. Specifically, I'm interested in how I can get started with BeautifulSoup or Selenium without getting blocked by the sites. Does anyone have a simple beginner project for me?
BeautifulSoup is actually overkill for most beginner projects - start with simple HTTP requests instead and check out the HTML structure in your browser before you need a parser framework. For starters, `requests` plus a bit of HTML parsing is totally enough, and you'll avoid the pitfalls with Selenium right away (it's slow and gets blocked faster). Try it out on a static website that doesn't have issues with scraping - local news, Wikipedia, or a simple API are better than going after the big players right off the bat. Pay attention to `robots.txt`, add delays between requests, and identify yourself as a browser (User-Agent), then most sites won't have a problem with you.
Start with `requests` and `BeautifulSoup` together, that's the standard combo and not overkill but exactly right for beginners. More importantly: check the site's `robots.txt` first and add delays between requests (`time.sleep()`), otherwise you'll get blocked pretty quick. For your first project I'd go with something simple like a news site or a product list where the structure isn't too messy. You don't need Selenium until the page loads JavaScript - that makes everything more complicated and isn't necessary at the beginning.
Both approaches have their merits, but I'd go for a middle ground: `requests` + `BeautifulSoup` is really the most practical combo to start with, because you'll solve 90% of all beginner tasks with it. But what a lot of people overlook is that you should check first whether the site is even allowed to be scraped - look at the `robots.txt` and add a User-Agent so the servers know there's a human behind it (don't just use the `urllib` default). As a concrete project: try extracting article titles and links from a simple site like a news website or a forum - that way you'll quickly see how HTML selectors work, and it's not in an ethical gray area. You'll need Selenium later, once the site has to load JavaScript.
Log into answer.
