How to Scrape Facebook Posts for Free Content Ideas
Facebook scraping is both easier and harder than you might expect. On the one hand, they don’t like bots that just come through and scrape content. The rate limits and the bot-filtering algorithms are likely to restrict or even ban you if you’re draining up too many resources.
On the other hand, Facebook has an API – the Graph API – that they use to allow apps to simply take data without needing to load everything on a full page. A good Facebook scraper is going to use API access to get data.
If you know how APIs work, you can read into the Graph API documentation in the Facebook developer center. What we’re looking at today is the /page node, which gives you information on posts.
Facebook’s API is no longer public, unfortunately, which means you need to make an app to get an access token to use for your scraping.
As far as the program to actually do the scraping, in this case I recommend this script made by Max Woolf. The linked article documents how it works and the data you’ll get, as well as the issues that can come from the data. Since for content ideas, you don’t need the time of the post necessarily. Mostly all I would look at is the content of the post and the likes the post received. Alternatively, if you don’t mind paying for a tool, you can buy the Facebook Data Scraper for $60. However, their website is currently offline, so it might be tricky to find a working copy.
Identifying Target Pages
Before you can scrape data to look for content ideas, you need to figure out what pages you actually want to scrape. There are probably millions of pages on Facebook, and there’s no way you can scrape them all.
The first way to narrow down the list of “everything” is to restrict it to your industry. If you’re a fashion blog, you would probably want to limit yourself to just fashion and fashion-adjacent industries. You can scrape shoe stores, apparel stores, beauty blogs, and other fashion blogs. You can also scrape health and fitness blogs, diet blogs, and other “keep your body in shape for fashion” style content.
Think about where your industry intersects with other industries. A fashion blog could scrape IT blogs and look for new trends in wearable technology. A fashion blog could scrape marketing blogs for ideas about how to market to a fashion-focused audience. Intersection gives you ideas.
Within the industry, you need to start identifying specific Facebook pages you want to scrape. There are two ways to do this. The first is by links, and the second is by keywords.
The links method involves going to your website and doing some link auditing. Pull your external links and a list of your backlinks, then look for the kinds of brands, companies, and sites you’re linking to. You can discard the Amazon affiliate links, the blog comment links to spam sites, the backlinks from unrelated sites, and so forth. What you’re looking for is ideas on sites that might have Facebook pages.
Whenever you find a link, either one you made or one that links to you, and the domain looks interesting, check to see if they have a Facebook page. If they do, and the Facebook page is active, add it to your list.
The keyword method is much more like keyword or competitive research. Simply think up – or you an existing list – keywords that relate to your industry. A fashion site can have a wide range of keywords, from brands of clothing or makeup to techniques to seasonal outfits to who knows what else. You’ll be more familiar with your brand than anyone else, so you can come up with the keywords.
Run the keywords through Google search one at a time and check out the “pages” tab of the search results, up at the top. This will show you the top pages that are most relevant to your keywords.
Protip: you can do the same thing with keywords in the Facebook ads manager to do some keyword targeting research. The ads manager will give you more immediately accessible detail on the pages, so you’re able to see which pages are most or least likely to be active and valuable.
You’ll have to filter the search results, since many keywords will lead to the same top few sites. You can do this by adding more long-tail keywords, or you can simply scroll past as many of the duplicates as necessary to get to new results.
Gradually build out your list of potential target Facebook pages and set them aside for use in the next step.
A Word of Caution
Before we begin, you should know two risks involved in this process. The first is that you may be tempted to use third party tools to do everything for you. For instance, people will often advertise tools like Fan Marketer, which is one of the spammiest websites I’ve ever seen and is almost guaranteed to be a scam. The tool is not used for content ideation, it’s used for complete automation. In other words, it’s used to grow an organic-looking Facebook page without having to look at it. You know, the kind that are eventually sold to spammers or used to relentlessly drop affiliate links to make money.
The second risk is that of scraping running afoul of Facebook’s data protection policies. This report – old as it is – is a tale of how scraping almost got someone sued into oblivion by Facebook. For the most part, Facebook is fine with scraping on a small or individual use scale. It’s only when this guy scraped immense amounts of data and then offered it for free that Facebook got upset. Why? Well, that’s data Facebook is selling to their ad partners. If you publish it free, they can’t sell it, and it hurts their bottom line. The lesson here is to not publish your scraped data, and be careful about the volumes you harvest.
Scrape Content
Now it’s time to put the scraper of your choice to use. You will need to query each of the pages in question and let the scraper run. What you’re looking for is the content of a post, the number of reactions the post received, and engagement metrics if possible. You can’t always get an accurate reading of engagement, but that’s fine; all you need is an idea of how well a given post performed.
When you’re done, you’ll have a spreadsheet full of data. Prune out or format the columns you don’t need. I generally end up with a new page of my spreadsheet for each Facebook page I scrape.
It’s worth noting here that Facebook’s API will occasionally hang slightly. This is most often the case when accessing old data. Any posts that are over a year old aren’t likely to be super valuable as content ideation sources, so it’s up to you if you want to work around this or not. Personally, when the scraper stalls out, I just move on to the next Page. It generally has enough data for me to find useful at that point.
Analyze Content
Now you have a big pile of data. What do you do with it?
Assuming you got a good reading of the reactions to the posts, you have a good way to see what posts did well and what didn’t. Just sort your data with the most reactions on top.
Some of the posts are going to be new product launches, announcements, news, and emotional trending content from when a global tragedy happened. Some will be about contests, or about some new business announcement. In short, a lot of the content at the top will be relevant to the fans of the business in question, but not to your fans or to other people in general.
You can safely remove all of this from your spreadsheet. It’s not going to give you any inspiration beyond “hey, running a contest/launching a product/expressing sympathy to shooting victims is good for engagement.” There’s no reason to keep it cluttering up your data feed.
What you’re left with are the posts with curated content, the posts with organic and shared content, and the kinds of content you might be able to imitate. You can see, right up at the top, the content that got the most engagement when it was posted.
Maybe you find out that people who follow a makeup blog particularly love exotic color combinations. Maybe you find out that there’s a lot of backlash against a specific line of clothing. Maybe you discover that people absolutely love a post about nudity on the runway.
These are all good content ideas. You can’t just take them wholesale, but you can use them as the foundation of content you can create. If people love content about exotic makeup colors, you can do some research and promote your best selling products along with it. If people really dislike a given set of products, you can write a review of them calling out their terrible business practices.
Take what the original post did, and fact-check to make sure it still holds true. You don’t want to disparage a brand that has changed ownership and changed their ways, after all. Once you’ve confirmed that it’s still relevant, you can write your own post. Do what the successful post did, but better.
The BuzzSumo Option
If you want to bypass a lot of the above work and effort, you can capitalize on someone else making it their business model to do it all for you. The ideas you generate are free, but the data you need to do it isn’t necessarily all that cheap.
BuzzSumo is a social data app that allows you to plug in a keyword – like the keywords that we came up with back in the setup for scraping – and will show you the most shared and most engaged content relating to that keyword.
Since we’re talking specifically about Facebook BuzzSumo has a tool to specifically analyze Facebook pages and identify the most shared content. The trouble is, in order to access the Facebook Analyzer, you need to pay for a pro plan. Moreso, it can’t even be a standard pro plan; it has to be either the Large or Enterprise level plans, which are $239 and $499+ per month, respectively. Very expensive, for the convenience of not having to do all of the work yourself.
Other Idea Sources
You don’t have to limit yourself to Facebook when you’re trying to come up with new ideas. Scanning and scraping Facebook posts is a good way to see specifically the length, phrasing, and formatting of the best content, but at the end of the day, all the formatting in the world won’t help you if your content is boring.
- Use topic generators like the Hubspot Blog Topic Generator. It’ll spit titles out at you and you can see if any of them are worthwhile to cover. All you need is a few keywords for it to use.
- Use blog aggregation sites like Alltop. News aggregators and blog aggregators aren’t great from a marketing perspective, but they give you an idea of what’s new and trending in your niche when you browse.
- Check out actual trends in news and search. Google Trends is excellent to see what’s rising and trending in your specific niche, and Twitter can show you more global trends with their hashtag sidebar.
Regardless of the method you choose, ideas are cheap. What matters is how well you’re able to adapt those ideas into serviceable posts, both for your site and for Facebook.