How to Scrape and Schedule Tweets For Your Profile
Part of running a good Twitter profile is having good, interesting content to share with your followers. You want to keep a minimum level of activity, often 2-3 tweets per day, but that gets difficult if all you have to go on is your own content. You can only tweet about your blog or your product so many times before people unfollow you.
Now, if you’re doing this the “organic” way, you’re going to end up spending a lot of time on Twitter. You need to be able to filter through your own organic feed, looking for content of sufficient quality that you’d want to share it with your followers. You might also need to do the same thing with list feeds and search results. This is particularly true if you’ve followed thousands of accounts yourself; it can be virtually impossible to get a good feel for what’s going on with so much noise.
Instead of doing all that work manually and tediously, why not let a trained app take over? There are a few different content curation engines you can use for Twitter, so let’s talk about them.
Curation With UpContent
One particular app I’ve been using recently is UpContent. UpContent is a content curation app that hooks into various social media scheduling apps, like Twitter itself directly or via HootSuite.
It’s pretty simple, all things considered. UpContent allows you to create a query, which can be as simple or as complex as you want it to be. The query is basically a string of keywords and operators, the simple AND/OR/NOT options, to help refine the content you’re getting.
Piece together a query and tell UpContent to run, and it will search the web for recent and valuable content that fits your description. They scrape the content into their own platform, so you can read full blog posts or social media posts directly from your dashboard. You can then create and schedule posts about those topics if you so desire.
Building a query can be a tricky task and will likely take a few iterations to get correct. If you put in too many keywords with too many negative keywords that are similar, you’ll end up with little or nothing in your results screen. If you put in generic keywords with little refinement, you’ll end up with generic viral results that everyone sees. Now, there’s nothing necessarily wrong with using those kinds of results, but they aren’t necessarily engaging. After all, your audience has probably seen them from other sources before.
If you don’t see anything you want to use, you can refine your query. If you see items you want to use, you can either create and schedule tweets immediately, or you can favorite the content within UpContent and save it for later. You’ll have a “saved” page full of the content you’ve favorited for use later, which you can browse at any time. To share th content on Twitter, you just need to click the arrow next to the card for that piece of content. You’ll need to authorize your account to post for you – they use Buffer or HootSuite – and it will be able to do so.
When you run a query, content is ranked according to four metrics. These metrics help you decide if that content is useful to you, or if it’s just appearing due to coincidence and popularity. The four categories are:
Relevance. Relevance is simply a measure of how applicable the content is to the query. Think of this like Google’s search results order for a similar query; the more relevant content floats to the top.
Recency. This is an optional sort you can apply to make sure you see the most recent content first. I like to use this filter to make sure I’m sharing stuff other accounts in my niche haven’t found yet. The more original the content you’re curating, the more value people will place upon you as a source of interesting content.
Shareability. This is a metric UpContent makes themselves. It’s a score they assign based on a variety of factors they identify when they scrape the content, and is a decent indication of whether or not the content is likely to be shared. More shareable content means more people on your Twitter feed will engage with it, and also tends to indicate that the content is interesting enough to spur a conversation. It’s not necessarily the most robust metric, but it does take a number of factors into consideration, including the presence of a Twitter Card, certain common keywords, and other clickthrough metrics.
Influence. The fourth and final metric is an influence score based on how likely a piece of content is to get links from other pieces of related content. UpContent doesn’t go into how they measure it, but it’s a semi-reliable way to tell how likely one piece of content is to get links and shares compared to another.
Obviously all of this is small scale indexing and analysis compared to some of the larger analytics companies out there. It doesn’t need to be very complex, though. When all you’re doing is finding content to curate on Twitter, you just need something that scores well regardless of what goes into that score. As long as the metrics are the same for everything they judge, it doesn’t really matter what those metrics are.
It’s worth noting that some major sources of content make it surprisingly difficult to simply scrape in this manner, likely as a way to prevent botters and content thieves from profiting off their page. It’s an annoyance for legitimate apps and users, though, so it’s a toss-up whether it’s worth it. Forbes is one big example; their bot-catching interstitial welcome screen prevents UpContent from really scraping those posts for you to read. Of course, that’s an easy remedy; just check Forbes yourself.
UpContent can be used for free, but you’re limited to just one user and just one topic. You have third party integrations, an activity log, and access to support, but that’s about it. The worst part about this is that you often want to have more than one query to pull content from different related niches. With the free personal plan, your only option is to continually edit and re-run queries. You can “save” different queries in a text file or something, but it’s still a pain to swap them out every time you want to see new results.
You can bump yourself up to unlimited topics, along with an RSS feed for custom topics, whitelabel dashboard, and training, for $10 per month. If you want team members, premium training, and priority support, you can pay $50 per month, but that’s basically overkill to me so I haven’t looked into it.
Crate
Like a loot crate, but better! Crate is another app that helps you curate content for your Twitter feed. You sign into it with Twitter and you start to build a crate.
What is a create in this context? It’s basically a box of content delivered to you in a digest form on a regular basis. You make a crate by specifying content sources you like. It might be specific blogs you like, or influencers who content you enjoy, or specific journalists who report on your niche fairly often. It could also be your competitors, so you can spy on them, or just generic keywords and topics, to get a broad overview of your niche.
You’ll get all kinds of content, too, not just blog posts. You also get infographics, data articles, and even videos if there’s enough information to verify that it’s about your topic. When the crate is delivered, you can browse through what they found and either post it to Twitter directly or use Buffer to schedule it for later posting.
Crate has one downside, which is that their index of content isn’t all that large. They boast over 200,000 articles in their index, but that isn’t even all that much these days. I’ve also heard some people having issues paying for an account, which could be a problem.
Crate has three plans. The free plan allows up to three crates, which is already better than UpContent, but you’re limited to only 25 shares per month and only 10 results per crate, both of which are much more limiting. You also don’t get Buffer integration with the free plan.
The plus plan is ten crates and 50 shares per month, with 50 results per crate. It also enables Buffer integration. It’s also $19 per month, which isn’t bad for the service, but could run out fairly quickly. They also have a $39 per month plan that is unlimited everything. I actually like this; it’s refreshing to see a truly unlimited plan for a cheap fee rather than one of those “contact us for an enterprise plan” gimmicks that’s always too expensive for small businesses.
Beat the Drum
Another alternative that works on a broader scale is DrumUp. DrumUp does the same sort of thing as the previous apps; it scans the internet based on your seed keywords and looks for content that fits your intentions. They trend towards fresh content and they scan constantly in real time to provide suggestions as soon as possible. If you want to supplement their own crawling, you can add the RSS feeds or social streams of specific accounts you want to follow as well.
Two things I really like about DrumUp. First, they have a Chrome extension that makes it really easy to curate content on the fly. It’s like Buffer, but instead of just going right into your schedule, it goes into your feeds along with the other DrumUp content. That way it’s all in one place for future curation, or immediate curation as you’re reading.
Secondly, they have mobile apps for both iPhone and Android. These apps allow you to access your dashboard and do your curation and scheduling on the fly. You can manage your Twitter feed for the day on the ride into work, assuming you take public transit and you’re not driving. Don’t play with your phone while driving, kids.
DrumUp starts at $15 per month. That plan gets you three social accounts, ten posts per day per account, three content streams, three RSS feeds, a stored library of up to 1,000 posts, and some additional content tools. For example, you can upload images, schedule custom posts, get hashtag and mention suggestions, and set up repeat posting for certain high value posts.
Their next step up is $39 per month and bumps up all the numbers as you might expect. You get 20 accounts, 15 posts, 20 streams, 5 RSS feeds, twice the number of posts in the library, but you also get analytics. Another step up gets you team members and higher numbers still, for $79 per month. Then the agency version is $159 for even more numbers, many of which are unlimited or can be raised through contacting the company directly. I’ve never had need for 100 different social accounts at once or a dozen different team members, so someone with a larger business than mine will need to tell me if that’s worthwhile.
Regardless of which of the options you choose, content suggestions and curation can be streamlined dramatically. There are a lot of different curation engines, but they all do the same basic thing; feed interesting, compelling content directly to you, so you don’t have to go out and find it. When the content is delivered to you, all you have to do is filter it and share it, and you’ll be able to watch the engagement roll in.