WP Content Crawler – Get content from almost any site, automatically!

Get content from almost any site to your WordPress blog, automatically!

Docs
| Demo
| Website
| Join our Discord server! (New)

FOR WHAT IT CAN BE USED

• Create a personal site which collects news, posts, etc. from your favorite sites to see them in one place
• Use it with WooCommerce to collect products from shopping sites
• Collect products from affiliate programs to make money
• Collect posts to create a test environment for your plugin/theme
• Collect plugins, themes, apps, images from other sites to create a collection of them
• Keep track of competitors
• You can imagine anything. The internet is full of contents

Before you buy, make sure you do the following:

• Watch the quick start video and use the plugin in the
  demo. You can also watch the other
  video tutorials to learn how to use
  the plugin. There are also many guides
  explaining how to do certain things with the plugin.
• Make sure the plugin can retrieve the data from the site you want to crawl by following the instructions in
  
  Can I get content from X site? FAQ.
• If you are still not sure if the plugin can retrieve content from a specific site, ask us in the comments
  section.
• You can check the FAQs
  if you have any questions. If the answer to your question is not there, you can always ask us in the comments
  section.

30-SECOND* SITE SETUP WITH CONFIG HELPER

*Config Helper is displayed when creating a new site. Its purpose is to speed up the initial setup for a site.
Although Config Helper works for many sites, not every site can be configured this easily, meaning that certain
sites may require a manual and more elaborate setup. Even for those sites, you can try to create the basic
configuration with Config Helper and then modify the site settings later. Additionally, it is possible to exit
Config Helper any time or disable Config Helper entirely.

QUICK START

HOW IT WORKS

It’s all about CSS selectors and you can learn how to use them in minutes by watching the introduction tutorial. The plugin’s Visual Inspector tool also helps you find CSS selectors easily by clicking onto the elements in the target sites. Here is the gist of it:

WHAT WP CONTENT CRAWLER CAN DO

Here is the list of some features of WP Content Crawler. To learn about all of the features, please see the features table below.

HTML code of social media posts existing in the target post page is automatically converted to short codes. By this
way, they are displayed on the front end of your website correctly. Additionally, all the iframe elements are
converted to a short code. An iframe short code is displayed only if its source is trusted. If the source of an
iframe is not trusted by default, you can manually add a trusted domain so that the iframe is displayed. With this
method, you can display media from third-party sites securely. Websites whose media are embedded automatically
include Instagram, Imgur, YouTube, Vimeo, TikTok, Kickstarter, Twitter, Pinterest, and so on.

SEE IT IN ACTION, LEARN IN MINUTES

VIDEO TUTORIALS

MAIN FEATURES

HAPPY CUSTOMERS

WHY WP CONTENT CRAWLER

Problems with crawling a website

• Not an easy task, requires advanced programming skills
• Every website is different and needs tailored crawling implementation
• Not just every website is different, but also pages of a single website can differ
• Pages and their source codes need to be investigated intensively to come up with a crawling plan
• Knowing how to save certain information in a specific place in WordPress requires knowledge about the internal structure of WordPress and how WordPress works
• If certain information should be saved into a specific field defined by a third-party plugin, one should modify the crawling implementation after researching for hours about how to save that information
• One should know about how HTML works and how to extract certain parts from HTML code
• One should handle all possible inconsistencies that might be in the source codes of websites to provide a robust solution that will keep working
• What if the posts need to be shared in regular time intervals?
• What if you want to crawl new posts added to a website after some time?
• What about translating the posts from one language to another?
• What if the posts needs to be paraphrased to provide a better search engine optimization for the website?
• What if some information should not be retrieved?
• What if certain information should be changed to make it suitable for your site?
• What if another site needs to be crawled, not just one?
• What if that other site needs a different crawling plan?
• What if you need to login to the website to crawl it?
• What if the website changes its source code?
• What if you want to update the crawled posts by recrawling them from the original website?
• What if you want to make sure if the information is retrieved exactly as you want it before automatically posting the posts to your website?
• What if you want to ensure your site’s security by making sure no malicious-code-executing code ends up in your site?
• And many more what-ifs that you might not even imagine unless you come across them

Our vision and mission

We believe that robust, reliable, and automated crawling capabilities should be available for anyone. We want to democratize this field by letting anyone have these capabilities, not just developers. With this purpose, we aim to provide a plugin that you will fall in love with and feel at home when using it. To let it accessible by anyone, we make the plugin low-cost and easy-to-use. We do not implement the features just to make sales. We plan and execute for the future. We always listen to your feedback and make required changes accordingly. We think that WordPress plugins should be developed with enterprise-level care. So, we intensively test the plugin before each release with automated end-to-end UI tests, currently over 1700 tests, that run in many different environments in the cloud for a total of over 40 hours to ensure the plugin is compatible with your server and WordPress environments and you, our valuable customers, get the quality and reliability you deserve.

How we solve these problems

We have been developing WP Content Crawler for almost 4 years such that we have come across almost all the what-ifs. Working with our customers and listening to their needs, we provide robust and reliable solutions to these problems. We believe that one should just provide from which site the information should be retrieved and what information should be retrieved from that website and then start crawling that site, without worrying about the complex behind-the-scenes operations.

To make it available to anyone, we provide a detailed online documentation that contains not just the description of the settings but how to use the settings to achieve your goals. Sometimes you might not feel like reading the documentation. We also provide interactive step-by-step guides that are available in the plugin, just one click away. You can start the interactive guides showing you step-by-step how you can do certain things any time and from any step you want.

One of the most distinctive features of WP Content Crawler is the ability to test almost any configuration. By this way, you will not come across any surprises after you enable automatic crawling. When testing, the errors related to your settings are shown so that you can fix them before they cause any problems.

WP Content Crawler has so many features that even we do not know how many of them are there. You can automatically crawl, update, and delete the posts, you can translate posts, spin posts, you can even define what fields need to be translated or spun if you do not want them all changed. You can find-replace almost anything. You can assign some information from the target post to a short code and place that information anywhere in the post. You can save WooCommerce products. You can save details for third-party plugins that we do not even know they exist. The features of the plugin are designed such that you feel that you are in control when you use them. We make them as flexible as possible to make them fit your needs. When designing new features, we always keep in mind that you might need a more advanced version of that feature and we design the features accordingly. We ensure that the features and the entire code of the plugin are maintainable and extendable so that we can always improve the plugin.

CHANGELOG
Changelog is kept in the documentation site. Click here to see the changelog.