Crawlbase (formerly know as ProxyCrawl) service is a full scraping package for web scraping. This service does not only offers regular Smart Proxy but also offers a proxy API known as crawler API and a scraper API, among its range of services directed at web scrapers and web scraping. In this review, our focus would be on their crawler API, scraper API, and Smart Proxy.
Web Scraping Full Package Provider
- A vast array of services for web scraping provided
- You only get to pay for successful requests for their crawler and scraper API
- Geo-targeting supported with a good number of countries supported
- Over one million IP addresses in its pool
- Smart rotation of IP address
- Pricing is based on the number of successful requests
- Free trial available for new users to test out the service
The name of this platform captures what it offers to users. Aside from providing proxies, it offers services important for web crawling and scraping. However, it is not as popular for its proxy service as much as it is for its scraping and crawling services, as most users use them for their API services.
The services include the crawler API, Scraper API, crawler, lead API, screenshot API, and storage service for storing your scrapped data in the cloud. The vast array of services provided by the Crawlbase service has made it to be a full package for web scraping, and a good number of big businesses are using their service, including some Fortune 500 companies.
The service has proven to be one of the top providers of both proxy and scraping API, and their services can be termed as high quality. One thing I have come to like about their service is that it is reliable, fast, secure, and you are in good company.
However, this does not mean you should jump right in to start using the tool. It might interest you to know that just like every other service, it has its weakness, and you would only want to use it if the weaknesses are not a big deal to you. One thing I do not like about the service is the fact that its Smart Proxy service is highly limited to only businesses, and upon that, their pool has less than 2 million IPs.
Is the service worth its salt? That’s what we’ll discuss in this review.
The Crawlbase Solution
As stated earlier, this service offers users a good number of products that are useful for web scraping, crawling, and even other forms of automation. In this section of the review, let take a look at their popular products and their usefulness.
It acts as a proxy server but in the form of an API. All that’s required to get the content of a page is to send a simple API request, and you will get the content you want without the complication of thinking of blocked requests, IP leaks, proxy failure,Captchas, and even browser crash.
The Scraper API can be said to be an advanced version of the Crawling API as it is built on the crawling API but with a different endpoint.
Unlike in the case of the crawling API that would provide you content and parsing out required content is up to you, the Scraper API takes care of parsing. You can see it as a full scraper that does not only handle HTTP requests but also parsing.
Their 3rd popular service is their proxies. Their proxy pool is a mix of datacenter and residential IP addresses, which you can use for your web scraping and crawling tasks.
One thing you need to know is that for you to make use of this service, the email address you will use for registering must be a professional email from a business domain and not the regular Gmail and Yahoo mail.
Get 20% Life-time discount: put promo code “privateproxyreviews” to check out!
back to menu ↑
How Crawlbase Work
At its most basic system, the Crawlbase platform acts like a proxy server that sits between your device and web servers you want to scrape data from. However, you do not have to see it as regular proxy servers, as it does more than just masking your IP address.
Except for their smart that you can see like that, their crawling and Scraper API services are much more than that, even though they still acts as proxies in some ways. The difference in the working mechanism between the 3 is what is the processing done by the Crawlbase API as it receives your request.
When you are using any of their solutions described above, your web requests do not go to the web servers you send them to directly. Instead, they are routed through the Crawlbase server. This is because the URL for the page you want to request is not used directly in the HTTP request but appended as a parameter in a Crawlbase API URL.
When the request gets to Crawlbase, it decodes the URL, and then send the request on your behalf, taking note of your geo-targeting option, and rotating between IP address to avoid blocks.
back to menu ↑
Pricing & Features
You can only make use of any of the services provided by Crawlbase if your budget permits you. Generally, if you ask me, I will tell you that Crawlbase has a pocket-friendly pricing system that is customer-focused.
They do not have a universal pricing system for their services. Let take a look the pricing of each of product being considered here.
As you can see from the image above, the Crawling API has a flexible pricing system that does not force you to have a long-term contract. The pricing is based on successful requests, and it varies depending on the number of requests you buy.
The Scraper API has more straightforward and rigid pricing. As you can see from the above, the package you subscribe to will determine the features available to you.
For Smart Proxy, I must confess, the pricing can be said to be worth it compared to what other providers with more advanced features offer. Not to mention it offers a free trial. It currently keeps you crawling and scraping websites anonymously in more than 45 countries, covering the world’s most influential countries. So you can use their Smart Proxy service on Google, Amazon, and LinkedIn.
Features of Crawlbase Platform
- Support geo-targeting for selected locations across the globe, but it depends on the package you buy.
- Pricing is consumer-focused and based on successful requests. However, you will need to make a payment before using the service.
- Offer free trial options for new users to try out the service before making a monetary commitment.
- Make use of a mix of datacenter and residential proxies for clients depending on the package they subscribe to.
- Supports millions of websites across different industries, including all of the popular websites on the Internet.
- Accessible customer support and easy-to-understand documentation.
back to menu ↑
How to Use Crawlbase?
At first glance, you might look at the service as a complicated one considering the fact that it does many things and assert itself as an all-in-one platform for web scraping and crawling at any scale.
However, digging into its documentation and actually making use of it, you will discover that it is one of the easiest to use out there. In this section, you will be shown how to make use of their Crawling API and Scraper API tool. We won’t be discussing how to use their Smart Proxy here as proxy usage is pretty obvious.
How to Use Crawlbase Crawling API
The Crawling API works like a regular Restful API service that you send HTTP web requests, and you get a response. No need to set proxies, handling browsers, bypassing Captchas, and or even worrying about IP blocks. All API calls start with api.crawlbase.com as the base URL. Below is an example of how to send a simple API call for downloading the homepage of Wikipedia.
There is more you can do, but that would include adding parameters. You can learn more about how to use the Crawlbase Crawling API from its documentation.
How to Use ProxyCrawl Scraper API
This one is almost the same as the Crawling API and even uses the https:// api.crawlbase.com as the URL but has a different endpoint https:// api.crawlbase.com/scraper. Let take a look at the simplest Scraper API call. Some sort of a ‘Hello World’ request, as in the case of Crawling API.
curl 'https://api.Crawlbase.com/scraper? token=USER_TOKEN&url=https%3A%2F%2Fwww.amazon.com%2Fdp%2FB00JITDVD2'
For the Scraper API, not all URLs can be used as only selected web services are available. Scraper API has support for some programming languages, including Node, Ruby, Python, PHP, and Go. You can learn more about the features and usage of the Scraper API in its documentation here.
The Crawlbase platform is not a free tool, and as such, you need to prove you have an active subscription by authentication. The only form of authentication is the API key authentication known as USER_TOKEN on the Crawlbase platform.
The Proxcrawl service has support for selecting proxies from specific locations, but sadly, it is not available to all users. For you to make use of their geo-targeting option, then you will need to buy specific packages and plans.
back to menu ↑
Are Crawlbase IP Residential IPs or Not?
There is no doubt that residential IPs are better when compared to datacenter IPs as they are used by Internet users in residential areas and assigned by Internet Service Providers (ISPs). Datacenter IPs are hosting IPs and associated with spam.
If you are looking to make use of residential IPs, then you need to know that Crawlbase makes use of a mix of residential and datacenter IPs in a bid to cut costs. For small packages, including the starter and advanced, only datacenter proxies will be used. To be on the safer side, it is important to speed to their customer agent and tell them you only need residential IPs, but that would be only if you are ready to pay more.
Looking at the above, you can tell that there are some of the services of Crawlbase we would recommend, and there are some we wouldn’t. for their Scraper API and Crawling API, they are quite effective, only get you to pay for what you need, and make your work easier as a web scraper.
For these two (Crawling and Scraper API), we recommend them even though they have room for improvement. But for their Smart Proxy, we do not recommend them at all, except you are a premium user that can pay to get things done your way.
This is because you will not even be accepted to use the Smart Proxy service without your email address being a business domain email. The proxy pool is also small compared to other providers with better pricing.
ProxyCrawl Review on Residential P2P Proxy Networks.