FAQ

1. How does Sorcerer Work?
2. Price of the Service
3. Supported Cloud Platforms
4. Download Process
5. Do I get charged for Failed Downloads?
6. Can I review downloads?
7. Limits
8. Why can't I download this URL?
9. How do I track my usage?
10. I'm bored, what should I do with my new heightened download powers?

How Does Sorcerer Work?

Sorcerer downloads files so you don't have to. The platform splits files into chunks and downloads them separately to hasten the pace of transfers. The Sorcerer platform allocates the necessary resources to move larger files and it scales to handle demand.

Pricing of the Service

The basic tier allows up to 25GBs to be downloaded and is priced at $15. The pro tier is $50 with the addition of a higher download quota and more features. The advanced tier is priced at $450 with even larger bandwidth quotas. The enterprise tier is priced at $2,000 and is perfect for developers who need less limits. Sorcerer is not responsible for any additional charges from your utilized cloud platforms.

Supported Cloud Platforms

Currently, Sorcerer is able to connect to Amazon S3, Google Cloud Storage, Microsoft Azure Storage, and many more. Some services such as Google Drive have lower maximum file size due to their user limits. Before a connection is made with a storage platform such as S3 or GCS, it is highly suggested that new credentials with limited access are created for safety. Additionally, we are currently in the process of expanding our integrations and any suggestions are welcome.

Download Process

Sorcerer starts a process by verifying if the chosen URL is valid. Next, the requested file is broken into chunks and the download process formally begins. The platform attempts to prevent the process from failing by retrying failed requests and chunks. If a failure happens, the cause is communicated back to the user to give a clear explanation of the error that occurred. Once all chunks are completed downloading, they are stitched together to complete the download.

Are failed downloads added to my usage?

Sorcerer does not add failed downloads to usage however, downloaded data is counted for manually cancelled downloads that are still in progress. To minimize cost for AWS S3 connections, it is highly suggested that a lifecycle rule is applied to the connected S3 Bucket to delete incomplete file data that is hidden in S3's dashboard. The instructions are provided by AWS here.

Reviewing a Download

Sorcerer provides a file size and type before a download is started. For video streams, it returns an estimated file size and available formats. Video streams do not provide a file size therefore Sorcerer has to approximate it based on available information.

Limits

The maximum file size that Sorcerer can download for AWS S3 and GCS is 5TB and 750GB for Google Drive. This is solely due to their maximum upload sizes. Additionally, all integrated storage platforms have request limits that could result in throttling. It is suggested to utilize multiple connections if an absurd burst of file downloads is to be requested to prevent Sorcerer from being throttled or restricted. Futhermore, large file requests should be properly timed apart due to the large number of requests needed to download them.

The Sorcerer API also employs a number of safeguards against bursts of incoming traffic to help maximize its stability. Users who send many requests in quick succession may see error responses that show up as status code 429. This is done primarily through a rate limiter that limits the number of requests received by the API within any given second.

Why can't I download this URL?

There are multiple reasons why a URL may not be supported by the Sorcerer platform.

• The URL does not exist. The file location simply does not exist and the url should be verified.
• The file is too big. The file size is simply too large and does not fit within the cloud storage provider's limits.
• The URL does not return a valid file size. If a size is not provided for the file, Sorcerer will be unable to successfully gauge when it ends. The lack of a provided file size could lead to infinite downloads if Sorcerer attempted to get the file without any reference to its ending.
• The URL requires user authentification. If a valid user login is needed to access the file from the requested site, Sorcerer will be unable to start the download process.
• The URL's server has blocked access or shutdown. If Sorcerer has made too many requests to the server, it could be blocked from accessing or finishing a download.
• The URL's server has has been censored due to legal reasons. If a file has been restricted to due to legal reasons or sanctioned by a government, Sorcerer will not be able to download it.
• The stream is encrypted. If a video stream is encrypted using a DRM, Sorcerer will not be able to download it.

How do I track my usage?

Sorcerer provides a dashboard with all the necessary metrics to keep you up to date on your bandwidth usage. Metrics such as daily bandwidth usage, upload count, success rate, and file types are available to you. If you have any suggestions, email us at support@sorcerer.io.

I'm bored, what should I do with my new heightened download powers?

The sky is the limit. Download files from open directories or connect your S3 buckets to AWS Lambda to process data files that you import.