How to Make Uploading 10x Faster
Do you feel like your web application could be responding a bit faster? We had the same feeling so we tried something new…
When we’ve noticed that our app was responding slowly, we tried the usual optimisations. But the effect wasn’t really satisfying. As it turned out, the slowest part of our application was uploading files. But is it even possible to make uploading faster?
How about a brief introduction before we get our hands dirty? If you’re in a web applications business, you’ve probably faced some issues with your app not being responsive enough. The bottleneck is usually caused by one of the following:
- Poorly optimised code
- Servers and clients’ devices running slow
- Bad network connection
The first one, whether it’s a legacy server code, non-optimal database queries or some browser-blocking JavaScript, can usually be tracked and fixed with relative ease (easy as “I know what should be done”, not necessarily “it’s a piece of cake”).
What if your code is already top-notch quality, but your servers begin to have trouble serving all the people who use your application? It actually sounds like quite a good sign for you and your business. You’ll probably end up switching to more powerful servers to handle all the incoming traffic. Or maybe it’s just a high latency issue? Then a Content Delivery Network (CDN) might be what you need.
Client devices being too slow is usually a non-issue in case of most web applications, because almost any device capable of running a web browser should provide a decent experience to the user.
We’re now left with only one possible cause: poor network connection. Thankfully download speeds are getting better, but you can still make your website load faster with assets minification and compression. Any modern web server is capable of serving gzipped files, which allows clients to download less data to see your website (and if you’re not gzipping yet, it’s probably a good time to consider doing so). But what if you want faster responses the other way around?
Welcome to the Dark Grounds of Uploading
What is so dark about it? For starters, if we were to compare download and upload speeds we will find out that a lot of people use asymmetric Internet connection (no source for this, but you’re also one of them, aren’t you?). And the asymmetry means upload speeds up to 10 times slower than download. So, while 60Mbps connection is not bad at all, 6Mbps upload sounds rather slow. This is actually not even 1MB/s.
Moreover, unlike servers, browsers have no native way to compress data before uploading. So here we are, stuck with slower network speeds and bigger files, or… are we?
We came up with a simple idea to solve this problem. While we can’t force people to get a faster connection, we can try to send less data — the same way we do it on servers. But how can we compress data in a browser before uploading?
Let’s Meet Pako
What is pako? According to the description, it’s a zlib port to javascript, very fast! In other words we’ve just equipped browsers with a power of gzipping data — this allows us to turn an idea into a working solution. And using it is almost a no-brainer. For example, before uploading a file, you would compress it with:
var compressed_file = pako.deflate(input_file);That was easy, wasn’t it? But let’s stop for a second and talk about when you should actually do this.
We Can’t Compress Everything
OK, we can but sometimes it just doesn’t make sense. For example, you won’t make images or videos any smaller with gzip (at least most of them — standard formats like jpg or mpeg are already well compressed).
Text data, on the other side, compresses really well. Depending on a file, we may actually expect up to 90% compression ratio. And it’s not only plain text that can be packed that well. In our case it was 3D related files (obj, stl, etc). Maybe you’ve got some Excel spreadsheets or huge Photoshop files waiting to be uploaded? Go ahead, and try to compress them first. You might be surprised by how much space this can save you.
But Compression is Slow
And so is uploading files. Remember how we’ve talked about asymmetric networks? With upload speeds that slow, there should be a point where compression time is considerably shorter than uploading. And it’s not really hard to reach this point. For most well-compressable files, you’re already there. Compressing and uploading smaller file will be faster than uploading uncompressed, bigger data.
But to make sure we don’t spend too long compressing, we will use the lowest compression level (1 out of 9). It’s the fastest one but still reduces file size significantly. In most cases you won’t gain much with higher levels.
As you can see we only get 2–3% better compression ratio with much, much longer compression times. (At this point I should either show you a chart with time/size comparison or, better still, you may check it yourself.)
To change compression level in pako, just pass a level option like this:
var compressed_file = pako.deflate(input_file, {level: 1});Now, that we know how to do this, let’s check the effect of the above actions.
Did We Really Make Uploading Faster?
Well, yes we did.
After testing this method for a while, we’ve decided to use it in our app. In our case, we were uploading some 3D model data described with JSON. Before compressing, they sometimes grew to 5–6MB. For those files we were able to achieve 90% compression ratio, which means uploading just 0.6MB instead of 6MB. Now, with 6Mbps upload speed, client will only wait 0.8s for the upload to complete instead of 8. Compressing these files takes below 0.1s, so we’ve achieved close to 10x speedup.
But you don’t have to believe me. In the end I’m just some random guy from the Internet.
Why Don’t You See for Yourself?
I’ve prepared a small demo app that you can use to see whether or not this might be usable in your case. Just select upload speed and add some files. The demo will take care of compressing it and simulating upload speed. Don’t worry — it’s not really sending your files anywhere, it’s just moving the progress bars so you can compare time required to send uncompressed and compressed data.
What Next?
There are a few more things to keep in mind. The most important is to remember, that your server now receives compressed data, so if you need to work with it, you’ll have to unpack it first. Unpacking is still way faster than packing, so it shouldn’t be an issue — especially considering how much extra time we’ve gained.
Another thing to keep in mind is that compression is not instantaneous. It may take a few seconds for bigger files (for some huge 450MB files, you may need to wait about 30 seconds). Be nice to browsers. To keep your website from freezing, either process files in chunks (pako has a support for this), or try throwing some Web Workers in there.
Well, that’s simply it. What use do you see for this solution in your apps?
If you enjoyed this post, please don’t forget to tap ❤! You can also follow us on Facebook, LinkedIn and Twitter.
