Saturday, April 6, 2013

Uploading to MEGA: the CPU bottleneck in VPS/servers

Many people have reported that they get slow speed uploading to MEGA - both using the browser or by using MegaUploader.

The test machine I use is a "4-years-old" Q6600, 4 cores, with a 5Mbps upload connection. It uploads at maximum speed (600KB/s).
However, I had access to a VPS recently, and I did some tests. Althogh it has a 1Gbps shared connection, it never gets more than 300KB/s per file. It was very strange.

With just one file, the upload speed was 350KB/s, with peaks of CPU of 80% - the VPS has only 1 core. However, uploading 2 files at the same time, it uploaded about 700KB/s, with the CPU at 80% with peaks of 100%! Trying to upload 3 files was a terrible idea. Yeah, I got (a unstable) 1MB/s upload speed, but the CPU was always at 100%, and the VPS was slooooow. I was kicked several times from RDP.

After some tests and traces, I arrived to a conclusion: the CPU is a BIG bottleneck!

Information is uploaded to MEGA in chunks of 1MB (well, technically the firsts chunks are smaller and then they get a size of 1MB).
MegaUploaders creates a thread for each file being uploading. Each thread ciphers a chunk (applies an AES cipher) and the creates a parallel task to upload that chunk. Meanwhile, it starts to cipher the next chunk. Once the next chunk is codified, it waits the upload thread to finish - in this way a queue is avoided, so memory usage is always low. As you can see, there are a lot of threads, and if you have a multi-core processor, it will be great.

With home connections, the bottleneck is the upload process. While the chunk is being uploaded, another thread is ciphering the next chunk. Once it is ciphered, it waits until the upload thread is free. In this way the CPU works, and then waits. That's the reason you see "CPU peaks" if you look at the task manager. Home PCs have normally two or more cores (nowadays is strange to have a CPU with just one core), and the upload speed is terrible, so MegaUploader squeezes the connection: if the CPU is faster ciphering each chunk than uploading it, the bottleneck is your conneciton.

However, with a VPS (a cheap one), we have the opposite case. Normally, the VPS may have just one core, or if you get a dedicated low-end server (like a Kimsurfi) you have an Atom core... with a 100Mbps upload speed - or maybe 1Gbps connection!

This means that the upload process is very fast... but the ciphering process is not. So the CPU doesn't have to wait the upload thread to finish - it is always working, and the upload process has to wait! And with just one core, the more threads you create - aka the more files you upload, the more context changes there are - and the VPS gets slower.

So, to sum up, uploading to MEGA requires not only bandwidth, but a lot of CPU. If you have a great upload connection, you will also need many Ghz and many cores in order to avoid a bottleneck.

If you use a cheap VPS/server to upload, you will experiment slow speed and high CPU consumption.

On my side, I will check the CPU consumption during the ciphering process - most of the cycles are consumed, not for ciphering the data, but for generating the CBC-MAC code, a 16 bytes key that is necessary when uploading the file - if that CBC-MAC is not correct, you get the famous "decryption error" when downloading. Maybe I will be able to improve the performance of that process. I hope it ;)

9 comments:

  1. Hi! Thanks for this amazing app!
    Is there any way to upload an entire folder CREATING this folder in Mega? I mean, not uploading every file in the folder at Mega root folder?
    Thank you!

    ReplyDelete
    Replies
    1. Hi! At this moment it is not possible, I will implement this feature in the future (don't know yet when, I develop it in my free time so I don't have a schedule!)

      Delete
  2. Hi, many thanks for this. I like to know how can i upload faster on my home connection? I have 10Mbps upload connection but the upload speed is 1/3 of it. A few months ago, my upload can goes up to full speed. My processor is i5 core. How do i fix this problem as the upload speed is such a pain! My download speed has no problem.

    ReplyDelete
    Replies
    1. Hi, do you get full speed when uploading via web?
      I have seen that in many cases the speed is limited by MEGA's servers.

      Delete
    2. Hi Andres, in the past like 2 months ago, i get full spead via web. But now, my speed uploading via web only 1/3 and sometimes uploading stops and restart. My downlaod speed is fast and has no problem. I tried with your program and sometimes its speed is fast however, it starts to get lower as time goes on. Any solution to it?

      Delete
    3. Hi, it seems to be a problem of MEGA servers. In the last days I have also noted that the speed is not good, depending on the server you are uploading - saturation, etc. The servers are balanced directly by MEGA, so you can't do anything :(

      Delete
    4. Thank you for your help. I notice too from other people that their speed is slow too. Maybe too much users affecting it? Im here to thank you for your hard work and please continue to update~ Thank you soooooo much =)

      Delete
    5. Sorry Andres, i been experiencing uploading problem since yesterday with version 0.6. Uploading half way, it stops uploading, it wont start or stop and just jam at the %. Is it a bug or any solutions to solve?

      Delete
    6. Hi!
      I am afraid it's a problem of MEGA, these days many users have reported problems uploading. However the problems are still present if uploading files via web, so it's not a problem of MegaUploader, but MEGA's problem, sorry :(

      Delete