Virtualization vs Private Cloud

I originally wrote this in answer to a question posted on the LinkedIn CloudStack Users Group, but my verbosity pushed the reply way over the allowable length in the LinkedIn comment form field. Plus, with all the hype around cloud these days, there needs to be more real-world examples of where cloud fits and where it doesn't.

The question was this (with minor edits for spelling and grammar):

Private cloud vs virtualization

I searched the net for the difference between private cloud and virtualization. I found some articles, but none of them could illustrate this difference by a real example, they gave just the concepts: self service, ubiquitous, resource pooling, …

My question is: what is the add value of private cloud, especially with cloudstack, for an enterprise that already has a virtualized datacenter, especially with vmware?

Virtualization is a component of cloud, whether public or private. To better illustrate the difference, let's consider a couple of use cases and examples:

Example 1.
An IT department, continually installing/reinstalling new servers, implements a virtualization solution so they can provision infrastructure faster and consolidate servers. They virtualize their servers using their hypervisor of choice along with management tools. They upload iso files into their management software so they can install new OSes into a new virtual machine. They have to be on the local network in order to manage the virtual machines or orchestration software. And if they are charging back capacity to their internal customers' budgets (Marketing, Sales, Engineering, etc), they're probably just splitting the cost between each group, or maybe evaluating how many virtual machines they stand up for each group.

Is this cloud? Not really. This is really just server consolidation, data center automation, etc. Their solution doesn't really meet all five characteristics of cloud computing:

  1. On-demand self-service (they still have to provision virtual machines for their internal customers).
  2. Broad, network access (this is for internal customers on the network)
  3. Resource pooling (this is where virtualization fits, so yes)
  4. Rapid elasticity (they still have to provision VMs, and they don't necessarily scale fast)
  5. Measured service (they're charging back to their users based on traditional budgeting, not based on actual usage)

Example 2.
A company has a headquarters office with a central IT staff that supports company-wide and departmental applications. They also have several branch offices with local IT staffs that focuses on break/fix repair of local desktops and network services. The branch offices may occasionally set up a local server and install some software at a manager's request, but they usually request central IT to provide supported servers or applications at HQ. Central IT is looking to provide better support for their branch offices without having to hire more staff, provide faster turnaround time when provisioning services for supported apps, and even allowing quick, easy servers on-demand to their branch offices for local, unsupported applications. So they install their hypervisor of choice, deploy storage in their preferred manner, and add some management software. However, in additional to providing ISO files for VM installation, they also prepare some disk images with pre-installed, supported OSes. Additionally, the management software provides for multiple users of different access levels to perform tasks such as launching virtual machines. Now, the Marketing department in a branch office can try out some new analytics software by logging into a portal, provisioning a new server, installing the trial software, and using it for a few days. If they don't like it, they turn it off and delete the VM. Engineering may deploy multiple VMs to set up a production application, but also spin up a few additional VMs to use as development and staging environments. They no longer have to put in capital requests for servers, nor do they have to search around some old supply closets and pull out some old dusty desktop system to use as a local staging server.

Is this a private cloud? Yes!

This company still using virtualization, but now they've added a level of self-service for branch offices (whether it's local IT using it or someone else) to consume services without necessarily requesting the limited resources of central IT. They can access this service from their branch office, possibly using a VPN connection over the Internet or an SSL/TLS web-based portal (broad network access). They can spin up additional capacity quickly and turn it off just as fast (rapid elasticity). And because of all this, central IT can now meter actual usage of each service by various departments on a monthly or even hourly basis and charge departments accordingly.

In Example 1 above, the IT department might deploy VMware ESXi and use its built-in management tools, or perhaps deploy Red Hat with KVM and virt-manager. Those are tools available to the server administrators to manage the virtual machines themselves. In Example 2, CloudStack would sit on top of those hypervisors and management tools to provide everything beyond basic virtualization: user self-service portals, prebuilt disk images, usage metering, access from branch offices, etc.

That's the difference between virtualization and private clouds.

Bandwidth, Transfer, and Arithmetic

Every once in a while, I need to explain bandwidth throughput and data transfer to someone. Many web hosts use transfer (GB per month) to measure server bandwidth usage. Other hosting companies, especially those offering colocation services, will tend to offer “95th percentile”, which is a measure of throughput (Mbits per second).

Transfer is the total amount of data sent (either incoming, outgoing, or both) in a given month. Throughput is the amount you're sending through your network pipe on a regular basis, usually by taking a sample every five minutes and averaging it over the course of the month.

Or to put it another way, transfer is when the water company bills you at the end of the month for how much you used (43,000 gallons of water between June 1st and June 30th). Throughput would be more like averaging out how much your pipe is pushing through (1 gallon per minute).

Ultimately, both measurements are the amount of data sent over time, so we can get a rough estimation of how much transfer (GB per month) can be transferred over a single committed connection speed.

Assumption:
Our hypothetical server is pushing data at a constant data rate of 1 Mbit per second, with no bursting (ie, we're not using 95th percentile measurement).

Starting with the following information:

8 bits equal 1 byte
2,592,000 million seconds equals 1 month

Step 1. Convert Megabits to Megabytes

Divide 1 Mbit by 8 to get Megabytes
1/8 MByte per second = 0.125 MB per second

Step 2. Change Seconds to a fractional Month (1 month = 30 days * 24 hours * 60 minutes * 60 seconds = 2,592,000 million seconds)

We now have:
1 Mbit per second =
0.125 MB per 1/2592000 month
This is just restating the measurements.

Step 3. To change the values to a full month, multiply *both* sides by 2,592,000 (we want to change the values, rather than the measurements, so we need to operate on both sides of the equation).

(0.125MB * 2,592,000) per (1/259200 * 2592000) =
324,000 MB per 1 Month

Step 3. Change MB to GB, divide by 1024 on the left:
324,000 MB / 1024 = 316GB

We now have 316GB per month of data transferred if someone sustains a constant throughput of 1Mbit/sec.

Rounding down to a nice 300GB per month, we can figure that a 10Mbit connection speed for your server will yield a maximum of 3000GB (~3TB) of transfer per month (300GB/month X 10Mbits/sec, or 300×10). A 100 Mbit uplink can yield up to 30,000GB (~30TB) of transfer if sustained at full speed.

Bursting patterns and 95th percentile measurements (where the service provider cuts out the top 5% of bursting traffic and averages only on the remaining 95%) will cause this to fluctuate a bit. After all, if you have a 100 Mbit port speed, you can burst up to a 100Mbits of throughput for a period of time, then come back down to normal. So you may send a 300MB file in about 25 seconds if you're sending at a full 100 Mbits / sec (100Mbits / 8 bits = 12.5MB/sec; 300MB / 12.5 MB/sec = 25 seconds). But you won't likely sustain that 100 Mbits after the file is sent. You'll drop back down to a slower throughput until he next large amount of data needs to be sent.

Additionally, that same 300MB file will take about 240 seconds to send at 10 Mbits/sec, and with a 1 Gbit uplink, you could potentially send it in as little as 3 second (traffic speed between routers, datacenters, and server disk read/write and processor speeds not included).

Client IP with X-Forwarded-For across multiple proxies

When you're running HA-Proxy or Nginx in front of Apache, you lose client IP address information. The TCP connection to your Apache server come from Nginx, so all your logs reflect one single client IP address. This is a problem if you run a forum, as it will calculate the number of site visitors by IP address. Worse, this may flag the flood controls on some forum software. At minimum, it throws off log analysis.

X-Forwarded-For is an HTTP header that allows allows Layer 7 (HTTP) Proxies to pass along the original, external client IP to the next destination. To use this, your reverse proxy, caching server, or load-balancer must be configure to add that header to HTTP requests, and the destination point must be configured to look for it.

Everyone has a tutorial for configuring X-Fowarded-For across two servers (say, Nginx+Apache, or HAProxy+Apache). But what happens when you have an HAProxy load balancer balancing between three Nginx caches, which forwards to Apache for PHP/MySQL? That client IP address needs to passed across three different HTTP servers.


Assuming that HAProxy is has address 1.2.3.4, Nginx is running on another server with 1.2.3.5, and Apache is at 1.2.3.6:

HAProxy will need to be configured with this option to pass the X-Forwarded-For header of the connecting IP.
option forwardfor


Nginx will need the following to both *receive* the X-Forwarded-For header from the HAProxy server, and then *add* the X-Fowarded-For header to the new connection to Apache.

In your main Nginx.conf file:

set_real_ip_from 1.2.3.4; # this is the HAProxy connecting IP address
real_ip_header X-Forwarded-for; # The specific header to be read

In your proxy configuration, either in nginx.conf or in a separate include file, you'll need something similar to this:

location / {
proxy_pass http://1.2.3.6;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}


Finally, Apache can use the Rpaf module to read the X-Forwarded-For header from Nginx:


RPAFproxy_ips 1.2.3.5 # The Nginx connecting IP address
RPAFheader X-Forwarded-For # Apache is looking for this header to use as the "client IP"

References:
http://haproxy.1wt.eu/download/1.5/doc/configuration.txt
http://stderr.net/apache/rpaf/
http://wiki.nginx.org/HttpRealIpModule#set_real_ip_from
http://wiki.nginx.org/HttpProxyModule#Variables

Rsync to Fat32 drives

I regularly provide one of my clients with a backup of his data in the form of an external hard drive. Since his server runs CentOS Linux and his computer is a Windows machine, I need to provide a drive formatted Fat32 so he can plug it into his computer and access the data without problem.

# mount -t vfat /dev/sdc1 /mnt/usb -o shortname=mixed -o utf8

The “shortname=mixed” keeps the case preserved, as otherwise vfat will convert any filename that's 8 characters or less to lower case (default behavior is “shortname=lowercase”) and cause problems for rsync. UTF8 is what Windows uses when mounting filesystems, so we specify that to ensure that we're mounting it the same way (default is to mount iso-8859-1, even though the underlying vfat filesystem will store filenames in UTF8 format).

My normal mirror command, “rsync -az /home /mnt/usb”, doesn't work because -a is a shortcut for the following options:


-a, --archive archive mode; same as -rlptgoD (no -H)
-r, --recursive recurse into directories
-l, --links copy symlinks as symlinks
-p, --perms preserve permissions
-t, --times preserve times
-o, --owner preserve owner (super-user only)
-D same as --devices --specials
-g, --group preserve group

Using -o will cause errors, as rsync will copy the file and then chown (change owner) the file. Fat32 doesn't support Unix owership or permissions, so rsycn will error on every file that is copied. Ditto for -p and -g. Symlinks aren't supported either, and we don't want -L to copy the destination file of the symlink (that will produce multiple copies of a file/directory, not desirable in this particular instance). The -D option is irrelevant because we are only copying website data, so we don't need special devices (/dev/*).

That leaves -r (recursive), -t (preserve times) for our vfat options. There's no need to use compression (-z) since we're not syncing across the network.

So the best command to copy form ext3 to Fat32 drive is something like this:


rsync -rtv /home /mnt/usb

I like using -v for verbosity, unless I'm running this within a shell script.

A good reference for further reading on Fat32 with Linux:
http://www.osnews.com/story/9681/The_vfat_file_system_and_Linux/

Repurposing Pallets

Years ago, I discovered the benefits of reusing pallets when I managed a [now-defunct] I Sold It franchise (I Sold It was a chain of eBay drop-off stores). We had a large item to ship that required a pallet for freight shipping, and I didn't want to spend $100 on pallets for a single use ($20 per pallet, quantities of 5 or more). A short search around the rear of our building yielded a couple of pallets, one of which was usable.

When I realized how many pallets are discarded every day around the Atlanta area, I initially thought it might be a good side business to gather up discarded pallets and sell them at a discount. Unfortunately, the economics of dumpster diving for pallets was not very profitable: a business owner is not likely to pay regular price for a used pallet, and to sell on any large scale involved a large commitment of time and resources – hours of driving and gasoline for the car to find and gather them. Though I had space to store them, gathering them up until I had enough to sell wasn't really an option. So my fantasy of a recycling pallet business dissolved as soon as I thought of it.


Adirondack Chair: Photo blatantly copied from Shelton's Flickr page, but used in a fair-use reporting context. If you click on the image to view all his photos, maybe he won't send me a DMCA takedown notice.Adirondack Chair: Photo blatantly copied from Shelton's Flickr page, but used in a fair-use reporting context. If you click on the image to view all his photos, maybe he won't send me a DMCA takedown notice.I had a pleasant surprise at Indie Craft Experience this past weekend. While browsing the usual selection of vendors selling knitted hats, recycled felt skirts, and other crafty stuff, I came across a guy sitting in a wooden chair that looked suspiciously like it was made from pallets.

I have a tendency to stop at any booth featuring wooden furniture items (especially at art fairs, where handmade woodworkers display beautiful and amazing furniture). So there was no hesitation to stop and look at this interesting piece of furniture.

Shelton Davis (the guy sitting in the chair) was there representing Repurposed Goods, a project to reuse discarded pallets into new creations. The chair was made completely from recycled pallets, and he also had on display a birdfeeder. He was selling complete DIY building plans for the chair and bird feeder, as well as a DIY “Ikea-style” kit for the bird feeder that one could build immediately.

All the plans were printed on recycled paper, and he also sells the plans as downloadable PDFs through his Etsy store. It turns out Shelton is an industrial designer, so not only is he able to create plans for pallet repurposing, he is able to convey the instructions in a simple and clear document.

I have his birdhouse instructions, and plan to purchase his Adirondack chair from his Etsy store once I have some time to devote to a bigger project.

I think I have found a reason to start stacking up pallets again.

Bad Proxies Causing Apache to Reach MaxClients

Recently, I was called to assist with a server that was constantly getting bombarded with HTTP connections and causing Apache to hit MaxClients. It took a couple of minutes to track the IPs with the most connections using this little command:


netstat -tpnC | grep httpd | awk '{print $5}' | cut -f1 -d: | sort | uniq -c
1 65.55.25.149
4 67.141.163.123
1 67.195.111.40
1 71.247.16.159
28 137.244.215.55

This showed a large number of connections from an IP 137.244.215.55.
Since we have ExtendedStatus enabled in httpd.conf, it's a simple matter to find what site is getting hit:


# lynx -dump -width=160 http://localhost/server-status | grep -e '...[1-9].*' | grep -v OPTIONS

Sure enough, there's the IP and the site getting hit, along with the request URI. I like to use this to see when a comment spammer is POSTing to WordPress blog or otherwise trying something malicious. The site getting hit was a Fantasty Football site.

However, I was a little concerned because the IP that had a large number of connections was a .mil IP address.


# host 137.244.215.55
55.215.244.137.in-addr.arpa domain name pointer uhhz-wpa-001.robins.af.mil.

So I simply used ConfigServer Firewall (a nice front-end to iptables) to block the address for an hour.


csf -td 137.244.215.55 3600


9:45:00 AM Other Tech: block the .mil!
9:45:45 AM Me: blocked 'em.
9:45:49 AM Me: just for an hour though.
9:46:39 AM Me: in case it's the airforce cybercommand thingy that's investigating a terror suspect and enemy combatant.
9:47:05 AM Me: because a fantasy sports site is where all the terrorists hang out....

All was well and good, but shortly after blocking it another .mil address hitting the same site with a large number of connections:


netstat -tpnC | grep httpd | awk '{print $5}' | cut -f1 -d: | sort | uniq -c
1 124.115.4.197
1 186.18.102.180
1 38.99.98.113
1 65.55.25.144
1 68.216.159.34
1 68.240.147.150
32 131.51.128.21

After blocking that one, a wellsfargo.com address showed up, then a few other random corporate addresses. This beginning to concern me, since this resembles some sort of DDOS behavior from a bunch of infected PCs (and the idea of .mil and Wells Fargo computers being infected didn't sit well with me). But it wasn't a very effective attack, since only one or two IPs would hit at the same time. And why would someone attack a Fantasy Football site with military and banking computers? Sure there are better better targets than that!

Next step was to watch the logs for a bit (tail -f /path/to/domain/access_log). I noticed some odd behavior. Usually, a browser hits a site, requests a page, and then requests all the supporting files (CSS, javascript, images, media files, etc), usually listing the original referring URL along the way. This was a WordPress blog, so most traffic was fairly normal along these lines. But grepping for the specific IP addresses in the log showed a more unusual pattern: a single request from a generic user-agent string (“Mozilla 4.0 (Compatible)”), followed by a large number of requests for all the links on the page. Something like this:


131.50.151.28 - - [16/Oct/2010:11:39:28 -0400] "GET /baseball/wp-content/themes/SomeTheme/style.css HTTP/1.1" 200 404 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=201009 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=201007 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200910 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200912 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=201006 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200908 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200911 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/xmlrpc.php HTTP/1.1" 200 404 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=201010 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200909 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=201008 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200905 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200811 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/xmlrpc.php?rsd HTTP/1.1" 200 408 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200809 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?p=5 HTTP/1.1" 200 408 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200808 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200810 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200904 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200906 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?p=1961 HTTP/1.1" 200 411 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200812 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?p=1989 HTTP/1.1" 200 411 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200907 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"

I watched this for a while, scratching my head. The “Mozilla/4.0 (compatible;)” was suspicious. It was immediately obvious that it was some kind of bot or spider. Bad bots like to disguise themselves or try to pass themselves off as real browsers to avoid detection or redirection based on their behavior. So I was beginning to think this was a really bad indexing/search spider. Except that it's hitting this one single site, and there there were also referrer links in some of the lines indicating traffic from other sites. And bots are usually operated from a single IP address – they don't spring up from other IPs when the first one is blocked.

Troubleshooting is often a team effort, and it certainly helps to discuss a problem and brainstorm ideas.


11:08:53 AM Me: i'm beginning to think that our military is not infected with bots, but are goofing off with fantasy football, which scares me even more.
11:10:31 AM Other Tech: The useragent is weird though
11:11:23 AM Me: yeah. maybe they're behind a proxy server that grabs all the linked pages for immediate caching.

Sure enough, a quick Google for “Mozilla/4.0 (compatible;)” yielded some hits of exactly that. Behind the corporate doors of Wells Fargo and various Air Force bases are a bunch of people reading up on Fantasy Football, and their collective proxy servers (probably Blue Coat) are slamming the server with a ton of requests to pre-fetch all other linked URLs from the first page, so that each visitor is hitting the server with dozens of connections.

Since this is a shared server, this is affecting not only the customer site in question, but all other sites hosted on this server. It's obvious we can't block by IP address, and there are too many variations to block entire ranges (which is a bad idea to begin with).

My solution was to redirect on the User-Agent string until the issue died down. I created a single HTML page on the server's main DocumentRoot (outside the customer's virtual host) and added the following lines to the customer's .htaccess file:


RewriteCond %{HTTP_USER_AGENT} ^Mozilla\/4\.0\ \(compatible;\)$
RewriteRule .* http://host.example.com/denied/ [R]

The actual index.html page in the Redirected URL sums it up like this:


Your "web accelerator" proxy is causing problems with our servers and customer sites.
Sorry, but you will not be able to access content here.
Please contact your IT Support department for assistance.


Addendum:
I found the script on this page very helpful when testing the User-Agent string in my .htaccess rule. While I am quick to telnet to a webserver to pass an HTTP request a simulate a browser visit, I don't know all the details of the HTTP protocol (including the format of User-Agent string). Scripting this to quickly connect to localhost, pass the request and the User-Agent and see if I received a 200 or 302 Redirect was extremely helpful.

A Lesson in Storage Costs

As the [former] Customer Service Manager at A Small Orange, comments, questions, and even complaints were frequently brought to my attention.

One common question is “why do you offer so little disk space compared to your competitors?” Or to phrase it another way, “why are your shared hosting plans so stingy?”

This is an understandable question. From the customer’s viewpoint, they can (as of 2010) buy a 1TB disk drive from Newegg.com anywhere from $59 to $89. “Drives are cheap!”, they cry. “Add more drives! I’ve got 3TB in my desktop PC, why can’t you do it on my server?”

Servers don’t quite operate this way.

For one thing, the hard drive in a server is expected to be reading and writing data almost constantly 24×7, instead of spinning up occasionally for downloads, torrents and gaming. That constant disk reading/writing will shorten the usable life of the drive. When a server’s hard drive dies, it’s not just one customer’s data that is lost…it can be several hundred customers’ data. So a server needs some failover, or redundancy. For a low-cost shared server, that might mean two drives in a mirrored (RAID-1) configuration so that one drive fails, the other one keeps going with a copy of the data until the first one can be replaced. A higher-end server might even have four or six drives, with some combination of mirroring or data protection (RAID 0+1 or RAID 5, for example) so that one or more drives can fail without the losing valuable customer data.

With this redundancy alone, we’ve gone from doubling the cost (two $90 drives for $180) to sextupling the cost (six drives at $534).

Now consider the performance of the drive. The cheapest $59 drive on Newegg would be be a “green” drive. It is a low power, low speed (5400RPM) drive, and has a very low cost per-Gigabyte. I have a couple of these drives in my home storage box and they run just fine. But I don’t access my data at home in the same way a web server will. There is an $89 drive on Newegg that is a little bit more powerful – mostly faster (7200RPM). It’s great for a PC desktop or home storage unit, and maybe even a low-traffic server.

But when you have constant usage, that drive needs to access server data fast and go to the next request because the slower the hard drive is, the fewer browser hits those websites can handle. The standard SATA drive has a 7200RPM rotation speed, and that’s pretty good for desktop usage and low-end servers. For higher performance, SCSI and the newer SAS drives are still the best. A 300GB SAS drive currently runs about $279 and they’re *fast*. They spin at 10,000-15,000 RPMs, and the SCSI protocol they use offers superior performance in server environments.

So now, instead of $180 for 1TB of storage (two SATA drives, mirrored), we now have higher performance drives giving us 300GB of mirrored storage for $558. That’s 300GB of fail-safe, high-performance server storage for $558 versus a single $89 1TB drive in your PC. Three times the cost, but twice the speed and performance.

For even better performance, using a hardware RAID controller instead of software RAID is important. This hardware is at least $200, so now you get 300GB for $758 or more.

Apples and Oranges indeed!

This should provide some perspective on the differences between consumer and business server storage. The next logical question is, how do other hosting companies provide so much more storage, like 50GB or more per customer?

First, many hosting companies use a control panel software (such as cPanel) on their servers. This software is used on a single server, with a limited number of hard drives (usually 2, 4, or 6) and turns it into a profit-machine. You have a set number of customers per server, and when you run out of space, you buy a new server with a limited number of drives and start filling it up with customers.

Using A Small Orange’s Medium hosting plan as an example, a single 1TB drive would allow up to 600+ customers to (1000GB divided by 1.5GB of space per customer). Of course, A Small Orange is not using single desktop hard drives, but mirrored high-performance drives. The drives actually are 146GB SAS drives, so about 97 customers could share that space if they all had the Medium plan.

The more customers a hosting company can stuff onto a single server, the more money you make. So many hosting companies use a technique called overselling. This means the hosting company will offer more space per customer (say, 10GB or 50GB or even unlimited storage) knowing that the most customers will never use it all. If every customer on that server filled up their 10GB or 50GB of space, the server drive space will fill up very quickly and cause problems. To prevent this from happening, there are clauses in the Terms and Conditions that define other limits, usually number of inodes. This prevents customers from filling up the disk with a large number of small files. Additionally, there are usually “fair usage” clauses with “unlimited storage” plans to allow the hosting company to suspend service if the customer is using more than a fair share of the overall disk amount.

With the price of Storage Area Networks and Network Attached Storage devices becoming more accessible to smaller operations, some hosting companies are moving away from fixed storage like physical disks. By implementing a high-performance network storage, a given server can increase it’s storage capacity on demand to meet the increasing storage demands of its customers. As this becomes more common, larger storage quotas (*without* overselling) will eventually become more commonplace.

Subversion dependencies on cPanel servers using Yum

If you're ever had to manage a cPanel server, you've probably had a request from a customer to install Subversion, and you probably tried the Yum package manager to install it.


root@server [~]# yum install subversion
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* addons: mirror.cogentco.com
* extras: yum.singlehop.com
Excluding Packages in global exclude list
Finished
Setting up Install Process
Resolving Dependencies
[...]
subversion-1.4.2-4.el5_3.1.x86_64 from base has depsolving problems
--> Missing Dependency: perl(URI) >= 1.17 is needed by package subversion-1.4.2-4.el5_3.1.x86_64 (base)

cPanel is written in Perl, so the cPanel software maintains it's own Perl installation and blocks Yum from installing Perl deps. The cPanel /scripts/perlinstaller utility will show that the module needed is installed, yet Yum will not proceed.

I used to download the per-uri RPM from rpmfind.net for my version of CentOS and install it. But an associate of mine showed me an easier way.

Edit /etc/yum.conf.
The very first line:
exclude=apache* bind-chroot courier* dovecot* exim* httpd* mod_ssl* mysql* nsd* perl* php* proftpd* pure-ftpd* ruby* spamassassin* squirrelmail*

Remove “perl*” from that line. Save and quit.
Now, install subversion. It should resolve the perl-URI dependency just fine.
Once it's completed, don't forget to edit the yum.conf file and restore the “perl*” to the exclude line, so Yum doesn't interfere with cPanel's Perl modules.

Yum Transaction Check Errors

A client tried to update his CentOS installation on his virtual server by running “yum update” and got the following error:

Transaction Check Error:
file /usr/share/X11/XKeysymDB from install of libX11-1.0.3-11.el5 conflicts with file from package libX11-1.0.3-9.el5
file /usr/include/popt.h from install of popt-1.10.2.3-18.el5 conflicts with file from package popt-1.10.2-48.el5

The quick and easy fix is to remove the “installed” package that presents the conflicting files. Then let yum reinstall with appropriate versions and dependencies.


[root@vps ~]# rpm -e libX11-1.0.3-11.el5
[root@vps ~]# rpm -e popt-1.10.2.3-18.el5
[root@vps ~]# yum -y update
...
[root@vps ~]# cat /etc/redhat-release
CentOS release 5.4 (Final)

cPanel Account copy: Package over quota

Many hosting providers use PHP built as an Apache module, so that when a hosting customer uploads files through a script, they are owned by the user “nobody”. This causes a major problem with quotas, since cPanel will only count files that are owned by the user. Files owned by nobody are not counted toward quotas. If a user installs an uploader script, then the user can extend their actual disk usage far beyond their quota.

This can be a problem when moving accounts from one server to another, if the account package is larger than the allowed quota, cPanel will not restore all the files to the new server. This usually results in accounts being restored but with incomplete file sets. Even worse, cPanel will report that the account was restored successfully:




/bin/gtar: ./.fantasticodata/PerlDesk: Cannot mkdir: No such file or directory
/bin/gtar: ./.contactemail: Cannot write: Disk quota exceeded
/bin/gtar: ./fantastico_backups/blog.backup.1137314599.tgz: Cannot open: No such file or directory
/bin/gtar: Error exit delayed from previous errors
Done


Account Restore Complete
Unlocking password for user,
passwd: Success.
checked 107 files….

The simple workaround is to increase the customer's quota in WHM before packaging the account. This way, the account has a large enough quota to restore all the files.

If the account is already packaged and access to the origin server is no longer available, then the only option is to edit the quota file stored in the cPanel package. This will allow the new server to restore the file with a larger quota limit, thus ensuring all files are restored. The following real-world example shows the backup package is copied to the destination server in the /home directory, ready to be restored. In addition, there are approx 1.5GB of files in the account, but the quota is set to 700MB.


root@server [/home]# gunzip USER.tar.gz
root@server [/home]# tar -f USER.tar -x USER/quota
root@server [/home]# cat USER/quota
700
root@server [/home]# echo 2000 > USER/quota
root@server [/home]# tar -rf USER.tar USER/quota
root@server [/home]# rm -rf USER/
root@server [/home]# gzip USER.tar
root@server [/home]# /scripts/restorepkg USER

Of course, you can solve this issue in advance by enabling PHP to run under suPHP or FastCGI, so that scripts will create files owned by the cPanel user, rather than nobody. Then, the user will be alerted when they hit quota.