Atbrox is a startup providing technology and services for Search and MapReduce/Hadoop, founded by alumni from Google, IBM, and research.

The TOP500 list is for supercomputers what the Fortune 500 is for companies. About 80% of the list consists of supercomputers built by Hewlett Packard or IBM, alongside Dell, Sun (Oracle), Cray, and SGI. The Parallel LINPACK benchmark determines ranking positions (while the derived Green500 also accounts for power efficiency).

Trends Towards Cloud Supercomputing

To our knowledge, the entire TOP500 list is currently based on physical supercomputer installations rather than cloud computing configurations (virtual clusters lasting long enough to run LINPACK), though that will likely change within a few years. Already, clear trends toward cloud-based supercomputing are emerging — particularly in consumer internet services and pharmaceutical computation.

Documented Cloud Deployments

  1. Zynga (FarmVille, Mafia Wars) — Uses 12,000 Amazon EC2 nodes (ref: Manager of Cloud Operations at Zynga).
  2. Animoto (online video production) — Scaled from 40 to 4,000 EC2 nodes in 3 days (ref: CTO, Animoto).
  3. Myspace (social network) — Simulated 1 million simultaneous users using 800 large EC2 nodes (3,200 cores) (ref: High Scalability).
  4. The New York Times — Used hundreds of EC2 nodes to process their historical archives in 36 hours (ref: The New York Times Archives + AWS = TimesMachine).
  5. Reddit (news service) — Runs on 218 EC2 nodes (ref: I run reddit's servers).

Back-of-the-Envelope EC2 Estimates

  1. Justin.tv (video service) — In October 2009, Justin.tv users watched 50 million hours of video at a reported cost of roughly 1¢ per user-video-hour (~$500k/month, or ~$6M/year). Assuming half of costs were compute at $0.085/node-hour, that would correspond to roughly $3M / (24 × 365 × $0.085) ≈ 4,029 EC2 nodes 24×7 — or roughly ~2,000 EC2 nodes after accounting for heavy video bandwidth costs.
  2. Newsweek — Reportedly saved up to $500,000 per year by moving to the cloud; assuming half in compute savings, roughly ~670 EC2 nodes equivalent 24×7 (less storage/bandwidth).
  3. Recovery.gov — Reportedly saved up to $420,000 per year moving to EC2, corresponding on the same rough basis to ~560 EC2 nodes equivalent 24×7.

Biotech & Pharma in the Cloud

  • Eli Lilly, Johnson & Johnson, and Genentech — Offloading computational chemistry and biotech HPC workloads to the cloud (ref: HPCwire, C&EN).
  • Pathwork Diagnostics — Using Amazon EC2 for cancer diagnostics (ref: HPCwire: Diagnosing Cancer in the Cloud).