# How to Use C++ Compiled Python (Shed Skin) for Amazon's Elastic MapReduce

- Canonical URL: https://amund.blog/archive-atbrox-cpp-compiled-python-emr/
- Author: Amund Tveit (https://amund.blog/about/)
- Published: 2009-10-07
- Tags: Compilers, Distributed Systems, Archive
- Original Source: atbrox.com (http://atbrox.com/2009/10/07/how-to-use-c-compiled-python-for-amazons-elastic-mapreduce-hadoop/)

---

Sometimes it can be useful to compile Python code for Amazon’s [Elastic Mapreduce](<http://aws.amazon.com/elasticmapreduce/>) into C++ and then into a binary. The motivation for that could be to integrate with (existing) C or C++ code, or increase performance for CPU-intensive mapper or reducer methods. Here follows a description how to do that:

  1. Start a [small EC2 node](<http://aws.amazon.com/ec2/#instance>) with AMI similar to the one [Elastic Mapreduce is using](<http://docs.amazonwebservices.com/ElasticMapReduce/2009-03-31/DeveloperGuide/index.html?introduction.html>) ([Debian Lenny Linux](<http://www.debian.org/releases/stable/>))
     * note: [We](<http://atbrox.com/about/>) used [Alestic](<http://alestic.com/>)’s [ami-ff46a796](<http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1615&categoryID=101>)
  2. Skim quickly through the [Shedskin tutorial](<http://shedskin.googlecode.com/files/shedskin-tutorial-0.2.html>)
  3. Log into the EC2 node and install the [Shedskin Python compiler](<http://code.google.com/p/shedskin/>)
     * Shedskin requires a few libraries: 1) [the Boehm-Demers-Weiser garbage collector for C++](<http://www.hpl.hp.com/personal/Hans_Boehm/gc/>), 2) [PCRE – Perl Compatible Regular Expressions](<http://www.pcre.org/>). The [Shedskin tutorial](<http://shedskin.googlecode.com/files/shedskin-tutorial-0.2.html>) for detailed install instructions. 
     * note: The Alestic Debian AMI is fairly slim, so we had to add some more software make Shedskin work, i.e. GDB 
  4. Write your Python mapper or reducer program and compile it into C++ with Shedskin
     * E.g. the command _python ss.py mapper.py_ – would generate C++ files _mapper.hpp_ and _mapper.cpp_ , a _Makefile_ and an annotated Python file _mapper.ss.py_. 
  5. Optionally update the C++ code generated by Shedskin to use other C or C++ libraries
     * note: with [Fortran-to-C](<http://en.wikipedia.org/wiki/F2c>) you can probably integrate your Python code with existing Fortran code (e.g. numerical/high performance computing libraries). Similar for Cobol (e.g. in financial industry) with [OpenCobol](<http://www.opencobol.org/>) (compiling Cobol into C). Please [let us know](<http://atbrox.com/about/>) if you try or need help with help that. 
  6. Add -static as the first CCFLAGS parameter in the generated Makefile to make it a static executable 
  7. Compile the C++ code into a binary with make and check that you don’t get a dynamic executable with ldd (you want a static executable) 
  8. Run [strip](<http://en.wikipedia.org/wiki/Strip_\(Unix\)>) on the binary to make it smaller 
  9. Upload your (ready) binary to a chosen location in Amazon S3 
     * e.g. via commandline with [S3CMD](<http://s3tools.org/s3cmd>), with a UI using [S3Fox](<http://s3fox.net/>) or [Cloudberry S3 Explorer](<http://cloudberrylab.com/?id=7>) or programmatically with [Boto](<http://code.google.com/p/boto/>). 
  10. Read [Elastic Mapreduce Documentation](<http://docs.amazonwebservices.com/ElasticMapReduce/latest/GettingStartedGuide/>) on how to use the binary to run Elastic Mapreduce jobs. 
     * note: [Peter Skomoroch](<http://twitter.com/peteskomoroch>) has written a [good tutorial for Elastic Mapreduce](<http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2294>)

Note: if you skip the shedskin-related steps this approach would also work if you are looking for how to use C or C++ mappers or reducers with Elastic Mapreduce.

Note: this approach should probably work also with [Cloudera’s distribution for Hadoop](<http://www.cloudera.com/blog/2009/09/10/cdh2-clouderas-distribution-for-hadoop-2/>).
