1. Introduction

SciExp²-ExpRun (aka Scientific Experiment Exploration - Experiment Running) provides a framework for easing the workflow of executing experiments that require orchestrating multiple processes in local and/or remote machines.

The contents of the framework are organized into:

• An extension of spur, to run processes on both local and remote machines, with improvements on error handling and cleanup.
• Various helper functions to manage package installation, file copying, process management (pinning, setting limits, etc.), or CPU frequency and interrupt configuration, among others.

2. Installing

The simplest, system-agnostic, way to install SciExp²-ExpRun is using the package from the official Python Package Index (PyPi [1]) using virtualenv and pip [2]. Using virtualenv will ensure you have a controlled local installation of a specific version for each of your projects, making sure your scripts always use the same SciExp²-ExpRun version to avoid any possible future version incompatibilities.

1. Install virtualenv and pip.

2. Create a python package “environment”:

   virtualenv --system-site-packages ~/my-exprun
   

You can create as many as you want; e.g., one for each different version of the SciExp²-ExpDef package that you want to test.

If you want to ignore the python packages installed on the system, you can instead run:

virtualenv ~/my-exprun

3. Install SciExp²-ExpRun:

   . ~/my-exprun/bin/activate
   pip install sciexp2-exprun
   

   Or to install a specific version (version 0.0.0 in the example):

   . ~/my-exprun/bin/activate
   pip install "sciexp2-exprun==0.0.0"
   

[1]	https://pypi.python.org

[2]	http://www.pip-installer.org/en/latest/installing.html

3. Running

If you have installed SciExp²-ExpRun using pip, just remember to activate your environment before using it:

. ~/my-exprun/bin/activate
# programs and paths are now properly set up

3.1. SciExp²-ExpRun and Make

In you use make to automate the execution of your scripts, here is a simple snippet you can add to your Makefile (the example uses version 0.0.0 of SciExp²-ExpRun):

all: deps/exprun
      # run our script using exprun
      ( . deps/exprun/bin/activate && /path/to/my/script.py )

# the ".done" file ensures a partial installation will not count as a success
deps/exprun: deps/exprun/.done
      $(RM) -R $@
      mkdir -p $(dir $@)
      virtualenv --system-site-packages $@
      ( . $@/bin/activate && pip install "sciexp2-exprun==0.0.0" )
      touch $<

3.2. Debugging aids

You can start your scripts with ipython --pdb /path/to/my/script.py to get into a debugging shell whenever an error occurs [3]. Sometimes it is also useful to start an IPython shell [4] somewhere deep in your code to interactively evaluate its current state:

from IPython import embed
embed()

Exiting the debugging shell will returning to normal execution.

[3]	http://ipython.org/ipython-doc/stable/interactive/reference.html#post-mortem-debugging

[4]	http://ipython.org/ipython-doc/stable/interactive/reference.html#embedding-ipython