How To Accelerate Issue Reporting and Replication

This article presents a way to speed-up reporting and replication of issues.

Every time you report an issue you need to include a description of the failing scenario, source code version information, simulation setup and steps required to reproduce the failure, the test and seed to run. The engineer assigned to fix the issue will have to manually setup the environment and follow the required steps. All these take time and are prone to human errors.

A simple way to reduce the effort is to pack the instructions and various steps into an executable and provide that instead of plain text comments. I call such an executable an “executable issue” (EI) and it is similar to an installer. All you have to do is to execute it in a clean environment (e.g. new terminal) and it will “install” for you the issue you want to replicate, it can even start the simulation.

To create an EI few steps are required:

• User provides minimal information: issue name/number, test and seed.
• Create EI’s folder that will contain all generated fragments
• Collect versioning information and save it in your preferred format (e.g. version.txt or labels.txt)
• Collect environment setup information and transform it in an executable script(e.g. setup.sh)
• Create a script that starts the simulation with the specified test and seed(e.g. run.sh)
• Create a README file that contains extra information (creator, date, issue number, issue description, steps to follow in order to start the simulation)
• Create or copy any other resources you think are required or useful
• Create an archive out of EI’s folder
• Create the executable issue installer
• Clean-up the fragments

All these steps are implemented in the template script bellow, which can also be downloaded here:

#!/bin/sh
# variable initialization
issue_name="NA"
test_name="NA"
seed="NA"

# you can easily add your own arguments
while [ $# -gt 0 ]; do
    case `echo $1` in
      -issue)
          shift
          issue_name=$1
      ;;
      -seed)
          shift
          seed=$1
      ;;
      -test)
      	 shift
      	 test_name=$1
      ;;
    esac
    shift
done

# check for mandatory arguments
if [ "${issue_name}" == "NA" || "${test_name}" == "NA" || "$seed" == "NA" ]; then
   echo "Usage: ./create_executable_issue.sh -issue  -test  -seed "
   echo "issue_name: an id, name or 1-word description of the issue" 
   echo "test_name: the name of the test to reproduce"
   echo "seed: the seed"
   exit 1;
fi

# initialize computed variables 
ei_dir=sim_${issue_name}
ei_tarball=sim_${issue_name}.tar
ei_date=`date +%Y_%m_%d`

echo "Create ei folder..."
mkdir ${ei_dir}

echo "Collect versioning information..."
#ClearCase Example: cleartool catcs > ${ei_dir}/issue_config_spec.txt
#GIT Example: git describe --exact-match `git rev-parse HEAD` > ${ei_dir}/issue_config_spec.txt

echo "Include setup information..."
cat > ${ei_dir}/setup.sh << EOF
# include here environment setup commands that have to be run before the start of simulation: 
# - versions related setup
#      ClearCase Example: cleartool setcs issue_config_spec.txt
#      GIT Example: cat issue_config_spec.txt | xargs git checkout 
# - simulator, library, VIP setup
# - other environment setup commands
# this script will be sourced before starting the compilation&simulation
EOF

echo "Include run information..."
cat > ${ei_dir}/run.sh << EOF
# include here compile&run commands for the test&seed arguments
EOF

echo "Create README..."
cat > ${ei_dir}/README << EOF
* Reporter: $USER
* Date: ${ei_date}
* Issue name: ${issue_name}
* Test: ${test_name}
* Seed: ${seed}
# include instructions on how to setup&run a simulation
# you can include other information you think is relevant for the user of the ei
EOF

echo "Create self-extracting issue tarball..."
# this example uses tar archiving, but you could use zip archiving in a similar way
tar cf ${ei_tarball} ${ei_dir}

if [ -e "${ei_tarball}" ]; then
    echo "Compress the tarball..."
    gzip ${ei_tarball}
    if [ -e "${ei_tarball}.gz" ]; then
        echo "Create the executable issue..."
        cat > ${ei_dir}.ei << AOF
#!/bin/bash

# here you can add more commands that will be executed before unpacking
echo "Unpacking Executable Issue...."

export TMPDIR=\`mktemp -d /tmp/some_ei_selfextract.XXXXXX\`
ARCHIVE=\`awk '/^___ARCHIVE_BELOW___/ {print NR + 1; exit 0; }' \$0\`
tail -n+\$ARCHIVE \$0 | tar xzv -C \$TMPDIR

CDIR=\`pwd\`
cp -r \$TMPDIR/${ei_dir} .
rm -rf \$TMPDIR

echo "Unpack completed!"
# here you can add more commands that you would like the executable issue to execute before unpacking

echo "Setup ei..."
cd ${ei_dir}/
chmod +rwx setup.sh run.sh
source setup.sh

# here you can add more commands that will be executed after setup
echo "Setup is complete! You can start the simulation by executing ./run.sh"

exit 0

___ARCHIVE_BELOW___		  
AOF

    echo "Append ${ei_tarball}.gz to ${ei_dir}.ei"	  
    cat ${ei_tarball}.gz >> ${ei_dir}.ei
    chmod +rwx ${ei_dir}.ei
    echo "${ei_dir}.ei created"
    echo "Cleanup fragments..."
    rm -rf ${ei_tarball}.gz
    rm -rf ${ei_dir}
    else
        echo "${ei_tarball}.gz does not exist"
        exit 1
    fi
else
    echo "${ei_tarball} does not exist"
    exit 1
fi

exit 0

This script can be integrated with regression tools, such that the run or regression analysis scripts can prepare EIs for all or some of the failures.

Cheers!