data.html

---
layout: page
title: Data
---

Through our <a href="experiments">experiments</a> we investigated the existence
and impact of <a href="theory">atoms of confusion</a>.  Through reverse
engineering winners of the <a href="http://ioccc.org">International Obfuscated C
Code Contest</a>, we generated a partial list of 19 potential atoms.  In the
atom existence experiment we tested how well subjects could hand trace those
atoms compared to functionally equivalent pieces of code with the obfuscations
removed.  Of the 19 proposed atoms, 15 were statistically confirmed to be
significantly confusing to our subjects.

<style>
td.last {
    width: 1px;
    white-space: nowrap;
}

tr.rejected col.atom {
  text-decoration: line-through;
}

code {
  white-space: pre;
  word-break: normal;
  word-wrap: normal;
}

#atom-table {
    border: none;
    font-size: 16px;
    width: 80%;
    text-align: left;
    border-collapse: collapse;
    margin-left: auto;
    margin-right: auto;
    margin-bottom: 15px;
    margin-top: 45px;
}

#atom-table th {
    border: none;
    font-size: 18px;
    font-weight: bold;
    padding: 12px 15px;
}

#atom-table td {
    border: none;
    border-top: 1px solid #ddd;
    padding: 10px 15px;
}

#atom-table .centered {
    text-align: center;
}

#atom-table tr:hover td {
    color: #339;
    background: #eff2ff;
}

</style>
<table id="atom-table" border=1>
    <colgroup>
       <col id='atom' span="1" style="width: 35%;">
       <col id='example' span="1" style="width: 30%;">
       <col id='effect' span="1" style="width: 17%;">
       <col id='p-value' span="1" style="width: 18%;">
    </colgroup>
<tr> <th> Atom </div> </div> </th> <th> Example </div> </th> <th> Effect Size </div> </th> <th> p-value </div> </th>  </tr>
  <tr class="accepted" > <td> <a href='#literal-encoding'> Change of Literal Encoding </a></td>  <td><code>printf("%d", 013) </code></td>  <td align="right"> 0.63 </td> <td> 2.93e-14 </td> </tr>
  <tr class="accepted" > <td> <a href='#preprocessor-statement'> Preprocessor in Statement </a></td>  <td><code>int V1 = 1
#define M1 1
+1;</code></td>  <td align="right"> 0.54 </td> <td> 8.53e-11 <b><a href="2016-snippet-study/errata#preprocessor_simplification">*</a></b> </td> </tr>
  <tr class="accepted" > <td> <a href='#macro-precedence'> Macro Operator Precedence  </a></td>  <td><code>#define M1 64-1
2*M1 </code></td>  <td align="right"> 0.53 </td> <td> 1.77e-07 <b><a href="2016-snippet-study/errata#macro_name">*</a></b> </td> </tr>
  <tr class="accepted" > <td> <a href='#assignment-value'> Assignment as Value </a></td>  <td><code>V1 = V2 = 3; </code></td>  <td align="right"> 0.52 </td> <td> 3.78e-10 </td> </tr>
  <tr class="accepted" > <td> <a href='#logic-control'> Logic as Control Flow  </a></td>  <td><code>V1 && F2(); </code></td>  <td align="right"> 0.48 </td> <td> 5.62e-09 </td> </tr>
  <tr class="accepted" > <td> <a href='#post-increment'> Post-Increment/Decrement  </a></td>  <td><code>V1 = V2++; </code></td>  <td align="right"> 0.45 </td> <td> 6.98e-08 </td> </tr>
  <tr class="accepted" > <td> <a href='#type-conversion'> Type Conversion  </a></td>  <td><code>(double)(3/2) </code></td>  <td align="right"> 0.42 </td> <td> 5.17e-07 </td> </tr>
  <tr class="accepted" > <td> <a href='#reversed-subscript'> Reversed Subscripts  </a></td>  <td><code>1["abc"] </code></td>  <td align="right"> 0.40 </td> <td> 1.52e-06 </td> </tr>
  <tr class="accepted" > <td> <a href='#conditional-operator'> Conditional Operator  </a></td>  <td><code>V2 = (V1==3)?2:V2 </code></td>  <td align="right"> 0.36 </td> <td> 1.74e-05 <b><a href="2016-snippet-study/errata#nested">*</a></b> </td> </tr>
  <tr class="accepted" > <td> <a href='#infix-precedence'> Infix Operator Precedence  </a></td>  <td><code>0 && 1 || 2 </code></td>  <td align="right"> 0.33 </td> <td> 5.90e-05 </td> </tr>
  <tr class="accepted" > <td> <a href='#comma-operator'> Comma Operator  </a></td>  <td><code>V3 = (V1+=1, V1) </code></td>  <td align="right"> 0.30 </td> <td> 2.46e-04 </td> </tr>
  <tr class="accepted" > <td> <a href='#pre-increment'> Pre-Increment/Decrement  </a></td>  <td><code>V1 = ++V2; </code></td>  <td align="right"> 0.28 </td> <td> 6.89e-04 </td> </tr>
  <tr class="accepted" > <td> <a href='#implicit-predicate'> Implicit Predicate  </a></td>  <td><code>if (4 % 2) </code></td>  <td align="right"> 0.24 </td> <td> 4.27e-03 </td> </tr>
  <tr class="accepted" > <td> <a href='#repurposed-variable'> Repurposed Variable  </a></td>  <td><code>argc = 7; </code></td>  <td align="right"> 0.22 </td> <td> 6.66e-03 </td> </tr>
  <tr class="accepted" > <td> <a href='#omitted-brace'> Omitted Curly Braces  </a></td>  <td><code>if (V) F(); G(); </code></td>  <td align="right"> 0.22 </td> <td> 8.64e-03 </td> </tr>


  <tr><th colspan=4 class="centered" style="border-top: 1px solid #ddd;"> Unaccepted Atom Candidates </div> </div> </th></tr>

  <tr class="rejected" > <td> <a href='#dead-unreachable'> Dead, Unreachable, Repeated  </a></td>  <td><code>V1 = 1;
V1 = 2; </code></td>  <td align="right"> 0.16 </td> <td> 0.059 </td> </tr>
  <tr class="rejected" > <td> <a href='#arithmetic-logic'> Arithmetic as Logic  </a></td>  <td><code>(V1-3) * (V2-4) </code></td>  <td align="right"> 0.10 </td> <td> 0.248 </td> </tr>
  <tr class="rejected" > <td> <a href='#pointer-arithmetic'> Pointer Arithmetic  </a></td>  <td><code>"abcdef"+3 </code></td>  <td align="right"> 0.03 </td> <td> 0.752 <b><a href="2016-snippet-study/errata#pointer_arithmetic">*</a></b> </td> </tr>
  <tr class="rejected" style="border-bottom: 1px solid #ddd;" > <td> <a href='#constant-variables'> Constant Variables  </a></td>  <td><code>int V1 = 5;
printf("%d", V1); </code></td>  <td align="right"> 0.00 </td> <td> 1.000 </td> </tr>
   </table>
<div width=40% style="text-align: center"> * See <a href='2016-snippet-study/errata'>errata</a> for details </div>

<br/><br/><hr/><br/>

<style>
  #atom-contingencies {
    width: 38rem;
    border: none;
  }

  #atom-contingencies th {
    border: none;
    padding-top: 10rem;
    text-align: center;
  }

  th.rotated {
    //height: 10rem;
    white-space: nowrap;
  }

  th.rotated > div {
    //transform:   translate(38px, 55px) rotate(-45deg);
    transform:   rotate(-90deg);
  }

  th.rotated > div > span {
    //border-bottom: 1px solid #ccc;
    //padding: 5px 10px;
  }
</style>

<a name='contingency-tables'></a>
<h2> Contingency Tables </h2>

<p>
  In our <a href="/2016-snippet-study/">Existence experiment</a> we analyzed our
  results by comparing pairs of obfuscated and transformed code. For each pair
  we categorized whether subjects got both snippets correct, only one, or
  neither. From these counts we could apply a clustering-adjusted variation of
  McNemar's test (supplied by <a href="https://cran.r-project.org/package=clust.bin.pair">clust.bin.pair</a>).
  Below are these data.
  </p>

<!-- print(xtable(atom.contingencies[,-1]), include.rownames=FALSE, sanitize.text.function=identity, type="html") -->
<!-- html table generated in R 3.3.1 by xtable 1.8-2 package -->
<!-- Mon May  1 15:57:33 2017 -->
<table border=1 id='atom-contingencies'>
  <tr> <th style="width: 50%"> <div><span> Atom </span></div> </span></div> </th> <th class="rotated"> <div><span> Both Correct </span></div> </th> <th class="rotated"> <div><span> Obfuscated Correct </span></div> </th> <th class="rotated"> <div><span> Transformed Correct </span></div> </th> <th class="rotated"> <div><span> Neither Correct </span></div> </th>  </tr>
<tr> <td> Change of Literal Encoding </td> <td align="right">  35 </td> <td align="right">   2 </td> <td align="right">  89 </td> <td align="right">  20 </td> </tr>
<tr> <td> Preprocessor in Statement </td> <td align="right">  39 </td> <td align="right">   5 </td> <td align="right">  73 </td> <td align="right">  28 </td> </tr>
<tr> <td> Macro Operator Precedence </td> <td align="right">  53 </td> <td align="right">   2 </td> <td align="right">  37 </td> <td align="right">   5 </td> </tr>
<tr> <td> Assignment as Value </td> <td align="right">  59 </td> <td align="right">   6 </td> <td align="right">  68 </td> <td align="right">  13 </td> </tr>
<tr> <td> Logic as Control Flow </td> <td align="right">  32 </td> <td align="right">  12 </td> <td align="right">  72 </td> <td align="right">  30 </td> </tr>
<tr> <td> Post-Increment/Decrement </td> <td align="right">  75 </td> <td align="right">   4 </td> <td align="right">  54 </td> <td align="right">  13 </td> </tr>
<tr> <td> Type Conversion </td> <td align="right">  75 </td> <td align="right">  10 </td> <td align="right">  52 </td> <td align="right">   9 </td> </tr>
<tr> <td> Reversed Subscripts </td> <td align="right">  70 </td> <td align="right">   6 </td> <td align="right">  40 </td> <td align="right">  30 </td> </tr>
<tr> <td> Conditional Operator </td> <td align="right"> 109 </td> <td align="right">   1 </td> <td align="right">  34 </td> <td align="right">   1 </td> </tr>
<tr> <td> Infix Operator Precedence </td> <td align="right"> 105 </td> <td align="right">   4 </td> <td align="right">  25 </td> <td align="right">  11 </td> </tr>
<tr> <td> Comma Operator </td> <td align="right">  53 </td> <td align="right">  17 </td> <td align="right">  50 </td> <td align="right">  26 </td> </tr>
<tr> <td> Pre-Increment/Decrement </td> <td align="right">  82 </td> <td align="right">  11 </td> <td align="right">  35 </td> <td align="right">  18 </td> </tr>
<tr> <td> Implicit Predicate </td> <td align="right"> 108 </td> <td align="right">   6 </td> <td align="right">  20 </td> <td align="right">  12 </td> </tr>
<tr> <td> Repurposed Variables </td> <td align="right">  54 </td> <td align="right">  15 </td> <td align="right">  33 </td> <td align="right">  44 </td> </tr>
<tr> <td> Omitted Curly Braces </td> <td align="right">  77 </td> <td align="right">  13 </td> <td align="right">  33 </td> <td align="right">  23 </td> </tr>
<tr> <td> Dead, Unreachable, Repeated </td> <td align="right"> 138 </td> <td align="right">   1 </td> <td align="right">   6 </td> <td align="right">   1 </td> </tr>
<tr> <td> Arithmetic as Logic </td> <td align="right"> 133 </td> <td align="right">   4 </td> <td align="right">   8 </td> <td align="right">   1 </td> </tr>
<tr> <td> Pointer Arithmetic </td> <td align="right">  67 </td> <td align="right">  21 </td> <td align="right">  23 </td> <td align="right">  35 </td> </tr>
<tr> <td> Constant Variables </td> <td align="right"> 142 </td> <td align="right">   2 </td> <td align="right">   2 </td> <td align="right">   0 </td> </tr>
</table>





<br/><br/><hr/><br/>

<style>

div.atom_title {
  font-size: 16pt;
  font-weight: bold;
  margin-bottom: 1em;
}

div.atom_description {
  margin-top: 2em;
}

div.atom_description:after {
  content: "";
  display: block;
  height: 1.5px;
  width: 50%;
  background: #CCC;
  margin-top: 1em;
}

span.example {
  font-size: 16pt;
  display: block;
}

span.example_type {
  float: left;
  width: 7em;
}

span.example_code {
   display: pre;
   unicode-bidi: embed;
   font-family: monospace;
   white-space: pre;
}
</style>

<div id='atom_descriptions'>

<a name='atom-descriptions'></a>
<h2> Atom descriptions </h2>

<div class='atom_description'>
<a name='literal-encoding'></a>
<div class='atom_title'>Change of Literal Encoding</div>
<p>All numbers are stored in binary inside of a computer, but for human
convenience we tend to represent numbers in decimal, and occasionally
hexadecimal or octal for certain uses. Even though different representations
can hold the same number, their accessibility to humans for different
computations can be very different.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'> 208 & 13</span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'> 0xD0 & 0x0D</span></span>
</div>



<div class='atom_description'>
<a name='preprocessor-statement'></a>
<div class='atom_title'>Preprocessor in Statement</div>
<p>Preprocessor directives must stand alone on their own line. After
the preprocessor runs, however, that line is treated as whitespace. As
a result, preprocessor directives may be present in the middle of an
expression as long as they are on their own lines. Since the preprocessor
directive and the source code are processed in different compiler phases,
they are processed independently. Yet, to the casual reader, they appear
to interact with each other.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'>

  int V1 = 1
  #define M1 1
  +1;

</span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'>

  #define M1 1
  int V1 = 1 + 1;

</span></span>
</div>



<div class='atom_description'>
<a name='macro-precedence'></a>
<div class='atom_title'>Macro Operator Precedence</div>
<p>Macros can be used to add many features to C, including guaranteed
inlining, duck-typing, and adding metadata like line number and file
name to program output. Unfortunately, macro references are impossible
to distinguish from other identifiers and can often act in ways that
variables and functions can not.  This can cause readers to be misled.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'>

  #define M1 64 - 1
  2 * M1

</span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'> 2 * 64 - 1 </span></span>
</div>



<div class='atom_description'>
<a name='assignment-value'></a>
<div class='atom_title'>Assignment as Value</div>
<p>The assignment expression changes the underlying state of the machine
when it executes. However, it also returns a value. Often when reading an
assignment expression people will forget one of the two effects of the
expression.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'> V1 = V2 = 3;</span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'> V2 = 3; V1 = V2;</span></span>
</div>



<div class='atom_description'>
<a name='logic-control'></a>
<div class='atom_title'>Logic as Control Flow</div>
<p>Traditionally, the && and || operators are used for logical conjunction
and disjunction, respectively, in predicates. Due to short-circuiting,
they can also be used for conditional execution.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'> V1 && F2();</span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'> if (V1) F2();</span></span>
</div>



<div class='atom_description'>
<a name='post-increment'></a>
<div class='atom_title'>Post-Increment/Decrement</div>
<p>The post-increment (and decrement) operator increases the value of
its operand variable by 1, while returning the original value of the
variable. Confusion here arises because the value of the expression is
different from the resultant value of the variable.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'> V1 = V2++; </span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'> V1 = V2; V2 += 1; </span></span>
</div>



<div class='atom_description'>
<a name='type-conversion'></a>
<div class='atom_title'>Type Conversion</div>
<p>The C compiler will implicitly convert types in various situations
when there is a mismatch, but sometimes this conversion also results in
an implicit change of outcome from what the author may have intended.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'> 3/2; </span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'> trunc(3.0/2.0); </span></span>
</div>



<div class='atom_description'>
<a name='reversed-subscript'></a>
<div class='atom_title'>Reversed Subscripts</div>
<p>Arrays can be indexed using the subscript operator, but underneath ``E1[E2]
is identical to (*((E1)+(E2)))''. Since addition is commutative, so too is the
subscript operator.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'> 1["abc"]; </span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'> "abc"[1] </span></span>
</div>



<div class='atom_description'>
<a name='conditional-operator'></a>
<div class='atom_title'>Conditional Operator</div>
<p>The conditional operator is the only ternary operator in C, and functions
similarly to an if/else block. However, the conditional operator is an
expression for which the value is that of the executed branch.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'> V2 = V1 == 3 ? 2 : 1;</span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'>

  if (V1 == 3) {
     V2 = 2;
  } else {
     V2 = 1;
  }

</span></span>
</div>



<div class='atom_description'>
<a name='infix-precedence'></a>
<div class='atom_title'>Infix Operator Precedence</div>
<p>There are 32 binary operators (operators which accept one operand
before and one operand after) in C. Each of these operators falls into
one of 15 precedence classes and has either right-to-left or left-to-right
associativity. Needless to say, the average programmer knows only a
functional subset of the information needed to correctly parse complicated
expressions of binary operations. </p>
<p>Our preferred method for removing precedence confusion is with
parenthesis. Other removal transformations are possible, such as
introducing intermediate identifiers. These other strategies can have
larger impacts on the structure of the code and so were avoided when
possible.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'> 0 && 1 || 2</span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'> (0 && 1) || 2</span></span>
</div>



<div class='atom_description'>
<a name='comma-operator'></a>
<div class='atom_title'>Comma Operator</div>
<p>The comma operator is used to sequence an otherwise ambiguous series
of computations. Whether due to its eccentricity, or its odd precedence,
the comma operator is commonly misinterpreted.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'> V3 = (V1++, V1);</span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'>

  V1++;
  V3 = V1;

</span></span>
</div>



<div class='atom_description'>
<a name='pre-increment'></a>
<div class='atom_title'>Pre-Increment/Decrement</div>
<p>Similar to post-increment and post-decrement, the pre-increment
and pre-decrement operators change a variables value by one. In
contrast to the other operators, pre-increment and pre-decrement
first update the variable then return the new value, instead of the old.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'> V1 = ++V; </span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'> V2 += 1; V1 = V2; </span></span>
</div>



<div class='atom_description'>
<a name='implicit-predicate'></a>
<div class='atom_title'>Implicit Predicate</div> The semantics of an expression
change based on the context in which its consumed. In the rvalue of a char
assignment, i.e., char c = expr the expression (assuming it itself makes no
variable assignments or updates) can result in up to 256 different states of the
of the program. By contrast, the same expression in the context of a predicate,
i.e., if (expr) can only result in a maximum of two different program
states. Often in the context of a condition, the reader can become confused as
to effect of a certain predicate value. We clarified these by explicitly adding
logical operators to the predicates.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'> if (4 % 2)</span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'> if (4 % 2 != 0) </span></span>
</div>



<div class='atom_description'>
<a name='repurposed-variable'></a>
<div class='atom_title'>Repurposed Variable</div>
<p>By convention, variables tend to have a single conceptual identify and
represent one idea. When a variable is used in different roles across the
lifetime of the program, its current purpose can be difficult to follow.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'>

  int main(int argc, char **argv) {
    argc = 7;
    ...

</span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'>

  int main(int argc, char **argv) {
    int V1 = 7;
    ...

</span></span>
</div>



<div class='atom_description'>
<a name='omitted-brace'></a>
<div class='atom_title'>Omitted Curly Braces</div>
<p>C looping and selection exhibit dynamic behavior over a trailing
statement. The trailing statement, optionally, can be enclosed in
braces for clarity, or to extend the number of sub-statements modified
by the loop or conditional. Confusion may arise when the braces are
omitted for brevity.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'> if (V1) F1(); F2();</span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'> if (V1) { F1(); } F2(); </span></span>
</div>



<div class='atom_description'>
<a name='dead-unreachable'></a>
<div class='atom_title'>Dead, Unreachable, Repeated</div>
<p>Redundant code is code that will either never be executed, or it's
effects are immediately invalidated. It can be counter-intuitive that
code exists to have no impact on the output of the program.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'> V1 = 1; V1 = 2; </span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'> V1 = 2; </span></span>
</div>



<div class='atom_description'>
<a name='arithmetic-logic'></a>
<div class='atom_title'>Arithmetic as Logic</div>
<p>Arithmetic operators are capable of mimicking any predicate formulated
with logical operators. Arithmetic, however, implies a non-Boolean range,
which may be confusing to a reader.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'> (V1 - 3) * (V2 - 4)</span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'> V1 != 3 && V2 != 4</span></span>
</div>



<div class='atom_description'>
<a name='pointer-arithmetic'></a>
<div class='atom_title'>Pointer Arithmetic</div>
<p>Pointers admit several operations like integer addition/subtraction,
but, in many cases, these operations are interpreted by the reader to
update the target data instead of the pointer data.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'> "abcdef"+3</span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'> "abcdef"[3]</span></span>
</div>



<div class='atom_description'>
<a name='constant-variables'></a>
<div class='atom_title'>Constant Variables</div>
<p>Constant variables are a layer of abstraction that, in the
context of a complex system, let us focus on the concept a value
represents rather than the value itself. When simply trying to
determine the output of a piece of code, having a layer of indirection
that hides the value of your data can cause difficulty.</p>
<span class='example'><span class='example_type'>Confusing:</span><span class='example_code'> V1 = V2; </span></span>
<span class='example'><span class='example_type'>Non-confusing:</span><span class='example_code'> V1 = 5; </span></span>
</div>