Theory vs. Practice

Diagnosis is not the end, but the beginning of practice. Martin H. Fischer

› How to evaluate "Competence"?

While talking about business trends with a circle of executives, the director of an R&D center in Asia lamented about the lack of "true" competences. Intrigued about what his company could be missing I invited him to explain what he meant.

He said that, far too often, pouring 20 or 200 persons on a project made no difference in terms of innovation. While several of their R&D labs work in different countries, all seem to produce similar results, and this is like an invisible wall which cannot be broken, whatever the investments.

This reminded me the years spent at large industry players like THALES in France or SPC Corp. in the US. Large companies often have a "postdoc" policy for recruiting R&D staff. University alumni often hire pals, people that look like them, or at least candidates with a common background.

Not willing to restart the ancestral debate between Engineers vs Academics, I promised to send him a small test that would both illustrate the nature of the problem – and a quick way to help resolving it.

At this time, I did not anticipate the scale of the feedback – nor the impact that such a benign confidence could have provoked.

The only criteria worth considering

A tree is judged by its fruits.

The only reliable criteria is past achievements. Not what people say, not what they had others saying about them – only what they have done, by themselves, alone.

How can this be checked?

Have someone competent (ideally your most experienced person in that field) ask them questions about the most difficult challenges they have met in the past and how they have resolved these problems.

Skilled people will be delighted to expose the subtle and elegant solutions they have involved at each step of the process.

Cheater will repeat the sterile and boring banalities that they have read in newspapers.

When you need deeper insights – The test

The test consists in asking people to optimize a small portion of source code. Seeking suitable code and not willing to enter into obscure technical details, I remembered a Swiss academic paper written by four Swiss Doctors (ETHZ, FHNW), some of them working at UBS (a Swiss bank) and Kudelski Security (a Swiss data-security expert).

The paper, based on established algorithms, aimed to deliver: "a hash function designed for resource-constrained hardware environments, as RFID tags... to minimize area and power consumption, yet offer strong security guarantees".

In a world deploying the "Internet of Things", the initiative has some value. And, indeed, the involved algorithms behave better than the average US standard, as the paper lengthly states.

The problem comes from somewhere else: the implementation, also called "execution" in some circles, probably for a reason. And there, just reading the code once makes it clear that the gap is real... and substantial:

1. have to remove obviously redundant calculations in all(!) the loops
2. have to correct the many functions returning an unused(!) zero value
3. have to replace all the slow (involving locks) tiny malloc() calls by alloca()
4. have to replace plain loops by inlined bzero() working on computer words
5. have to fix other 'minor' code optimizations and cleanup for "Java-like" C code
6. have to add the missing calculation in the paper's text and code comments
7. have to redefine local functions and variables as static (namespace pollution)
8. have to create a header file reasonably isolating too generic defines
9. have to add conditional preprocessor commands to reduce code bloat
10. have to protect various critical data like IVs with the const modifier

Such a summary cannot reflect the scale of the problems, so here is one of the functions extracted from the original code (the erratic indentation and formating were left untouched):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

int final(hashState *state, u8 *out)
{
  int i;
  int outbytes=0;
  u8 u;

  /* append '1' bit */
    state->x[8*(WIDTH-RATE)+state->pos*8] ^= 1;

  /* permute to obtain first final state*/
  permute(state->x);

  /* zeroize output buffer */
  for(i=0;i<DIGEST;++i)
    out[i]=0;

  /* while output requested, extract RATE bytes and permute */       
  while (outbytes < DIGEST ) {
    
    /* extract one byte */
    for(i=0;i<8;++i) {
      u = state->x[8*((WIDTH-RATE))+i+8*(outbytes%RATE)] &1;
      out[outbytes] ^= (u << (7-i));
    }
    
    outbytes += 1;
    
    if (outbytes == DIGEST )
      break;

    /* if RATE bytes extracted, permute again */
    if ( ! ( outbytes % RATE ) ) {
      permute(state->x);
    }
  }

  return 0;
}

To better meet the initial goal of consuming less CPU resources, this code was quickly rewritten to get a smaller, more readable (and therefore easier to maintain) version which is also faster and more scalable:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

void quark_end(quark_ctx *ctx, u8 *dst) 
{
   /* append '1' bit */
   ctx->x[8 * (QUARK_WIDTH - QUARK_RATE) + ctx->pos * 8] ^= 1;
   permute(ctx->x); /* permute to obtain first context */
   {
      const u32 *src = &ctx->x[8 * (QUARK_WIDTH - QUARK_RATE)];
      int i = 0;
      for(;;) { 
         const u32 *s = &src[8 * (i % QUARK_RATE)];
         dst[i] ^= ((s[0] & 1) << 7)
                 | ((s[1] & 1) << 6)
                 | ((s[2] & 1) << 5)
                 | ((s[3] & 1) << 4) /* use sparse context bits */   
                 | ((s[4] & 1) << 3) /* to build each dst[] byte */
                 | ((s[5] & 1) << 2)
                 | ((s[6] & 1) << 1)
                 | ((s[7] & 1) << 0);

         if(++i == QUARK_DIGEST) break; /* done */

         if(!(i % QUARK_RATE))
            permute(ctx->x); /* after QUARK_RATE bytes extracted */
      }
   }
}

Pointless code (buffer zeroing), redundant tests (while() and if() break;), and duplicated calculations (in the main loop) were the most obvious issues. The code above achieves further savings by using less variables, by allowing better locality to make it possible to optimize CPU registers and caches usage, by using one single assignation in the now unrolled nested loop... and by not returning an unused constant. Other functions of the original C code use more complex calculations and therefore would gain even more from being revisited in a similar way.

Given the few features used by this code, almost everything that could have been done wrong is illustrated here. We should note that other academic papers (Swiss, French or US) are written with the same dire ignorance of how to efficiently use computers.

Remember, this "Dream Team" of four academic experts came from the best Swiss universities – and was hired by the best head-hunters for the best Swiss companies. These Doctors know how to calculate the abstract "computational" cost of an algorithm... but, once in the real life, the lack of quality of their implementation severely defeats the purpose of the theory.

In this matter, Switzerland is in good company because all the universities teach the same "made in USA" educational programs. Unsurprisingly, they all get similar results.

What is the contrary of Diversity?

"University" says the joke. A common basis is certainly needed to communicate and nobody will question that. But how can anyone innovate or merely do better than others if we all have to use the same tools? Ironically, the problem comes from what is not taught in schools... and why:

Let's recapitulate: our goal here is to make it possible for companies to evaluate people's competence – not at remembering the thousands of Java or C# function names – but rather at doing what your competitors won't be able to do. Management delegates the hiring process to HR (Human Resource) departments... where the skills they are supposed to rate are also lacking.

Using key employees as another filtering layer will work until they reach their own level of incompetence: at this point, they will preserve their job by disqualifying possible challengers and will rather hire incompetent coworkers for their lower level of perceived danger. When the practice is widespread the costs cumulate, exactly like in... universities.

OK, how to avoid this (purely human) trap? By introducing a small dose of disruption. It doesn't mean the kind that likes to bother others. What all organizations need is both a predictable and large basis and a few seasoned explorers to open new, better routes:

A common belief is that major innovations require factors like market power, finance, and academic experts. The reality is that, by entering a game started long ago by precursors, these late players are merely validating the disrupting nature of innovations that raised their attention after threatening their comfort zone. In contrast, small players innovate because, lacking any other option, they have to.

Here is the bounty. Seize it or others will do it before you – not because it's "fun" or "cool" (this won't feed your family unless you are a story-teller) – but because uncontrolled disruption will always happen at the expenses of incumbents.

I was stunned to see a copy of the computer-printed letter above returned with an hand-written reply from this R&D director who expressed both enthusiasm and gratefulness... as well as a follow-up of the gradual steps of improvements. Proof that power not always implies comfort-zone seeking. To all those trying to improve every day, keep up the good work, History is on your side!