Consultancy is a business based on the sell of expert labour. As such, consultants are expected to be busy as much time as possible at a commercial rate; i.e.: If the commercial rate is 100 €/hour, and the consultant works 1,800 hours/year, the company expects the consultant to produce a number as close to 180,000€/year as possible. I will call this the consultant’s nominal production.
When a consultancy company recruits technical professionals, some understand that we can trick the system. The trick is called software. If we automatize a part of our work, we have more time to slack, while “producing” closer to our nominal production. Maybe if a consultant is motivated enough, will produce over the nominal rate by working all the time and automatizing the over time that is most of the times “required by the job”.
Many smart individuals automatize their work in order to work less while keeping the production on schedule. Sometimes, the management is the one fostering this automation in order to raise the productivity. In this regard, the production of software is a natural consequence of the competitiveness in the consultancy business.
Now comes the horror; Most people understand the effects of automation. Very few understand the consequences. It turns out that there is a complete discipline that has evolved around the idea of task automation and its consequences. It is called software engineering. This fact is often neglected, either by ignorance or by deliberate action: I’m no programmer, I’m an engineer, physicist, mathematician, etc…
Fear the excel engineers
The first level of horror is composed by the people that believe that Microsoft Excel is the best program ever produced, and hence it is ideal for everything they do. Including software. These people might be very knowledgeable about a certain topic, and they feel that they can put some of that knowledge into an Excel workbook to be able to produce faster.
The problem lies in the practical impossibility of fact-checking a “program” produced in excel. Hundreds of formulas in dozens of sheets might be correct at some point in time. However, the chance that an Excel workbook remains correct as it evolves and changes hands is close to zero. That is why whenever someone claims that they have an “Excel sheet for that” I tremble.
Yeah, VLOOKUP doesn’t do what you think it does…
Fear the Matlab engineers even more
Deep down, Excel engineers know that what I just said is true, but they cannot do it better because of the lack of formal programming skills. Some people have had programming formation at the university. Those are the ones that use the proprietary programming languages learnt at the university. Those would mostly be Matlab, Mathcad, Mathematica, GAMS, etc… Each of those programs require a license, and none of them provides modern programming paradigms. In fact most of them foster the all-in-one-file mentality despite having some sort of modular capabilities.
The problem here is more subtle.
These people using these proprietary languages, are very productive. Usually they have convinced the management about the countless benefits of their software up to the point that the management believes that they have a product. These Matlab (insert here your proprietary language) products never start intentionally, they start as the automation of a tedious calculation. But because they prove the concept of the calculation they were programmed for, they secure internal funding to continue. Because those languages lack the possibility of better software structures (Object structures, maybe functional programming, …) and because the people that produce them think that writing 3000-line functions is a good practice (everything is in one place you know…), those programs become unmaintainable pieces of spaghetti code.
Subtle but destructive.
Eventually these people will realize that what they have done is a cumulative summation of terrible practices. Hence, they will start trying to convince the management to fund a project to re-write the software in C++, Java or Python. Don’t get me wrong, if you need Matlab, Mathcad, Mathematica, GAMS, etc… and you can afford the license, go for it. But do not pretend to be doing any serious software with it. Those languages are fine to produce a proof of concept, which is the typical application that they serve in the university.
Software done right saves money, the opposite is also true
The cost of the Excel engineers is the cost of having done everything wrong and be unable to realize about it. Horror story #1, Horror story #2 Mind the dates of those stories.
Usually the cost of the Matlab engineers is not the incorrectness but the cost of rewriting everything so it does make sense.
What to do?
For the managers:
- The first thing to do always is to document the process. Make the technical people write clear literature for yourself and for the future generations of users. Some people in the organization would refuse to do it because they think they would be revealing their secret sauce. Try to convince them to come around. Otherwise fire them or reduce the team dependency on them, they are not that good.
- The next thing to do: Whenever you see that some internal tool is taking traction: make everyone stop and think where is that going. If it is going to become more than a script, make everyone involved make a design, if the budget allows it, bring good software engineers in the process and give them control.
- The cost of not doing this is usually much higher than the cost of doing it right. Bear in mind that a bad design is something that will be making people unhappy for years, and if people are miserable at their job they quit. Design is paramount.
For the consultants:
- Document everything. If you use a real programming language, tools like Sphinx make everything easier.
- Learn software engineering principles: Design patterns, best practices, etc. This includes acknowledging your limitations.
- Pack everything into well documented libraries to be customized for each project.
- Beautiful and easy to read is better than “efficient”. Write code as if Donald Knuth was going to have a look at it.
I hope this helps, it certainly has helped me realize of my limitations, to go back to learn the things I don’t know instead of come up with suboptimal solutions, to foresee problems and to try to avoid the re-writing trap.
Santiago Peñate Vera 