How many times have you heard sentences like “if it is not broken, don’t fix it” or “do not reinvent the wheel”? many times for sure. The underlying message is that, if there is something out there that already works, do not waste your time making it your self. This article advocates exactly for the opposite.

We live in a time where we delegate most of our activities on proven technology that we do not question or understand. For instance. the average person does not know how a car works in detail, or how it was made. Only when a person needs to build a car, will encounter all sorts of barriers, that is why almost no one starts a car company. We assume that building a car ourselves is going to be more expensive than buying one, so we accept it and move on.

But what if we decided to build the car anyway?

Shu-Ha-Ri is a concept from the Japanese martial arts that describes the stages of achieving the mastery of a subject;

• Shu: Copy and learn the traditional knowledge, techniques, heuristics and proverbs.
• Ha: Break with the tradition, start to innovate discarding or modifying the forms.
• Ri: Completely depart from the forms and open the door to creative technique, and arrive in a place where we act in accordance with what our heart/mind desires, unhindered while not overstepping laws.

When we accept and not question the world around us with sentences like do not reinvent the wheel. We are in a perpetual state of ignorance (not even in the Shu stage), where we deny ourselves the possibility of becoming masters in something.

By using a technology, we accept all the conditions and design decisions that were involved in the development of that technology. That means that we cannot modify that technology to serve us, but rather we need to adapt ourselves to the conditions imposed by it. If we refuse to accept those conditions we would have to go through the shu-ha-ri process in order to first reproduce the technology, and then redesign it for our own benefit. The key is to be able to asses if the long and hard process of becoming a master is worth the cost of breaking apart with the convenience of available technology.

Some times it is.

I am electrical engineer in power systems and it is safe to say that the field is one of the most conservative ones among the engineering disciplines. Yet, the field is facing tremendous challenges that demand innovation. During the first years of practice, I observed that everyone dealt with the calculations in a very dogmatic manner, relying on the available calculation software as if it was something coming from the gods, as something that you should embrace and not change. But I saw myself caged by those very old fashioned programs, so I decided to change.

I started developing my own software to serve my needs. Can you imagine yourself programming a new word processor just because you don’t like Microsoft Word? That is exactly what I did. It took me a huge amount of time to learn and often re-invent how certain parts of the program should work, because those steps are not documented anywhere, not in books, not in papers and not even in other software. I developed my own power systems solver, which I utilise to work with. I am the blacksmith who built his own forge. That is like that because I refused to adapt to the ill conditions imposed by the available technology and I decided to try to impose my own design decisions.

Because those design decisions adapt better to the needs of the power systems of the future, I can work in the needs of the power systems of the future with far less effort than using the traditional software that was designed for a world with almost no renewable power and where the time-varying planing did not exist. This situation has saved me hundreds of hours of having to adapt my work to the software, because I had my own software that I was able to easily adapt for my work.

The benefit of reinventing the wheel is not only the capricious advantage of having my own technology, the real benefit is to get to master the basic principles and to be able to incorporate ground breaking innovation into the software I make. I am in the stage of Ri. That is the true value of reinventing the wheel; the innovation, the long term cost saving, and the journey itself.

The Oxford dictionary describes the word model in several ways, but the one that I want to talk about is this:

A simplified description, especially a mathematical one, of a system or process, to assist calculations and predictions.

i.e.: ‘a statistical model used for predicting the survival rates of endangered species’ 

A model is a static thing. We make a model and we use it. We may even update it!

The effects

The word model has an intriguing effect; It turns something complex into something simple in the minds of the interlocutor. Those of us that make technical software to deal with sophisticated modelling of the reality, face the effects of the word “model”.

Imagine a computer program that uses hundreds of physical and mathematical laws working together to simulate and predict the prices of the electrical markets in the long term (…) Humm, it sounds hard.

Now, replace that thought with “a market prediction model”. A market prediction model may very well be an excel sheet with a linear regression. We’ve just made our life so much simpler!

Right there is where the devastating effect of the word model resides. The word “model” simplifies the role of the modeller to something in the reach of the interlocutor; We all can model, so what you do should not be that difficult.

People say model when they actually mean software

The word “model” is often used to describe software. I would say that the word software is a generalisation of the word model in the sense that we are describing. However, a model does not run on the customer premises. A model does not handle the thousands of required inputs. A model does not cooperate with other models automatically and in parallel to provide a faster and more accurate answer. Software does, and software is hard to make.

Models are good. Models in software are great

We all can model, we know that. But if you can put a model into a computer program and deliver it, you have a product. Products are what make money, models do not and the management needs to recognize this fact. Models are fine, but the revenue comes though software, because that is what you can sell and maintain. A model in software provides credibility before the client. Trying to sell a model to a client shows that you are an amateur. Unfortunately many clients can not recognize that either.

If you can make models and put them into a computer program you are a pro. Remember this words about the word “model” when you, modeller that makes software, negotiate your next salary.

After having been working in consultancy in a variety of companies, I have witnessed the following process over and over;

• Consultancy project “A” is sold to a customer.
• Software “A” is made to meet the customer requirements.
• Because of resources restrictions, the software “A” is done as a monolithic block. It is “integrated”.
• Software “A” works, and gets the management recognition. The bespoke software “A” becomes a part of the company catalogue for other customers.

some months pass

• The management proposes to offer software “A” for the project “B”. This seems logical because “A” and “B” are in the same domain of expertise.
• Engineers agree to modify software “A” to do “B”.
• After a month of work, engineers find out that adapting “A” to “B” is impossible, and after convincing the management of it, they produce software “B”, again as a monolithic block to match the requirements of project “B” due to the resources constraints.  Best practices? there’s no time for that.
• Software “B” works. Now there are two bespoke software pieces to be offered in the future.
• …
• The cycle continues, while the management claims that they need to improve revenue, cut down costs, etc.

Any clue?

This is the usual modus operandi of a software as a product company. In such organizations, the software is seen as a commodity, with little to no added value other that the person-hours used to produce it. Because the software is seen as a commodity, there is no added value in a good design nor the use of best practices in order to recycle that knowledge (to scale). However the management asks for actions to scale, which in business neo-language, translates into more hours per the same salary per employee.

Software that scales

The first intuitive feature of a computer program is that it automatizes something, it does something faster or better than a human being. This is the low hanging fruit of producing software and it is almost always achieved. Even the most horrific piece of software will be faster than a human being at calculating.

The second, and most important feature of software is that it can be incremental. This one is harder to achieve, because it is not obvious. This feature implies that the addition of every new feature takes less and less human work as the software evolves, making the potential marginal revenue higher. This feature is explained in the book  Clean Architecture by Robert C. Martin.

If there comes the time when to add a new feature is more complicated than to re-write the whole application, that is the empirical prove that the software is not scalable.

Practical keys

So, how does one make scalable software in a consultancy-like business?

Make libraries and never monolithic “integrated” blocks. Once you have a working library that is tested, you can add communications, GUI, etc. When another project comes, the library shall be the core of the new project, having the chance to discard the customized layers added on top.

Write technical documentation of the libraries. This separates the wheat from the chaff, and shows that the code does actually have a scientific background. At least an informed reader can discern if the science behind the code is good enough for the new purpose.

Make libraries and technical documentation available in the organization. If a department (or even an individual) is being scalable, there is no good reason to keep that isolated. Everyone in the company should be aware of the company’s state of the art, and have the chance to benefit from it.

By implementing these simple rules the projects revenue explode. I have examples where months of developments reduced to hours for an equivalent outcome.

Conclusion

• Make libraries (packed knowledge)
• Document the knowledge.
• Make the knowledge available.

I spoke about software because the examples are perhaps clearer to see, however this applies to all human activities including regular consultancy. If from every project one can extract the core of it in a document of some sort, it is already scaling up for the next time by having ready-to-consume knowledge. And that is the key.

GridCal is about to hit version 2.0 after three years of development. So I take the opportunity to announce it. After all it is good and it is free.

GridCal is an open source power systems calculation software that I started because there was nothing like it. There are other competitive open source software programs, but none has a work flow like the comercial programs with graphical user interface, easy to use, powerful, etc. I believe that GridCal surpasses many comercial software in that aspect. I made GridCal so that it is a user friendly program and a calculation library for people to use it as they desire.

Developing an open source project is quite a journey. A journey that has served a double purpose for me; To learn how electrical models work for real and to create an invaluable tool to work with. It takes commitment and a certain work ethic to be able to develop something that others can rely on and extend without your explicit involvement. The commitment part is specially true in my case because I do it in my spare time after working 10 hours a day in a consultancy company.

I have been contacted by Germans, Koreans, Americans, Russians… people at university courses and more seasoned engineers alike. I really enjoy knowing how people use the software and if they find bugs or difficulties that I can solve.

Electrical models

I have had a particular hard time finding models that actually work in a computer implementation. I have spent a fortune on electricity books, and very few provide the essential details that are necessary to build an efficient computer implementation. Some missing topics would be how to compute a line loading, graph algorithms to check if there are islands, or state of the art short circuit among other topics. The electric sector is not as good as the computer science sector in sharing knowledge. Many people develop models and keep them for their own, publish a paper or two with incomplete information and it all stays there.

Luckily I found that the models of another open source project (MatPower) were quite comprehensible and were implemented with speed in mind. The way R. D. Zimmerman, C. E. Murillo formulated the circuit equations, make the implementation much simpler and efficient, and this formulation cannot be found in books. That is because the people who write them do not write software themselves, hence their explanation is  academic (simplistic if you will) and it is hard to make it work in the real life.

Open knowledge

Open source software in the electric field should be used in each and every university. Students would be then able to see the “guts” of the program and learn how it is done in real life. They would be able to see methods that work for real size grids (hundreds or thousands of nodes), instead of a simple method that will lead them nowhere if they want to implement it by themselves to learn. I faced that myself.

I was a researcher, and I was once in trouble by relying on a comercial program which limitations blocked my research. That led me to start programming open source simulators in my own time to be able to do research at work with them. Crazy I know, but I have it now and I use GridCal at work for real projects, and my co-workers do too.

Conclusion

Regardless if you are a student or a professional, give my program a try. And if you feel like giving some feedback or contributing I’ll appreciate it very much. The program can be obtained here.

Enjoy.

When we go to school we are taught about the scientific method; Think of a premise, test it, and if the results are not satisfactory, change the premise and test it again. This has led us to an unprecedented level of innovation over the last two centuries.

Science is about reproducible evidence. If anyone with comparable means and knowledge cannot reproduce a published experiment, such experiment is probably wrong, and its publication should be dismissed unless the authors make the underlying results public and disclose all the possible (and not published) trickery. Otherwise a scientific article is nothing but paper for wrapping the fish.

The academia should be the paladin of such procedure, a temple of truth, right?

WRONG.

The science of hype

Once in a while I like to explore the state of the art of a subject within electrical engineering called “power flow”. Last Christmas I did so.

Lately there has been a break trough in the methods used to solve the problem. However the new method is not mature and cannot be compared yet to the traditional algorithms, but it has a great potential. So once in a while I check if the corners have been polished. This new method is a high profile subject and it is not well understood yet, so if you submit a paper about it, you’re probably going to get it published.

Ding**ding**ding**ding**! I found two winners! Two articles published in 2017 in the most prestigious electrical engineering institution (IEEE) had incorrect formulations of the problem leading to a useless implementation and a complete waste of my time.

I do read the papers on the power flow subjects, and yes I actually program the methods and I publish the source code (I have the means and the knowledge to do it). As a good scientist, I wrote to the authors asking for clarification. One of them promised me some prove of the method (never heard from it again) The other asked me for my references, after providing them along with his method corrected, and source code, I received no further reply.

Both papers had very high claims and really nice looking charts. Both failed to provide any improvement over the existing method while claiming otherwise. Both of them were from research groups from reputed universities. Both were published in a prestigious magazine. Both were a fraud.

Unfortunately, this is not the first time I spend hours verifying a published method only to find some inconsistency or simply that the methods performs poorly when announced otherwise. The truth does not sell magazines apparently.

The incentives turn scientists into hustlers

The universities measure a researchers success in terms of how many papers he/she gets published in so called “high impact magazines” over a period of time. This means that a researcher is in a position where he/she can make up an article from the beginning to the end, with no evidence whatsoever, and still get it published if it looks fine enough (right wording, plenty of famous references, familiar yet “new”, etc.) If you get caught you might lose your reputation and so on, but who does test the stuff published in papers? Well, I do.

Once I was told by a chairman that to copy from one person is to plagiarize, to copy from many is research. I was outraged by this comment. It reflects perfectly the mentality that prevales in the academia. Yet somehow it contains truth, but the words seem so wrong to me.

In academia the game is to publish, to publish a lie if necessary. That will make you pass the bureaucratic requirements and remain as a researcher and thrive.

Another bleeding subject is the authorship. Many times, the poor PhD student (the legitimate author) appears in second or third position in the article authorship, just because being the lead author brings more points to the so called researcher (the department chairman usually) The students accept in fearing of the possible retaliation. This creates a vassalage relationship that endures over time and infiltrates in the academic culture.

Sometimes, the results are legit although not great. Then the facts must be exaggerated and oversold in order to convince the magazine reviewers (also in the game) that the article is worthy of publication.

Hustlers in academia sell hype and bad science.

Separating the wheat from the chaff

Many authors hide facts and publish half way results, or simply wrong statements in order to attract attention and obtain the validation that comes with a published article. This goes against the very purpose of publishing: To lead others towards a research path or avoiding the suffering of going through an unfruitful one. Only optimistic articles seem to be publishable.

‘Novel, amazing, innovative’: positive words on the rise in science papers

My former boss in R&D used the following parable: The seed makers in The Netherlands do not keep their secrets for themselves for too long. They know that making information and seeds available to other seed makers will raise the overall quality of the seeds, improving the benefits for the guild. This illustrates perfectly how research works; Do something, get some benefit and make it available. Others will do the same and you will benefit further from your initial effort.

In my opinion, open source software is the perfect way to do science. It is open, verifiable, and if it is wrong, you can correct it easily. I have met open source researchers both in engineering and biology, and all of them are very successful (they meet their papers quotas) while publishing strictly verifiable findings.

The key is that the development of open source software has lead them to the forefront of their fields. The mastering of the basics along with an open platform where to innovate incrementally has enabled them to go one step further.

Therefore, magazines like IEEE should start asking for hard evidence of the articles published, and open source code is hard evidence in many cases where the publication involves a computer algorithm. This would increase the reputation of the publication and would mean that the published facts are the truth and nothing but the truth, raising the level of the whole field of expertise.

Certainly it would make the subscription fee worth the money.

In this post I am going to discuss how the market has substituted the power plant dispatch, and which mechanisms have been designed so that the market of electricity can provide a feasible real world solution. (Spoiler: there’s still some dispatch optimization being done)

Traditionally, power plants were dispatched so that the overall cost of production was the lowest. This is known as “economic dispatch” and it is an optimization problem that minimizes the cost of satisfying a certain demand with a combination of power plants, while keeping the grid loses to the lowest possible values and avoiding the violations in voltage and loading of the electrical grid components. In my opinion this is the best possible way to operate a power system, since the lowest electricity cost is guaranteed by the mathematical method. Meaning the consumer will pay the minimum possible cost for the given available set of power plants, for which the operational costs have been computed correctly.

Here lies the problem. Given a set of power plants and being those operated at the minimal system cost, why would an electric company invest in more efficient and cleaner technologies? The answer is that they would not invest at all in cleaner technologies unless there is a clear economic benefit such as a feed in tariff.

The market mechanisms are introduced to foster the improvement of the existing power plants and to promote cheaper technologies such as wind and lately solar. So, what is the market? and how can it be applied to the dispatch of power plants?

For every hour or every 15 minutes or any other interval considered by the market for the day ahead, every power plant bids an amount of energy that it can generate at their marginal cost (cost that covers their operation plus a benefit). Low operational cost technologies such as wind and solar can offer their energy at a very low price, whereas very high operational cost technologies such as gas turbines must offer their energy at a higher price. For the same auction, the energy retail companies offer to buy amounts of energy for certain prices. Those that “desperately” need to have a certain amount of energy offer the highest price (knowing that the final price would likely be lower than their bid) and those companies that have flexibility and can afford not to buy the energy in this auction, offer the lowest price.

The generator bids and the retail bids are added up in the respective generation and demand bid curves, and then the crossing point determines the matched demand and its price.

Does the market matched demand have to be the system actual demand? Of course not. It can be anything. In fact every day there is a more or less important deviation of the market matched energy with the actual system demand. Furthermore, the system actual demand of the day ahead for which the market solution is being computed is unknown, so there is always uncertainty.

I will call the market crossing point, the market solution to the system dispatch. In a market with hourly intervals the market solution is composed by 24 values. Of course this solution does not minimize anything, in a perfect competition market it would, but the electric system cannot be in perfect competition because I, as consumer cannot choose where to get the electrons from. Because of this, the transmission system operator (TSO) needs to run an optimal power flow (an optimization problem much like the optimal dispatch) to check whether the market solution makes any physical sense. Imagine the following:

There is a country with an electricity market. In this country a group of very talented people, develop a technology that can offer infinite power at zero cost. Therefore these people could offer all the country energy for 1 cent of Euro, and it would still be profitable. In the market, this technology would leave out all the possible competitors, being the final market solution that all the country energy is generated by the infinite power plant at the cost of 1Euro cent.

This would imply that all the country energy would come from a single point, and this would cause massive voltage violations and overloads in the existing infrastructure, since the infrastructure is not as cutting edge as the new power plant.

The TSO has the duty of checking that the market solutions are feasible, moreover, they must ensure that in real time, the actual generation matches the demand exactly. That is why they must complement the market solution mismatch with other services.

The market mechanisms only looks at how to satisfy a demand at the minimum cost (given the bids, which have little to do with the real cost of the system), this is why the TSO must ensure that the economic solution also satisfies the real world constraints. Again the advantage of the market is that it allows more diverse players than the traditional scheme.

Any particular power plant is retributed by the amount of energy that it managed to sell on the market, plus the amount of ancillary services energy that it managed to sell to the TSO. I’ll discuss how to optimize these revenues for a power plant in another post.

Does the market hypothesis hold?

The efficient market hypothesis as a price optimizator only works if there is a large number of market players. Why?

If there is a limited number of market players, those will agree prices, or artificially put power plants in maintenance in order to benefit from the market demand. Therefore the market prices will not be the real prices but artificial ones.

On the other hand if there are many players offering and demanding energy, it is much more difficult that they agree prices with enough time. Furthermore, if there are more players the demand mismatch and the TSO corrections are likely to be lower since the market generation solution will be more distributed, and probably closer to the load points.

The more players in the game, the more efficient the market is. The opposite is also true.

Final remark

The proper planning of a power system will always produce a better price than any market mechanism, because it would minimize the waste of energy, and would maximize the performance of economic investments system wise. This solution is philosophically closer to socialism, where the individuals are surrogated to the system and there is no freedom to act or innovate (Our infinite power plant would not be accepted).

The market mechanism allows freedom and diversity, but requires the existence of many actors maximizing their own benefit without looking at other actors, to work. Still it will not minimize the energy waste or the energy price.