Quantifying the benefits of open energy modelling?

Modelling Open modelling
1

Dear openmod community,

This is kind of a naive, and most certainly a wide question, but I was wondering: how can we qualify and quantify the benefits of open energy modelling ? In other words: how to assess that the open energy modelling we put in practice is usable and useful ? Do you have methods or examples of such work ? For instance:

  • Surveys towards the models developers / users / results recipients ?
  • Criteria on the tools themselves, such as the work of Cao et al. [1] or Groissbock [2] ?
  • …?

To be precise, I am not asking about open energy modelling interests, limits or good practices: I think I now have a pretty good understanding of these after the truly interesting presentations and exchanges we had in Berlin at the openmod workshop and on the forum; plus thanks to the reading of articles such as [3-6]. To rephrase it once again, my question would be: how to quantify the identified interests of such good practices ?

Thanks a lot !

References:

  1. Cao, Karl-Kiên, Felix Cebulla, Jonatan J Gómez Vilchez, Babak Mousavi, and Sigrid Prehofer (28 September 2016). “Raising awareness in model-based energy scenario studies — a transparency checklist”. Energy, Sustainability and Society. 6 (1): 28. ISSN 2192-0567. doi:10.1186/s13705-016-0090-z. Open access.

  2. Groissböck, Markus (1 March 2019). “Are open source energy system optimization tools mature enough for serious use?”. Renewable and Sustainable Energy Reviews. 102: 234–248. ISSN 1364-0321. doi:10.1016/j.rser.2018.11.020. Closed access.

  3. Pfenninger, Stefan, Joseph F DeCarolis, Lion Hirth, Sylvain Quoilin, and Iain Staffell (February 2017). “The importance of open data and software: is energy research lagging behind?”. Energy Policy. 101: 211–215. ISSN 0301-4215. doi:10.1016/j.enpol.2016.11.046. Open access.

  4. Bazilian, Morgan, Andrew Rice, Juliana Rotich, Mark Howells, Joseph DeCarolis, Stuart Macmillan, Cameron Brooks, Florian Bauer, and Michael Liebreich (1 October 2012). “Open source software and crowdsourcing for energy analysis”. Energy Policy. 49: 149–153. ISSN 0301-4215. doi:10.1016/j.enpol.2012.06.032. Working draft.

  5. Morrison, Robbie (April 2018). “Energy system modeling: public transparency, scientific reproducibility, and open development”. Energy Strategy Reviews. 20: 49–63. ISSN 2211-467X. doi:10.1016/j.esr.2017.12.010. Open access.

  6. Pfenninger, Stefan, Lion Hirth, Ingmar Schlecht, Eva Schmid, Frauke Wiese, Tom Brown, Chris Davis, Matthew Gidden, Heidi Heinrichs, Clara Heuberger, Simon Hilpert, Uwe Krien, Carsten Matke, Arjuna Nebel, Robbie Morrison, Berit Müller, Guido Pleßmann, Matthias Reeg, Jörn C Richstein, Abhishek Shivakumar, Iain Staffell, Tim Tröndle, and Clemens Wingenbach (2017). “Opening the black box of energy modelling: strategies and lessons learned”. Energy Strategy Reviews. 19: 63–71. ISSN 2211-467X. doi:10.1016/j.esr.2017.12.002. Open access.

PS : do not hesitate to ask me precision if this is not clear, or to ask me to change tags / category if necessary (I was not too sure about where to ask this).

2 Likes
2

The jury is still out but not for long?

First I would like to throw energy system data into the mix. The gains from sharing and curating open data collectively are self‑evident. And, as will be seen, it is often hard to split models from data in any case.

This blog looks the relative merits of open versus non‑open energy system modeling in the context of public policy development. It sidesteps other considerations to focus on the “efficacy” of the various approaches in use today.1 And I would say that, under current practices, there is no clear conclusion as to which approach performs best.

Established policy models are often constituted as excludable economic clubs, typically with five figure annual membership fees, very significant contracts from public institutions, and dedicated inputs from publicly-funded science. Examples include the IEA World Energy Model (WEM), MARKAL/TIMES models, WEC models, and the GTAP database and software. And these clubs usually cover code and data in roughly equal measure.

There are relatively few single institution closed models that are also influential. The key example would be PRIMES from NTUA, used by the European Commission.2,3,4 These projects are more akin to proprietary software development than the consortiums just listed. And their documentation and drill‑down transparency tends to be patchy at best. The German government is unusual in that it makes wide use of single institution models from both research agencies and dedicated policy consultancies.

The US government National Energy Modeling System (NEMS) is one of the few examples of a national government maintaining a substantial in‑house project. And although technically the core code is public domain, it is not possible for external parties to build and run NEMS — nor is that use case intended or supported.

On the other side of the fence, one sees the emergence of robust open modeling projects making substantial efforts to engage in good software practices. Retrospectively in the case of OSeMOSYS — which has forked all over the place since its early beginnings. And from the outset in the case of oemof. These are the only two open projects I am going to mention by name, so don’t treat their appearance here as an endorsement. See Muschner (2020) for a comparison of the two frameworks.

Another consideration is the uptake of open models by small or less developed countries that cannot easily sign on for the club or closed approaches. See my comments here for some specific examples.

The one area where open source development is lagging well behind is optimization solvers. The open source GLPK and cbc solvers are clearly inferior to their commercial counterparts, such as Gurobi and CPLEX, by a wide margin.

Also interesting to note that many of the previous club projects are becoming open incrementally. Knowledge bases from the World Bank Group provide one example, for instance energydata.info. In addition, the European Commission is steadily migrating from closed models like PRIMES to open models like Dispa‑SET. But the Commission is also funding the “open book” METIS suite of models whereby external parties undertake the core development and provide the Commission with the legally encumbered source code to run in‑house 5 — not a practice I warm to.

That trend toward openness is often tempered by withholding or overpricing necessary workflow tooling, post-processing software, databases, and other essential components. Which raises questions about the underlying motivations for publishing the core code in the first place — is it merely a response to public pressure for better optics?

The bulk of open energy system model development occurs within universities and there are still unmet challenges to improve the quality of academic software. The Research Software Engineering (RSE) initiative is one entity pursuing this issue and the adoption of open source development practices is a key theme.

To conclude, I deliberately replaced “benefits” with “efficacy” to narrow the discussion. And even so, it is hard to say whether club and closed development outperforms open development or vice versa. But what is clear is that energy modeling for policy support is heading open for many reasons, most of which do not involve development efficacy. Of course I am happy with that shift and the underlying logic. And I believe the gains from open development and community curation more generally — as evidenced by projects like the Linux kernel and OpenStreetMap — will arrive quite soon. And when they do, there will be no turning back from the advantages that accrue from collective open development beyond some indeterminate tipping point.

Feel free to add your views and reactions! R

Notes

  1. Wiktionary defines “efficacy” as the “ability to produce a desired effect under ideal testing conditions”.

  2. European Commission (23 November 2016). Modelling tools for EU analysis. Climate Action — European Commission. Accessed 14 November 2020.

  3. The last model description of PRIMES to my knowledge is: E3MLab (April 2015). PRIMES model 2013‑2014: detailed model description. Athens, Greece: E3MLab/ICCS at National Technical University of Athens. Publication date from PDF metadata.

  4. Capros, Pantelis, Maria Kannavou, Stavroula Evangelopoulou, Apostolos Petropoulos, Pelopidas Siskos, Nikolaos Tasios, Georgios Zazias, and Alessia DeVita (1 November 2018). “Outlook of the EU energy system up to 2050: the case of scenarios prepared for European Commission’s “clean energy for all Europeans” package using the PRIMES model”. Energy Strategy Reviews. 22: 255–263. ISSN 2211-467X. doi:10.1016/j.esr.2018.06.009. Closed access. USD 31.50 to purchase.

  5. Barberi, Paul, Paul Khallouf, Tobias Bossmann, and Laurent Fournié (May 2020). Introduction to METIS models. Brussels, Belgium: European Commission. Directorate-General for Energy.

References

Muschner, Christoph (2020). An Open Source Energy Modelling Framework Comparison of OSeMOSYS and oemof (MSc). Stockholm: KTH.

2 Likes
3

For those who need numbers, an earlier posting on code resue within our community:

4

Thank you very much for this thorough and interesting overview of the current situation regarding the open energy modelling and data initiatives efficacy.

I agree that the “gains from sharing and curating open data collectively are self‑evident”, as well as other gains from open energy modelling practices: long-term availability of the source code, … But from another standing point (probably the one of the devil’s advocate), if the improved reproducibility / credibility / cooperation enabled by open practices seems obvious, has it been proven ?
I think we have plenty of arguments and experiences underlining the gains from open practices compared to private ones : for instance if the IEA reports are available resources, its data and if I am not mistaken, its tools are not open and rarely presented, which do not enable a full understanding of the results and interpretations without any access to the detailed assumptions.

Going further with the idea of qualifying and quantifying the benefits of open energy modelling, we could imagine (or has it already been undertaken ?) sort of a social experiment where we present the results from private and open processes to an uninvolved third party and gather its feedback. The third part could be an energy modeller or a policy maker for instance (expert and non-expert reader in the sense of Cao et al. [1]).

Finally regarding the efficacy, perhaps it can be interesting to differentiate:

  • the performance of the tool or the method, i.e. the efficacy in the run time, with a clear advantage for private optimization solvers,
  • the efficacy for the reader, i.e. the efficacy in the design time (scenario construction) and in the results understanding, where open practices seem relevant and useful.

Here as well, feel free to add your views and reactions!

5

The International Energy Agency (IEA) executive summary (IEA November 2020) is free‑of‑charge to download but remains under full copyright. The underlying report costs €120 per PDF (and making and distributing copies is breach of copyright and actionable). The World Energy Model (WEM) is closed source but documented (IEA January 2020). At least some of the data used remains subject to IEA Policies and Measures Databases (PAMS) terms and conditions and is available to participating government agencies on those terms but not to citizens.

The IEA has certainly been strong in advocating for rapid decarbonization (after earlier issues involving US political influence, see wikipedia). For instance, IEA’s executive director, Fatih Birol recently opined in April this year (Ambrose 2020):

The plunge in demand for nearly all major fuels is staggering, especially for coal, oil and gas. Only renewables are holding up during the previously unheard of slump in electricity use.

The point is not accuracy or even efficacy, but rather that public interest analysis of this nature should not be undertaken behind closed doors using private information.

References

Ambrose, Jillian (30 April 2020). “Covid-19 crisis will wipe out demand for fossil fuels, says IEA”. The Guardian. London, United Kingdom. ISSN 0261-3077.

International Energy Agency (November 2020). World energy outlook — Executive summary. Paris, France: IEA Publications.

IEA (16 January 2020). World Energy Model documentation — 2019 version. Paris, France: International Energy Agency (IEA). Publication date from PDF metadata.

6

A very interesting topic. Thank you for a lively discussion! A few thoughts related to development in general.

Going further with the idea of qualifying and quantifying the benefits of open energy modelling, we could imagine (or has it already been undertaken ?) sort of a social experiment where we present the results from private and open processes to an uninvolved third party and gather its feedback.

It was a brilliant talk during FOSDEM 2020 about benefits of the open source in general and General Public Licenses in particular for business. Frank Karlitschek has mentioned something similar to such an experiment comparing development of the proprietary ownCloud and an open source Nextcloud (40:00 to 42:00 of the talk recording). One of the key benefits revealed was the developers’ attitude. They were clearly more happy working in the open source project.

I have checked ScienceDirect to find that the open source continues to gain popularity among researchers:

sciencedirect_nov2020
sciencedirect_nov20201500×1500 129 KB

Joined with developers’ happiness, we can hope that this quantity will sooner or later result in quality, can’t we? :wink:

1 Like
7

UKERC model survey for the United Kingdom

The United Kingdom Energy Research Centre (UKERC) is currently traversing the question of the relative merits of closed versus open energy models for public policy analysis. On a recent blog, Strachan and Pei‑Hao (2020) present the following classification:

  • open description models: concise methodological summary, outline documentation, and link to outputs and applications
  • open access models: as [above] plus full documentation, data sets, and a user group for access and shared responsibility for model development
  • open source models: fully transparent and accessible models available for any user to download and apply

The UKERC is also, as of December 2020, undertaking a comprehensive survey of energy system models used within the United Kingdom. The resulting report should make interesting reading.

Sources:

  • Strachan, Neil and Pei‑Hao Li (9 December 2020). Energy models and transparency. United Kingdom Energy Research Centre (UKERC). London, United Kingdom. Blog.
2 Likes
8

European Union open source strategy

The European Union introduced its open source strategy in October 2020 with key objective three being to:

encourage sharing and reuse of software and applications, as well as data, information and knowledge

Moreover, the implementation of the strategy will be guided by six principles:

think open, transform, share, contribute, secure, and stay in control

The reaction by the Free Software Foundation Europe (FSFE) is also listed below.

Sources:

1 Like
9

Great thread!

I can contribute an analysis and article I did some years ago applying “the transparency checklist” by Cao et al (2016) to an energy system modelling study. The results are used for the development of the Open Energy Platform.

End of this year I plan to repeat the experiment to measure the progress.

2 Likes
10

Hi all,

This is an interesting thread and I believe I can add something to the discussion.

I have been in the power systems consultancy business for some time now and I have had the chance to write a couple of business cases in favor of open software architectures (not necessarily open source).

The core of those business cases is the inability to customize software for:

  1. Automating work (#1 priority in power system planning departments today due to the massive amounts of data to process)
  2. Customizing the simulations.
  3. Coupling software packages (this is related to 1 and 2)

Those three needs require a ridiculous amount of hacking with closed source models (i.e. Plexos + PSSe). The hacking may save the day but creates a technical debt that becomes impossible to pay in the long run. When working in diverse projects with large amounts of data, you need to have off-the-shelf libraries that are easy to combine in order to minimize the amount of software writing, data checking, excel wrestling and software button clicking.

Those business cases (that I unfortunately cannot disclose) show that having an ecosystem of tools that share the same data model, save several million € in a 5 years investment plan. Those savings come from the several M€ in labor not wasted in excel sheets and hacky scripts that cannot become tools without rewriting.

So IMHO, open source is great but I would not even consider an open source program that is not written as a library and somewhat documented. Software needs to follow good practices. This includes to have the proper interface separation, which means that the calculation core should be able to operate without the input/output mechanisms and GUI. In that way anyone can recycle that piece of knowledge easily.

So the quantification is easy: measure how much it takes with the closed source tools and then how much it take when using open source tools adapted to speak to each other. This effort typically takes 1.5~2 years to make and pays itself already during the development if you follow a bit-size approach to development (Agile?).

Hope it helps,
Santiago

3 Likes
11

The issue of open source methods being adopted by electricity sector system operators is slowly gaining attention in Europe. Oleg Tšernobrovkin from Estonian TSO for electricity and gas, Elering, recently presented their e‑Gridmap project to provide customers with planning information without the need for dedicated requests. His presentation (01:05:48) was made to a Renewables Grid Initiative hosted seminar on 17 December 2020. This YouTube video is timestamped (00:44:15) at a 2 minute question from me on whether Elering would consider open sourcing the underlying code. The answer is quite encouraging yet also somewhat revealing. In addition, the LF Energy (LFE) project is also promoting open standards and implementations, firstly within the European electricity sector.

2 Likes
12

Hi Robbie,

I could not attend to that meeting but I was aware of it. Indeed the Estonians are doing the right thing.

The issue with TSO/DSO’s adopting open source is the lack of support, or in other words, the liability. Electrical companies have a huge customer mindset (I just buy, I don’t think -> I am not responsible of the results, the program manufacturer is) This mindset is incredibly hard to fight. What the Open source culture tells you is: “You are free, but also responsible of the results” and that is harder to sell, but times are changing.

Likewise there is a massive business opportunity related to this and Open Source. We can see that in other domains (RedHat, Canonical, etc.). That is why the LFE exists, and also why it only accepts funded open source projects; because of the support and guarantees.

Regardless, we are in a very exciting moment to be in this industry because all the changes are forcing it to become innovative, which is something that didn’t happen in several generations.

3 Likes
13

Energy economics professor Claudia Kemfert, DIW, Berlin, Germany, after saying the benefits of open source were hard to measure empirically, expressed it this way:

So far, Open Source has done us no harm, quite the contrary.

Published interview:

2 Likes
14

@SanPen Thank you for your business insights. I agree with you that software need to follow good practices, only making a code open is not sufficient. Would you have examples in mind of codes written as a libraries (open or not) which you consider good? Same question for bad and what especially what makes them bad (feature or lack of feature) in your opinion?

I attach an article I found interesting as a collection of good practices for developing libraries

15

I would like to add another aspect to this discussion that already came up at some points and is strongly connected to open-source modelling:
The benefit of open collaborative development

From my personal experience, the decision making process in larger communities is sometimes more difficult (takes more time, needs better documentation of arguments…) but leads to better results and solid consensus.
Does anybody knows literature or analysis that has a look at this topic?

1 Like
16

Hi Pierre,

For instance, I can think of OSeMOSYS as as bad example. If the program is what I have just linked, there are: No modules, global variables, all in one file, and probably more flaws. I know OSeMOSYS is popular, and probably it is popular because of the lack of structure which makes it straight forward at the beginning but a nightmare in the long term. I have worked commercially with such GAMS-like optimization programs and they should be banned altogether.

I can think of Matpower as a better example: There are plenty of functions, each function does one thing, etc. But Matpower is flawed in the sense that the results are feed back to the inputs (non-thread-safe: very bad) and the information flows in loops instead of in a straight line. Siemens’ PSSe works like that too and it is horrendous.

I’m patting my own back here, but I have taken great care to not to fall into bad practices while developing this. And I did so, to overcome the issues mentioned in the previous packages. I re-wrote the engine of GridCal up to 6 times now, to be every time better, more formal, more maintainable etc. The price is a program that is very hard to dig into if you don’t know why those decisions were made, but the API is dead simple: Create grid, add generator, etc…

I hope it helps

17

Thank you for your thourough answer @SanPen !

18

A couple of more general contributions to the debate — well beyond that of public‑interest analysis.

Consulting firm McKinsey highlight the link between open source development and favorable company status (Srivastava et al 2020).

The Linux Foundation (2020) list six sectors utilizing open collaboration and thus “enabling user‑centered innovation, achieving faster development cycles, time to market, and increased interoperability and cost savings”. Eight LF Energy projects covering the energy sector are briefly outlined on pages 19–22. (Disclosure: I contribute to an LF Energy working group on data architecture.)

References

Linux Foundation (September 2020). Software-defined vertical industries: transformation through open source. San Francisco, California, USA: Linux Foundation. URL provided is the landing page with registration necessary to download the actual document.

Srivastava, Shivam, Kartik Trehan, Dilip Wagle, and Jane Wang (20 April 2020). How software developers can drive business growth. McKinsey.