Moving Beyond Factsheets: Framework Comparison/Synergies/Consolidation

In this Do-a –thon we want to discuss and (further) develop a concept for open-source (OS FW) framework comparison.

Here is why we would love to do this:

  • We want to consolidate the OS FWs and gain trust in their results and usability!
  • We want to know which OS FW is the most applicable for which specific task
  • We want to investigate in the argument being constantly stated within the community that
  • Parallel code development can be reduced through the open source concept
  • Finally, we want to identify synergies and propose concepts on how to use them

This is how the workshop agenda could look like:

  1. Ask ourselves ”What are the requirements for a comparative scenario following the rules of Open Science?”

  2. Introduce factsheets and discuss/add criteria to be compared (follow-up on this thread Breakout Group: Transparency - Model comparison - Factsheets)

  3. “What else needs to be done to allow comparison (connectors, common ontology (follow –up on this thread https://forum.openmod.org/t/developing-a-common-ontology-for-energy-system-analysis/783)” from this we want to establish a regular Do-a-thon regarding comparisons and synergy identification

The results of this workshop can make an important contribution to open_MODEX- an ongoing three-year open science research project in which 5 OS FWs for energy system analysis are compared and synergies are derived.

We’ll be happy to receive your feedback and ideas!

5 Likes

@sar_b Are you going to compare the communities as well as the frameworks? That would be interesting too. I don’t know which projects are on your list, but most seem to have evolved different norms, audiences, and levels of formalism.

@robbie.morrison I agree and this is definitely foreseen in the project and yet could be part of the criteria discussion at OpenMod Arhus. Regarding open_MODEX the 5 FWs being compared are urbs, oemof, GENESYS-2, OSeMOSYS, Balmorel. They are represented through their respective hosting institution.

Koebler et al (2019) offer a very readable account of free software (often FOSS) development today and how it can be best supported. Smith (2018) writes specifically about scientific software in this context. The remaining references (below) are cited in the Koebler article and are classics in their own right. They provide useful background.

References

Eghbal, Nadia. (14 July 2016). Roads and bridges: the unseen labor behind our digital infrastructure. New York, USA: Ford Foundation. PDF version.

Koebler, Jason, Daniel Oberhaus, and Emanuel Maiberg (14 February 2019). The complicated economy of open source software. Motherboard.

Lerner, Josh and Jean Triole (March 2000). The simple economics of open source. USA: National Bureau of Economic Research.

Raymond, Eric S (2001). The cathedral and the bazaar : musings on Linux and open source by an accidental revolutionary. Sebastopol, California, USA: O’Reilly Media. ISBN 978-0-596-00108-7. Extended version of his essay originally published in 1998.

Smith, Nathaniel J (25 May 2018). The unreasonable effectiveness of investment in open-source infrastructure. njs blog.

Regarding points 2 and 3: I would argue that the definition of factsheets as consolidated ‘static sheets’ might be hard as there are a lot of heterogeneous communities with different (orthogonal) perspectives on the energy system leading to huge, confusing forms. (This is basically one of our lessons-learned from an attempt to define framework factsheets in open_eGo).
In SzenarienDB we are currently developing a collection of ontologies of energy scenarios and aim to define scenario factsheets as instantiations of parts of this ontology. You might be able to extend those ontologies for your purposes. There is also a proposal for an ontology session for the next workshop as a follow-up for a session on the previous workshop.
I am looking forward to your session :slight_smile:

Link to WS working document (password protected):
https://next.rl-institut.de/s/FtJd7WfMbFM5Pxk

Thank you for the fruitful session. I will upload the presentation and a summary. Therefore I would like to take a glance into the shared excel spreadsheet with the mathematical concepts and processes. The link is password protected. Could anyone of your WG add the PW to the shared document? Thanks!

Here our result graphics!
IMG_20190522_150404|666x500

Summary of the WS

Thanks again for participating in the Workshop. Everyone who may be interested in a follow-up can read through the summary.

Please, feel free to add your comments and (hopefully) continue the discussion in this forum thread!

Hi @sar_b   I didn’t attend this workshop unfortunately, but I skimmed through the summary you just posted. My one reaction is that while most energy system models are ultimately a mix of engineering and social characterization and some form of classical or psuedo optimization to navigate the available solution space, not all energy system models fit the GAMS | MathProg | python‑Pyomo | julia‑JuMP implementation paradigm. To expand:

  • engineering characterization is normally straightforward, usually convex linear if possible, falling back to mixed-integer linear (but rarely beyond for reasons of tractability)

  • social characterization refers to explicit endogenous decision‑taking and can include agent‑based modeling and social physics (see later)

  • solution criteria are usually some definition of least cost or of maximized value of trading, but other constructs like constraint programming and non‑conventional optimization can be deployed

The point I am trying to make is that the conventional approach: stated equation set → algebraic modeling formulation → interpreted language implementation via GAMS or MathProg or python‑Pyomo or julia‑JuMP is just one of many formulation and implementation paradigms. But clearly the dominant paradigm in our community. So your overview should, I believe, cater for paradigms that fall outside this particular set.

To give an example. I once designed and started coding an energy system model based on social physics after reading Pentland (2014) (also before the Cambridge Analytica scandal, I hasten to add).

So if a framework of frameworks is to be generally useful, it should be as encompassing as possible and span themes like social physics, constraint programming, semantic object‑object design (as opposed to the use of classes for programming convenience), agent‑based modeling, and system dynamics, as well as the classical technology‑rich least‑cost AML formulation. Also statistical modeling, stochastic optimization, and heuristic pseudo‑optimization. And, of course, hybrid implementations selected from the preceding.

I intentionally left out the field of machine‑learning because, arguably, ML is not a systems approach? Nor is ML guaranteed to be accurate, which might (and probably does) pose problems when applied to capacitated systems HTH.

References

Pentland Alex (2014). Social physics : how good ideas spread : the lessons from a new science. New York, USA: The Penguin Press. ISBN 978-1-59420-565-1.