On the Sustainability of Academic Software: The Case of Static Analysis Tools

Joenio Costa
Federal University of Bahia
Christina Chavez
Federal University of Bahia
Paulo Meirelles
Federal University of São Paulo


Software Sustainability

human, social, economic, environmental, technical

Robert Goodland. 2002. Sustainability: Human, Social, Economic and Environmental.

Software Technical Sustainability

Concerned with the long-term usage of software and its capacity to evolve with changing conditions and requirements.

B. Penzenstadler and H. Femmer. 2013. A Generic Model for Sustainability with Process- and Product-specific Instances.

Academic Software

Software developed to collect, process, or analyze research results published in the academic literature.

ALLEN, A. et al. 2017. Engineering academic software (dagstuhl perspectives workshop 16252).

James Howison and James D. Herbsleb. 2011. Scientific software production: incentives and collaboration.

How measure Academic Software Sustainability?

Three exploratory studies:

  1. Publicization
  2. Life cycle
  3. Recognition

Study 1

Publicization of Academic Software

url, license, source code, and download availability

found 60 academic software projects of static analysis

Inclusion criterion and keywords

Criteria Keywords
Mentions the project, software or tool tool or framework
Makes software available for download download or available
Identifies project URL http, https or ftp
Static analysis domain static analysis or parser

Inclusion criterion for extraction

Criteria Explanation
Identifiable Is it possible to identify a software project among the outputs of the article?
Available Can we find mention to the URL for download the software project?

Official online presence (RQ1.1)

from 60 academic software, 15 projects (25%) do not have an official online presence

Sofware available for download (RQ1.2)

from 60 academic software, 24 projects (40%) are not available for download

Source code available (RQ1.3)

from 36 academic software available for download, 34 projects has their source code available

Software license available (RQ1.4)

from 34 projects with the source code available, 13 did not provide any software license, and 21 used free software license

Study 2

Life cycle of Academic Software

evolution stage

CAPILUPPI, A. et al. 2007. Adapting the staged model for software evolution to free/libre/open source software.

Evolution stage of academic software (RQ2.1)

Evolution stage Academic Software %
Initial development 20 33%
Evolution 2 3%
Servicing 6 10%
Phaseout 3 5%
Closedown 24 40%
Unknown 5 8%

Study 3

Recognition of Academic Software

number and types of mentions

A mention means any occurrence of the name of the academic software in a scientific publication

Mentions to academic software (RQ3.1)

only 2 projects – GUIZMO and protopurity – were not found in searches carried out in ACM and IEEE

Uses of academic software (RQ3.1)

there are 124 “usage” mentions to a set of 26 projects

Contributions to academic software (RQ3.1)

there are 43 “contribute” mentions to a set of 17 projects, that is, only 28% of projects received source code contributions from studies after its initial publication

Discussion and Conclusions

Evolution of the recognition of academic software for static analysis published in ASE and SCAM

Growth of 38% per year in the number of mentions

from 60 academic software projects of static analysis studied, 13 projects (22%) did not have academic recognition

Influence of the life cycle stage on the recognition of academic software

large number of mentions for close-down projects – ESBMC, PARSEWeb, and TestEra – including recent publications between 2016 and 2017

Academic software projects are not yet recognized by Science as first-class citizens

Carole Goble. 2014. Better software, better research.



Future Work

  • Why some tools in closedown stage continue to be cited?
  • The results found regarding academic software recognition are similar to general software engineering publications?



Esta apresentação está disponível em:


(código-fonte: http://github.com/joenio/joenio.me)

Licença Creative Commons

Histórico de apresentações

Onde e quando esta apresentação foi realizada

  • 21 Setembro 2018, USP-ICMS, São Carlos, SP, CBSOFT SBES 2018