Open Science Best Practices in Data Science and Artificial Intelligence
DOI:
https://doi.org/10.52825/cordi.v1i.299Keywords:
FAIR, Reproducible Research, Open ScienceAbstract
In the past years, scientific research in Data Science and Artificial Intelligence has witnessed vast progress. The number of published papers and digital objects (data, code, models) is growing exponentially. However, not all of these research artifacts are findable, accessible, interoperable and reusable (FAIR), contributing to a rather low level of reproducibility of the experimental findings reported in scholarly publications (reproducibility crisis). In this paper, we focus on Data Science and Artificial Intelligence Open Science best practices, i.e., a set of recommendations that eventually contribute to the management and development of digital artefacts that are as FAIR as possible. While several guidelines exist, we add best practices for the FAIR collection, processing, storing and sharing of scholarly findings via Research Knowledge Graphs. The final list of recommendations will be openly available on the NFDI4DS webpage as an interactive web application.
Downloads
References
M. Wilkinson, M. Dumontier, I. Aalbersberg, et al., “The FAIR Guiding Principles for Scientific Data Management and Stewardship,” Scientific Data, vol. 3, no. 160018, Mar. 2016. DOI: 10.1038/sdata.2016.18.
A. Belz, S. Agarwal, A. Shimorina, and E. Reiter, “A Systematic Review of Reproducibility Research in Natural Language Processing,” in Proc. of the 16th Conf. of the Europ. Chap. of the Assoc. for Comput. Ling., Association for Computational Linguistics, May 2021, pp. 381–393. DOI: 10.18653/v1/2021.eacl-main.29.
M. Hutson, “Artificial Intelligence Faces Reproducibility Crisis,” Science, vol. 359, no. 6377, pp. 725–726, Feb. 2018. DOI: 10.1126/science.359.6377.725.
F. D. Maurizio, C. Paolo, and J. Dietmar, “Are We Really Making Much Progress? AWorrying Analysis of Recent Neural Recommendation Approaches,” in Proc. of the 13th Assoc. for Comput. Machinery (ACM) Conf. on Recomm. Systems, Copenhagen, Denmark: Association for Computing Machinery, Sep. 2019, pp. 101–109. DOI: 10.1145/3298689.3347058.
L. Rupprecht, J. C. Davis, C. Arnold, Y. Gur, and D. Bhagwat, “Improving Reproducibility of Data Science Pipelines Through Transparent Provenance Capture,” in Proc. VLDB Endow., VLDB Endowment, Aug. 2020, pp. 3354–3368. DOI: 10.14778/3415478.3415556.
R. B. Yousuf, S. Biswas, K. K. Kaushal, et al., “Lessons from Deep Learning Applied to Scholarly Information Extraction: What Works, What Doesn’t, and Future Directions,” 2022. arXiv: 2207.04029 [cs.IR].
K. Gregory, P. Groth, A. Scharnhorst, and S. Wyatt, “Lost or Found? Discovering Data Needed for Research,” Harvard Data Science Review, vol. 2, no. 2, May 2020. DOI: 10.1162/99608f92.e38165eb.
A. Radford and K. Narasimhan, “Improving Language Understanding by Generative Pre-Training,” 2018.
OpenAI. “OpenAI.” Accessed: 2023-04-09. (2023), [Online]. Available: https://openai.com.
A. Lucic, M. Bleeker, S. Bhargav, et al., “Towards Reproducible Machine Learning Research in Natural Language Processing,” in Proc. of the 60th Annual Meeting of the Assoc. for Comput. Ling.: Tutorial Abstracts, Association for Computational Linguistics, Jun. 2022, pp. 7–11. DOI: 10.18653/v1/2022.acl-tutorials.2.
GO FAIR. “M4M Workshop.” Accessed: 2023-04-08. (2018), [Online]. Available: https://www.go-fair.org/events/m4m-workshop/.
GO FAIR. “M4M #2: Preclinical trials + M4M #3: Funders.” Accessed: 2023-04-08. (2019), [Online]. Available: https://www.go-fair.org/events/m4m-2-preclinical-trialsm4m-3-funders/.
GO FAIR. “The Second GO FAIR Workshop for the German Research Community.” Accessed: 2023-04-08. (2018), [Online]. Available: https://www.go-fair.org/2018/10/08/on-the-road-to-fair/.
Go FAIR. “The 3rd Germany GOes FAIR Workshop for the German Research Community.” Accessed: 2023-04-08. (2019), [Online]. Available: https://www.go-fair.org/2019/06/04/3rd-germany-goes-fair-workshop-report/.
OpenAIRE, R. Europe, FAIRsFAIR, and EOSC-hub. “Services to Support FAIR Data.” Accessed: 2023-04-08. (2019), [Online]. Available: https://eosc-portal.eu/events/workshop-series-services-support-fair-data.
NeurIPS. “NeurIPS 2021 Paper Checklist Guidelines.” Accessed: 2023-04-08. (2021), [Online]. Available: https://neurips.cc/Conferences/2021PaperInformation/PaperChecklist.
J. Pineau, P. Vincent-Lamarre, K. Sinha, et al., “Improving Reproducibility in Machine Learning Research (A Report from the Neurips 2019 Reproducibility Program),” Journal of Machine Learning Research, vol. 22, no. 164, pp. 1–20, Jan. 2021.
A. Rogers, T. Baldwin, and K. Leins, “‘Just What Do You Think You’re Doing, Dave?’ A Checklist for Responsible Data Use in NLP,” in Findings of the Assoc. for Comput. Ling.: EMNLP 2021, Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 4821–4833. DOI: 10.18653/v1/2021.findings-emnlp.414.
J. Dodge, S. Gururangan, D. Card, R. Schwartz, and N. A. Smith, “Show Your Work: Improved Reporting of Experimental Results,” in Proc. of the 2019 Conf. on Empirical Methods in NLP and the 9th Intern. Joint Conf. on NLP (EMNLP-IJCNLP), Association for Computational Linguistics, Nov. 2019, pp. 2185–2194. DOI: 10.18653/v1/D19-1224.
The Turing Way Community. “The Turing Way: A Handbook for Reproducible, Ethical and Collaborative Research (1.0.2).” Accessed: 2023-04-08. (2021), [Online]. Available: https://doi.org/10.5281/zenodo.7625728.
NAACL. “NAACL 2021 Reproducibility Checklist.” Accessed: 2023-04-08. (2021), [Online]. Available: https://2021.naacl.org/calls/reproducibility-checklist/.
ACL. “ARR Responsible NLP Research Checklist.” Accessed: 2023-04-08. (2021), [Online]. Available: https://aclrollingreview.org/responsibleNLPresearch/.
Nature. “Reporting Standards and Availability of Data, Materials, Code and Protocols.” Accessed: 2023-04-08. (2023), [Online]. Available: https://www.nature.com/natureportfolio/editorial-policies/reporting-standards.
A. Spirling, “Why Open-source Generative AI Models are an Ethical Way Forward for Science,” Nature, vol. 616, no. 413, Apr. 2023. DOI: 10.1038/d41586-023-01295-4.
OpenAI, “GPT-4 Technical Report,” 2023. arXiv: 2303.08774 [cs.CL].
T. L. Scao, A. Fan, C. Akiki, et al., “BLOOM: A 176B-Parameter Open-Access Multilingual Language Model,” 2023. arXiv: 2211.05100 [cs.CL].
T. Gebru, J. Morgenstern, B. Vecchione, et al., “Datasheets for Datasets,” Communications of the ACM, vol. 64, no. 12, pp. 86–92, Dec. 2021. DOI: 10.1145/3458723.
A. L. Lamprecht, L. Garcia, M. Kuzak, et al., “Towards FAIR Principles for Research Software,” Data Science, vol. 3, no. 1, pp. 37–59, Jun. 2020. DOI: 10.3233/DS-190026.
M. Barker, N. P. Chue Hong, D. S. Katz, et al., “Introducing the FAIR Principles for Research Software,” Scientific Data, vol. 9, no. 622, Oct. 2022. DOI: 10.1038/s41597-022-01710-x.
G. Rehm, “The Language Resource Life Cycle: Towards a Generic Model for Creating, Maintaining, Using and Distributing Language Resources,” in Proc. of the 10th Intern. Conf. on Lang. Resources and Evaluation (LREC’16), European Language Resources Association (ELRA), May 2016, pp. 2450–2454.
Docker. “Docker.” Accessed: 2023-04-14. (2023), [Online]. Available: https://www.docker.com.
ELG. “European Language Grid.” Accessed: 2023-04-14. (2023), [Online]. Available: https://live.european-language-grid.eu.
S. Auer, V. Kovtun, M. Prinz, A. Kasprzik, M. Stocker, and M. E. Vidal, “Towards a Knowledge Graph for Science,” in Proc. of the 8th Intern. Conf. on Web Intelligence, Mining and Semantics, ser. WIMS ’18, Novi Sad, Serbia: Association for Computing Machinery, Jun. 2018, pp. 1–6. DOI: 10.1145/3227609.3227689.
M. Y. Jaradeh, A. Oelen, K. E. Farfar, et al., “Open Research Knowledge Graph: Next Generation Infrastructure for Semantic Scholarly Knowledge,” in Proc. of the 10th Inter. Conf. on Knowledge Capture, Marina Del Rey, CA, USA: Association for Computing Machinery,
Sep. 2019, pp. 243–246. DOI: 10.1145/3360901.3364435.
R. M. Kinney, C. Anastasiades, R. Authur, et al., “The Semantic Scholar Open Data Platform,” 2023. arXiv: 2301.10140 [cs.DL].
C. Bless, I. Baimuratov, and O. Karras, “SciKGTeX – A LATEXPackage to Semantically Annotate Contributions in Scientific Publications,” 2023. arXiv: 2304.05327 [cs.DL].
Streamlit. “Streamlit.” Accessed: 2023-04-13. (2023), [Online]. Available: https://streamlit.io.
NFDI4DS. “NFDI4DataScience.” Accessed: 2023-04-12. (2023), [Online]. Available: https://www.nfdi4datascience.de.
Downloads
Published
How to Cite
Conference Proceedings Volume
Section
License
Copyright (c) 2023 Ekaterina Borisova, Raia Abu Ahmad, Georg Rehm
This work is licensed under a Creative Commons Attribution 4.0 International License.
Accepted 2023-06-29
Published 2023-09-07
Funding data
-
Deutsche Forschungsgemeinschaft
Grant numbers NFDI4DS (no. 460234259)
