Towards a Concept for Building a Big Data Architecture with Microservices
DOI:
https://doi.org/10.52825/bis.v1i.67Keywords:
Big Data, Microservice, Success Factors, Software Design, Software ArchitectureAbstract
Microservices and Big Data are renowned hot topics in computer science that have gained a lot of hype. While the use of microservices is an approach that is used in modern software development to increase flexibility, Big Data allows organizations to turn today’s information deluge into valuable insights. Many of those Big Data architectures have rather monolithic elements. However, a new trend arises in which monolithic architectures are replaced with more modularized ones, such as microservices. This transformation provides the benefits from microservices such as modularity, evolutionary design and extensibility while maintaining the old monolithic product’s functionality. This is also valid for Big Data architectures. To facilitate the success of this transformation, there are certain beneficial factors. In this paper, those aspects will be presented and the transformation of an exemplary Big Data architecture with somewhat monolithic elements into a microservice favoured one is outlined.
Downloads
References
M. Volk, D. Staegemann, and K. Turowski, “Big Data,” in Handbuch Digitale Wirtschaft, ser. Springer Reference Wirtschaft, T. Kollmann, Ed., Wiesbaden: Springer Fachmedien, 2020, pp. 1–18, ISBN: 978-3-658-17345-6. DOI: 10.1007/978-3-658-17345-6_71-1.
F. X. Diebold, “On the Origin(s) and Development of the Term ’Big Data’,” Social Science Research Network, Rochester, NY, SSRN Scholarly Paper ID 2152421, Sep. 2012. DOI: 10.2139/ssrn.2152421.
O. M¨ uller, M. Fay, and J. vom Brocke, “The Effect of Big Data and Analytics on Firm Performance: An Econometric Analysis Considering Industry Characteristics,” Journal of Management Information Systems, vol. 35, no. 2, pp. 488–509, Apr. 2018, ISSN: 0742-1222, 1557-928X. DOI: 10.1080/07421222.2018.1451955.
M. Volk., D. Staegemann., S. Bosse., R. H¨ausler., and K. Turowski., “Approaching the (big) data science engineering process,” in Proceedings of the 5th International Conference on Internet of Things, Big Data and Security - Volume 1: IoTBDS,, INSTICC, SciTePress, 2020, pp. 428–435. DOI: 10.5220/0009569804280435.
D. Staegemann, M. Volk, C. Daase, and K. Turowski, “Discussing relations between dynamic business environments and big data analytics,” Complex Syst. Informatics Model. Q., vol. 23, pp. 58–82, 2020. DOI: 10 . 7250 / csimq . 2020 - 23 . 05. [Online]. Available: https://doi.org/10.7250/csimq.2020-23.05.
A. Freymann, F. Maier, K. Schaefer, and T. B¨ohnel, “Tackling the Six Fundamental Challenges of Big Data in Research Projects by Utilizing a Scalable and Modular Architecture,” in 5th International Conference on Internet of Things, Big Data and Security, IoTBDS 2020. Proceedings, 2020, pp. 249–256, ISBN: 978-989-758-426-8.
L. Hung and F. Jackie, Hype Cycle for Emerging Technologies, 2012. [Online]. Available: https://www.gartner.com/en/documents/2100915/hype- cycle- for- emerging-technologies-2012 (visited on 02/14/2021).
A. Parlina, K. Ramli, and H. Murfi, “Theme Mapping and Bibliometrics Analysis of One Decade of Big Data Research in the Scopus Database,” Information, vol. 11, no. 2, p. 69, Feb. 2020, Number: 2 Publisher: Multidisciplinary Digital Publishing Institute. DOI: 10.3390/info11020069.
W. L. Chang and N. Grady, “NIST Big Data Interoperability Framework: Volume 1, Definitions,” Oct. 2019, Last Modified: 2020-01-07. [Online]. Available: https://www.nist.gov / publications / nist - big - data - interoperability - framework - volume - 1 -definitions.
T. Breur, “Statistical Power Analysis and the contemporary “crisis” in social sciences,” Journal of Marketing Analytics, vol. 4, no. 2, pp. 61–65, Jul. 2016, ISSN: 2050-3326. DOI: 10.1057/s41270-016-0001-3.
N. Khan, M. Alsaqer, H. Shah, G. Badsha, A. A. Abbasi, and S. Salehian, “The 10 Vs, Issues and Challenges of Big Data,” in Proceedings of the 2018 International Conference on Big Data and Education, ser. ICBDE ’18, New York, NY, USA: Association for Computing Machinery, Mar. 2018, pp. 52–56, ISBN: 978-1-4503-6358-7. DOI: 10.1145/3206157.3206166.
A. Gandomi and M. Haider, “Beyond the hype: Big data concepts, methods, and analytics,” International Journal of Information Management, vol. 35, no. 2, pp. 137–144, Apr. 2015, ISSN: 0268-4012. DOI: 10.1016/j.ijinfomgt.2014.10.007.
A. Gardiner, C. Aasheim, P. Rutner, and S. Williams, “Skill Requirements in Big Data: A Content Analysis of Job Advertisements,” Journal of Computer Information Systems, vol. 58, no. 4, pp. 374–384, Oct. 2018, ISSN: 0887-4417. DOI: 10.1080/08874417.2017.1289354.
C. Avci, B. Tekinerdogan, and I. Athanasiadis, “Software architectures for big data: A systematic literature review,” Big Data Analytics, vol. 5, Aug. 2020. DOI: 10.1186/s41044-020-00045-1.
A. M. Fern´andez, D. Gutie´ rrez-Avile´s, A. Troncoso, and F. Mart´ınez–A´ lvarez, “Automated Deployment of a Spark Cluster with Machine Learning Algorithm Integration,” Big Data Research, vol. 19-20, p. 100 135, Mar. 2020, ISSN: 2214-5796. DOI: 10.1016/j.bdr. 2020.100135.
A. Tiwana, “Chapter 5 - Platform Architecture,” in Platform Ecosystems, A. Tiwana, Ed., Boston: Morgan Kaufmann, Jan. 2014, pp. 73–116. DOI: 10.1016/B978-0-12-408066-9.00005-9.
D. Namiot and M. Sneps-Sneppe, “On micro-services architecture,” International Journal of Open Information Technologies, vol. 2, p. 4, 2014.
C. Fan and S. Ma, “Migrating monolithic mobile application to microservice architecture: An experiment report,” in 2017 IEEE International Conference on AI Mobile Services (AIMS), Jun. 2017, pp. 109–112. DOI: 10.1109/AIMS.2017.23.
A. Bucchiarone, N. Dragoni, S. Dustdar, S. Larsen, and M. Mazzara, “From monolithic to microservices: An experience report from the banking domain,” IEEE Software, vol. 35, pp. 50–55, May 2018. DOI: 10.1109/MS.2018.2141026.
P. Karwatka, Monolithic architecture vs microservices, Jan. 2020. [Online]. Available: https://divante.com/blog/monolithic- architecture- vs- microservices/ (visited on 02/14/2021).
F. Ponce Mella, G. M´arquez, and H. Astudillo, “Migrating from monolithic architecture to microservices: A rapid review,” in Proceedings of the 38th International Conference of the Chilean Computer Science Society, Sep. 2019.
F. Martin and L. James, Microservices - a definition of this new architectural term, Mar. 2014. [Online]. Available: https://martinfowler.com/articles/microservices.html (visited on 02/14/2021).
M. Amundsen and M. Mclarty, Microservice Architecture: Aligning Principles, Practices, and Culture. Sebastopol, CA: O’Reilly Media, Inc, USA, Aug. 2016, ISBN: 978-1-4919-5625-0.
P. Drews, I. Schirmer, B. Horlach, and C. Tekaat, “Bimodal Enterprise Architecture Management - The Emergence of a New EAM Function for a BizDevOps-based fast IT,” Oct. 2017. DOI: 10.1109/EDOCW.2017.18.
M. E. Conway, “How do committees invent,” design organization criteria, 1968.
D. Faitelson, R. Heinrich, and S. Tyszberowicz, “Functional Decomposition for Software Architecture Evolution,” in, Jul. 2018, pp. 377–400. DOI: 10.1007/978-3-319-94764-8_16.
A. Krylovskiy, M. Jahn, and E. Patti, “Designing a Smart City Internet of Things Platform with Microservice Architecture,” in 2015 3rd International Conference on Future Internet of Things and Cloud, Aug. 2015, pp. 25–30. DOI: 10.1109/FiCloud.2015.55.
L. Sun, Y. Li, and R. A. Memon, “An open IoT framework based on microservices architecture,”China Communications, vol. 14, no. 2, pp. 154–162, Feb. 2017, Conference Name: China Communications, ISSN: 1673-5447. DOI: 10.1109/CC.2017.7868163.
R. K. Naha, S. Garg, D. Georgakopoulos, P. P. Jayaraman, L. Gao, Y. Xiang, and R. Ranjan, “Fog computing: Survey of trends, architectures, requirements, and research directions,” IEEE Access, vol. 6, pp. 47 980–48 009, 2018. DOI: 10.1109/ACCESS.2018.2866491.
S. Zhelev and A. Rozeva, “Using microservices and event driven architecture for big data stream processing,” in AIP Conference Proceedings, vol. 2172, Nov. 2019, p. 090 010. DOI: 10.1063/1.5133587.
K. Miao, J. Li, W. Hong, and M. Chen, “A Microservice-Based Big Data Analysis Platform for Online Educational Applications,” Scientific Programming, Jun. 2020, ISSN: 1058-9244 Pages: e6929750 Publisher: Hindawi Volume: 2020. DOI: 10.1155/2020/6929750.
D. Staegemann, M. Volk, N. Jamous, and K. Turoski, “Exploring the applicability of test driven development in the big data domain,” in Proceedings of the 2020 ACIS, Dec. 2020. [33] K. Lehmann and A. Freymann, “Demo Abstract: Smart Urban Services Platform a Flexible
Solution for Smart Cities,” in 2018 IEEE/ACM Third International Conference on Internetof- Things Design and Implementation (IoTDI), Apr. 2018, pp. 306–307. DOI: 10.1109/IoTDI.2018.00052.
R. Peinl, F. Holzschuher, and F. Pfitzer, “Docker Cluster Management for the Cloud - Survey Results and Own Solution,” Journal of Grid Computing, vol. 14, no. 2, pp. 265–282, Jun. 2016, ISSN: 1572-9184. DOI: 10.1007/s10723-016-9366-y.
J. Scott, Using microservices to evolve beyond the data lake. [Online]. Available: https://www.oreilly.com/content/using-microservices-to-evolve-beyond-the-data-lake (visited on 02/21/2021).
M. Kiran, P. Murphy, I. Monga, J. Dugan, and S. S. Baveja, “Lambda architecture for costeffective batch and speed big data processing,” in 2015 IEEE International Conference on Big Data (Big Data), Oct. 2015, pp. 2785–2792. DOI: 10.1109/BigData.2015.7364082.
T. Zsch¨ ornig, R. Wehlitz, and B. Franczyk, “A Personal Analytics Platform for the Internet of Things - Implementing Kappa Architecture with Microservice-based Stream Processing,” in Proceedings of the 19th International Conference on Enterprise Information Systems, Jan. 2017, pp. 733–738. DOI: 10.5220/0006355407330738.
Z. Tejada, Big data architectures - Azure Architecture Center. [Online]. Available: https://docs.microsoft.com/en-us/azure/architecture/data-guide/big-data (visited on 02/14/2021).
A. Ali and M. Abdullah, “A Survey on Vertical and Horizontal Scaling Platforms for Big Data Analytics,” International Journal of Integrated Engineering, vol. 11, Sep. 2019. DOI: 10.30880/ijie.2019.11.06.015.
S. J. Fowler, Production-Ready Microservices: Building Standardized Systems Across an Engineering Organization, 1st edition. Sebastopol, CA: O’Reilly Media, Dec. 2016, 4. Scalability and Performance - Production-Ready Microservices, ISBN: 9781491965979.
D. Taibi, V. Lenarduzzi, and C. Pahl, “Processes, Motivations, and Issues for Migrating to Microservices Architectures: An Empirical Investigation,” IEEE Cloud Computing, vol. 4, no. 5, pp. 22–32, Sep. 2017, Conference Name: IEEE Cloud Computing, ISSN: 2325-6095. DOI: 10.1109/MCC.2017.4250931.
Downloads
Published
How to Cite
Conference Proceedings Volume
Section
License
Copyright (c) 2021 Aamir Shakir, Daniel Staegemann, Matthias Volk, Naoum Jamous, Klaus Turowski
This work is licensed under a Creative Commons Attribution 4.0 International License.
