EIX: Group Assignment
“Big data analytics (BDA) capability and decision-making: The role of data-driven insight on circular economy (CE) performance in Malaysian manufacturing firms.”
1.0 Introduction
Big data analytics (BDA) has risen
to prominence as an essential component in delivering actionable insights for
decision-making (Dubey, Gunasekaran, Childe, Blome, & Papadopoulos, 2019).
The application of advanced analytic techniques to large amounts of data is
known as big data analytics. Corporate intelligence, customer relations, and a
lot of analytic applications can benefit from BDA (Russom, 2011). In order to
foster evidence-based decision-making, organisations require efficient ways of
processing huge volumes of heterogeneous data into useful comprehensions.
(Gandomi & Haider, 2015). The circular economy (CE) is a business idea that
has garnered a lot of traction among practitioners and researchers. The
successful adoption and execution of this corporate management model, however,
remains a difficulty. Big data analytics (BDA) should be used as the foundation
for informed and data-driven decision making in CE supply chain networks
(Gupta, Chen, Hazen, Kaur, & Santibañez Gonzalez, 2019).
According to Shamim, Zeng, Khan,
& Zia (2020), researchers have focused their attention on the relationship
between decision-making (DMQ) performance and BDA in developing market
enterprises due to the roles of BDA itself. However, empirical study on the
antecedents of data-driven insights (DDI) for improving DMQ and their impact on
circular economy performance is sparse. The previous research on the role of
BDA in enabling and informing decision-making has mostly concentrated on
organisational performance. Although BDA collects useful data on manufacturing
activities at various phases of the production cycle in order to maximise
resource use (Gupta et al., 2019), how data-driven insights could help
manufacturers update their product and process expertise is still not clear
(Ghasemaghaei & Calic, 2019). Little attention has been made in the
existing literature on BDA competence to understanding that data-driven
insights role in decision-making (Janssen, van der Voort, & Wahyudi, 2017)
and, as a result, improving CE performance in Malaysian manufacturing firms.
Concerns about design products,
productivity improvements and processes that embrace reusable and regenerative
design are emerging in developing and developed markets (Sauvé, Bernard, &
Sloan, 2016). According to (Gupta et al., 2019), we want to study whether
manufacturing firms in Malaysia can do well in CE activities with the help of
BDA, the ability to apply Big Data Analytics matters in acquiring insights that
lead to better decision-making (Acharya, Singh, Pereira, & Singh, 2018) and
the Malaysian Manufacturing firm's outcomes due to the assistance of BDA in
making decisions (Ghasemaghaei & Calic, 2019). However, BDA is said to have
a positive relationship with decision making. The addition of BDA can improve
the ability to make effective decision-making. (Dubey et al., 2019). BDA helps
decision-makers by revealing hidden linkages and facts that are difficult to
deduce using the human intellect. This aids decision-makers in making sound
decisions based on sound analytical underpinnings (Osman, Elragal, &
Ståhlbröst, 2022). However, in order to generate more data insights, many
researchers suggested that BDA should be increased as well (Ghasemaghaei &
Calic, 2019). But, most studies have ignored the data-driven insights when
decision-making is made. BDA and decision-making can be facilitated by
data-driven insights as well as can improve CE performance.
The importance of this study is to
examine the role of BDA to help the manufacturing firms in Malaysia to increase
the CE performance. We aim to address the research questions: how decisions in CE activities can be done
well by manufacturing firms in Malaysia with the help of BDA, how does the
capacity to use BDA matter in terms of assisting businesses in gaining insights
that lead to better decision-making and how were the Malaysian Manufacturing
firm's outcomes due to the assistance of BDA in decision-making. To be clear,
we want to know the roles of BDA capability in manufacturers in increasing
their quality of decision-making and CE performance. In this study, we
demonstrate the knowledge-based view (KBV) as a key in increasing the quality
of decision-making and organisational learning will increase the efficiency of
knowledge application. We organize this paper as follows: CE and BDA literature
in the section then we describe and introduce the hypothesis development as
well as the sampling technique. In the last section, we discuss the
contributions of the study, discussion and the directions of the future
research.
2.0 A
digital-enabled circular economy
Awan et al
(2021) described digital-enabled circular economy as a digitally empowered CE
is a new notion that may help enterprises improve resource use, efficiency, and
production. Kristoffersen (2020) addressed the problem that both CE and Digital
Technologies (DTs) are new fields, there is little systematic guidance on how
DTs can be used to fully realize the potential of circular strategies for
increasing resource efficiency and productivity, as well as little
understanding of the supporting business analytics (BA) capabilities required
to do so.
Cagno (2021)
stated that CE aims to close the material loop by changing from a linear to a
circular economy, reducing material extraction, waste disposal, and, as a
result, environmental strain. By using a variety of productivity and
efficiency-enhancing as well as restorative measures to keep goods, components,
and materials in use for longer, the CE envisions a global economy in which
value generation is disconnected from the use of scarce resources (Blomsma and
Tennant, 2020) and potentially to contribute to multiple UN Sustainable
Development Goals (SDGs) (Schroeder, Anggraeni, & Weber, 2019). Since they
allow for the creation and processing of data and information necessary for
circular business models and the complex needs of circular supply chains,
digital technologies are an enabler for the upscaling of the circular economy
(Berg, 2020). Digitalisation may be
regarded one of the Circular Economy's (CE) enablers since it gives insight and
information into products and assets, such as knowledge of their location,
condition, and availability (Antikainena, 2018).
There is a
dearth of support for enhancing existing and developing new ways for DTs to
help the CE through smart circular strategies (Kristoffersen et al., 2019,
Kristoffersen et al., 2020) and the industry's implementation of CE principles
has been slow (Circle Economy, 2020; Haas et al., 2015; Planing, 2015;
Sousa-Zomer et al., 2018). According to IBM, BDA may help firms make better and
quicker decisions, model and forecast future events, improve business
intelligence, and increased operational effectiveness (Chai, Labbe &
Stedman). This work contributes by analyzing the existing literature on big
data analytics. As a result, some of the many big data tools, methodologies,
and technologies that may be used are described, as well as their applications
and potential in diverse decision domains (Elgendy & Elragal, 2014).
3.0
Theoretical background and hypothesis development
Big Data
Analytics (BDA) is crucial in affecting the organisations’ decision making and
is advantageable for the performance of the Circular Economy (CE). According to
Wamba et al. (2017), there is a positive link between organisational
performance and BDA. For instance, Lehrer et al (2018) found that BDA improved
the probability of innovation performance. A series of literature conducted on
decision-making based on the theory of organizational learning. Knowledge
resources are difficult to duplicate since they are complicated (Alavi &
Leidner, 2001). In order to comprehend the data-driven insights and
decision-making with organisations, scholars have built the foundations for the
theory applications whereas organisational learning and Knowledge-Based View
(KBV) (Ghasemaghaei, 2019; Ghasemaghaei and Calic, 2019).
In his
research, Grant (1996) points out important insights from the KBV outlook,
proposing that the major strategic resource of organisations is knowledge and
these crucial resources of knowledge are rooted in the information technologies
and systems like software agents and data mining techniques (Alavi &
Leidner, 2001). Researchers have dedicated a great amount of focus on the
examination of how BDA leverages organisational performance (Rialti et al.,
2019), manufacturing performance (Dubey et al., 2019), environmental
sustainability (Dubey et al., 2019), the optimization of resources (Zhao et
al., 2017) and the advancement of performance of Circular Economy (Gupta et
al., 2019).
This means
that BDA will increase the efficiency of knowledge application through better
organisational learning. In this study, we argue that organisational learning
is crucial in the management of knowledge and knowledge sharing beyond the
firm's boundaries. BDA helps in the development of a sustainable competitive
advantage in which it promotes the production of hard-to-imitate and valuable
knowledge resources. Firms' learning skills are profoundly founded on
organisational capacity to encourage effective decision-making. The study has
examined the learning theory as the foundation for how organisations produce
data insights (Ghasemaghaei & Calic, 2019). Organisational learning theory
is critical for firms to take advantage of new possibilities from external
sources of information, and it may help them gain a competitive advantage.
Researchers have proposed that a quick generation of data-driven insight could
be through organisational learning capability to improve the efficiency of
decision-making (Ghasemaghaei & Calic, 2019) despite the previous study has
indicated that resource utilisation has increased by BDA (Song et al., 2017).
LSDM is
known as Large-Scale Decision-Making which is a new and quickly evolving study
topic that is becoming increasingly popular in real-world decision-making
settings. In the context of LSDM, Ding et al. (2020) have found that big data
is especially beneficial. Meanwhile, Palomares et al. (2013) proposed that LSDM
study gives useful insights for determining the best solution to real issues
and overcoming the non-cooperative behaviour of main decision-makers. According
to Ding et al. (2020), LSDM is a technique that is time-consuming and
difficult. Liu et al. (2014) explained that LSDM is the circumstance that
involved a number of 20 and above members' participation while this number
could be beyond the personnel inside the firm.
LSDM is
often used to address complicated problems in the operations management and
electronic commerce field of study. As a result, LSDM is associated with BDA,
which measures how many participants from the same industrial cluster are
collaborating together to solve complicated resource scarcity concerns. The
movement of decision-making from conventional to LSDM in the big data era
illustrates this situation has been boosted by Tang and Liao (2019) in their
latest efforts. Big data has long been regarded as critical in LSDM for dealing
with natural resource constraint issues. Decision-making on big data-driven is
often acceptable in leading organisations (McAfee et al, 2012). Tang and Liao
(2019) found that the potential benefits of employing big data technologies for
decision-making have begun to be studied in operations and management science
research. Ding et al. (2020) have emphasised the need of utilising appropriate
preference representations in LSDM for developing a data-driven large-group
decision-support system. The complicated issues are facing by modern
organisations could be solved associated with LSDM; yet, few studies has be
conducted on how the LSDM in dealing with complicated issues could be done
through the application of big data tools and a decision-support system.
Therefore,
BDA participation in CE is emerging as a crucial topic of study considering its
significance in organisations. According to Chen et al. (2012), BDA is a means
of putting practices and methodologies into action “that analyse critical
business data to help an enterprise better understand its business and market
and make timely business decisions” (p.1166). Decision-making quality uses in
his research and Raghunathan (1999) defined it as “the quality of the decision
made by the decision-maker” (p.280). In recent, through knowledge-share
practices, the research has introduced a relationship between decision-making
quality and BDA (Ghasemaghaei, 2019). Ghasemaghaei (2019) has proposed that
knowledge sharing has substantial implications for the outcomes of
decision-making. Despite the fact that BDA has been identified as a crucial
source for producing business value and improving company performance, there
has been very little study on the influence of BDA on decision-making quality.
Knowledge focus on data analytics has emerged as an interesting topic in a few
manufacturing firms in the last few decades (Alavi & Leidner, 2001;
Ghasemaghaei, 2019). However, further study is needed to determine if and under
what conditions an organisation may use current information to develop new
knowledge and execute effective actions (Alavi & Leidner, 2001).
3.1 Business
intelligence and analytics and data-driven insights
Business intelligence and analytics
(BI&A) was recently put under the limelight by academics and businesses in
the past 20 years ago (Chen et al., 2012). BI&A is an integral tool for
acquiring and assimilating intelligence upon conducting business and
identifying business innovation. By definition, BI&A is “the techniques,
technologies, systems, practices, methodologies, and applications that analyse
critical business data to help an enterprise better understand its business and
market and make timely business decisions” (Chen et al., 2012, p. 1166). Its
role is well researched however there is room for discovery in understanding
the impact of BI&A on circular economy (CE) performance. According to Dubey
et al. (2019), BI&A relates in accordance with the ability to enhance
innovation. Therefore, businesses that utilise BI&A will have much advanced
resources that they can take into consideration when deciding. The utilisation
of BDA makes the process to draw insights from a vast pool of data easier and
it elevates the understanding of prescriptive, predictive and descriptive data.
Therefore, business or project managers should heed the call to equip
themselves with the skills to foster effective data-driven insights. Hence, the
following hypothesis is suggested:
H1: Business intelligence and
analytics positively relates to a firm’s data-driven insights.
3.2 Big data analytics and data-driven insights
Big data
analytics enables businesses and organisations to make smart decisions by
disclosing previously concealed truths (Ding et.al, 2020). Scholars with a
dynamic capability perspective have frequently studied BDA to investigate the
link between BDA management and organisational success (Wamba et al., 2017).
The flexibility of BDA equipment, organisational competencies, and staff
experience all contribute to BDA. BDA equipment emphasises the similarity
between chronological (past data) and current tasks to obtain knowledge into
tasks, BDA management capabilities emphasise predicting future possible
consequences derived from BI&A information and data, and BDA personnel
expertise emphasises the decision-making process carried out to continue to
improve future outcomes, according to Wamba et al. (2017). BDA human
competencies, it is said, act as catalysts for mobilising management to
comprehend multiple business processes in order to answer changing demands in
the big data world. Firms may adapt BDA for strategic use by cultivating BDA
management capabilities. Hence, the following hypothesis as suggested: H2:
Business intelligence and analytics positively relates to a firm’s circular
economy performance.
Organisational
BDA can be an explanatory component in data-driven insights to learn from
previous practices and evaluate their influence on future outcomes.
Organisational BDA is regarded as an essential capacity that affects
organisational success in the research. BDA is a knowledge-based capacity in
the context of this research (Shamim et al., 2019). Wamba also suggested that
improving organisational understanding capacities can aid managers in better
understanding previous and present patterns as well as forecasting future
trends. Firms may use BDA to enhance internal processes, operations, and
organisational efficiency by identifying opportunities from various types of
data (Rialti et al., 2019). The following hypothesis suggested as:
H3: Big data analytics capabilities
positively relate to a firm’s data-driven insights.
According to
the research, main advantages can be derived in terms of material reuse by
improving material efficiency and incoherence of product design (reduction of
waste from the manufacturing process and material reuse), as well as other
sustainability-related benefits by attempting to integrate technological
infrastructure, boosting data management to track real-time material in the
product's life cycle, and connecting personal skills. Because BDA is recognised
as a vital facilitator of the circular economy, it may aid firms in aligning
resources with long- and short-term plans (Awan et al., 2021; Kristoffersen et
al., 2020). This is in line with Gupta
et al. (2019), who claimed that effective BDA use is vital for improving
resource circulation by boosting the efficacy of the material and, as a result,
increasing the effectiveness of company operations. Most literature also
suggests that BDA skills enable businesses to successfully leverage
infrastructure and manage personnel talent to build reuse and recycling-friendly
processes and products. In light of this new reality, Corporate leaders are
examining the effect of big data to see how it might be leveraged to produce
insights from organised and unstructured data for improved decision-making.
Hypothesis 4 is suggested as,
H4: Big data analytics capabilities
positively relate to a firm’s circular economy process.
3.3
Data-driven insights and decision-making quality
Data-driven decision-making has
permeated all levels of management, and good data management has become a
critical capability for success (Rejikumar,2018). Indeed, these decisions
determine success, with proper judgments assisting in the achievement of goals
and incorrect decisions leading to failure. Data is the most important
component in making good decisions (Yu, 2022). It is the centre point of
success for firms and sectors. However, the use of data in decision-making
presents strategic issues such as data errors, broad aggregate measurements,
and a lack of real-time data for decision-making. Hence, there is a need to
change present procedures and adopt a data-driven decision-making methodology
(Siddiqi, 2020).
H5: Data-driven insights positively
relate to a firm’s decision-making quality.
3.4 Big data decision-making and circular economy
performance
Prior research has progressively
stressed the necessity of good decision-making in product life cycle management
(Kristoffersen, Blomsma, Mikalef, & Li, 2020). Data-driven insight
influences manufacturers to make rational and effective decisions. Because a
huge volume of information is used to produce inventive solutions and solve
problems, an organization's learning skills incorporate and employ powerful
insights into the best course of action to improve decision-making
(Ghasemaghaei, 2019). However, according to (Božič & Dimovski, 2019), there
is a scarcity of research on BI&A-influenced decision-making and its
implications for value-added business operations. Choosing the optimal course
of action includes learning about the optimal courses of action and utilising
advance technology for the redesign or reuse of services and goods to maximise
material recovery. Increase the efficiency of materials and the efficiency of
end-of-life products to redesign elements and create business value through an
efficient decision-making process.
We suggest that if CE-related decisions are derived by obtaining some
relationships and carefully implemented based on reliable data insights, companies
can better able to discover new patterns by using visualisation tools in
adapting to different environmental challenges, thereby improving effectiveness
and competitiveness. As a result, we recommend the following:
H6:The effectiveness of big data
decision-making is positively related to the circular economy.
3.5 The mediating role of data-driven insights
In order to improve its CE performance, the organisation
(firm) is stressing the issue more. As a result, enhanced decision-making
participation is split into two categories: knowledge-based resources as well
as organisational learning. Learning theory resources, as stated by
(Puranam and Swamy, 2016), play a significant role in shaping outcomes. This is
due to the fact that data collection, processing, interpretation, and synthesis
allows firms to increase their performance and competitive edge. Data-driven
insights are increasingly posing a challenge to organisations, resulting in
better decision-making (LaValle et al., 2011). An organisation can improve
decision-making by introducing data-driven insights, which boosts added-value
company's operations.
It has been proved that BI&A aids in the identification
of control of knowledge techniques as well as produces new perspectives
on decision-making. Consequently, our findings assist businesses in
making informed decisions. Also, we predict that data-driven insights from
organisational BDA capabilities will be more strongly linked to BI&A.
Concern for the efficient use of data-driven insights fosters a long-term
approach to business decision-making, as well as a concentration on BI&A. The
perspective is formed based on a concept proposed by March (1997). The primary
premise of this strategy is that credible data is used to implement choices in
the firm. Information processing, collecting, and interpretation have an impact
on an organisation's decision-making process.
We believe that the BDA's capabilities can transform the
process of decision-making by using data-driven insights. Hence, we recommend
the hypotheses below:
H7: Data-driven insights mediate the
relationship between business intelligence and analytics and decision-making
quality.
H8: Data-driven insights mediate the
relationship between big data analytics capabilities and decision-making
quality.
4.0 Sampling technique
4.1 Sample and Data Collection
The ASEAN
has launched the Digital Asean project to address the challenges that would
underlie a regional digital economy in ASEAN, allowing the Fourth Industrial
Revolution's advantages to be fully realised and turned into a force for
regional economic inclusion. In
response to this endeavour, the Malaysian government announced the MyDIGITAL
goals. The Malaysia Digital Economy Blueprint is being developed for the period
2021 to 2030 in order to determine the direction and specify the strategies,
initiatives, and objectives that will be used to provide the groundwork
for the digital economy's growth, including bridging the digital divide. The
Blueprint will ensure that the country is equipped to adopt digital technology
by harnessing present opportunities. As a member of ASEAN, the Malaysian
government is actively working on the idea of MyDigital, laying out 22
strategies that include 48 national initiatives and 28 sectoral initiatives as
part of a plan to assist the digitization of the economy. It will be implemented
in three parts over a ten-year period, from 2021 to 2030. The first phase,
which runs from 2021 to 2022, focuses on strengthening the foundations and
speeding up digitalization. From 2023 to 2025, the second phase will focus on
promoting inclusive digital transformation. In the last phase, which runs from
2026 to 2030, Malaysia is expected to become a regional market for digital
products and digital solutions providers (Economic Plan Unit, 2021). As a
result, Malaysia is an appropriate setting for this research.
The
study focused on the population of big data-driven manufacturing firms in
Malaysia. Meanwhile, the amount of population is unknown to researchers. Hence,
we conduct the study using the non-probability research method. The sampling
technique used in this study is convenience and snowballing sampling technique.
In the initial data collection, we emailed the Google Forms questionnaire link
to ten firms. We access the firms’ contact details through the Federation of
Malaysian Manufacturers’ website. In the email, we asked the recipients to
forward the questionnaire to the personnel who work with big data and
decision-making in the related firms. We received the response from 7 firms and
30 employees and we were able to use all of them in the data analysis. The
duration of data gathering took three weeks from April 11 to May 2, 2022. The
firms that took part in this study are manufacturing companies that employed
between 20 to 200 people and firms aged around 2 to 20 years. We gathered
information from essential frontline employees and managers, as well as middle
and high managers, who were involved in big data-related operations. To exclude
any influence that these factors could have on decision-making, we controlled
firm age, firm size, respondent level of managerial, respondent education,
respondent age, and respondent experience in this study.
4.2 Data Analysis Plan
The
study is based on a research method that has been established in order to
achieve the research objectives of the study. The research method begins with a
consultation about the research’s problem statement with the supervisor.
Discussions include a variety of
research methods such as identifying problem statements, research
topics, and research objectives. After that, the consultation continues to
discuss the process of collecting information on the problem that has been
discussed, identifying respondents and analysing the data.
The study
will be analysed right after we managed to get our thirty (30) respondents in
the questionnaire. The component in the questionnaire is divided into two
sections. Section A consists of the background of the respondents such as name
of company, type of company, year of company, total revenue, total employee,
position, age of respondents, and level of education. Next, descriptive
statistical methods were used in order to analyse the data in section B. This
is to provide a high-level of analysis of the data. For information, a 7-point
likert scale was used to collect respondent's answers to the study.
Once the data is collected, the data will be
analysed using statistical software. We
choose Statistical Package for Social Science (SPSS) version 20.0 considering
SPSS is the data analysis tool that is commonly used by other researchers. This
software allows for precise data analysis, which was then used to result at a
well-informed conclusion and set of suggestions. To analyse closed-ended
questions, a 7-point likert scale is used.
Descriptive statistics as well as inferential statistics were applied in
this study to interpret the survey questions on the level of significance where
(p <0.05). While, for the statistical tool, we have decided to use
hypothesis testing. Hypothesis testing
method has also been used in previous study. Then, we will create the shell
table. The objective of this table is to construct a dataset with the predicted
total number of sections and rows in the end table. Table 1 and 2 (Appendix) are the shell tables
that we can generate from our questionnaire.
4.3 Measures
This
study was done using the quantitative method, which is based on a structured
questionnaire in data collection. There are five variables measured in the
questionnaire. These items have been self-developed, adapted, and adopted.
Eight items from Shamin et al. (2019) were used to assess decision-making
quality. Božič &
Dimovski (2019) and Gold et al. (2001) inspired us to construct nine
items to access BI&A. Eight items from Akter et al. (2016) and eight items
from Ghasemaghaei and Calic (2019) were adopted to measure BDA and data-driven
insights respectively. The CE performance measuring scale was designed by the
authors based on available literature and implemented in the research. On a
seven-point Likert scale, respondents were asked to rate how strongly they
agreed with the following statements: "1-strongly disagree" to
"7-strongly agree."
5.0 Results
5.1 Reliability
and validity
Muijhs in 2004 contended that validity is one of
the most crucial design features of any measurement instrument study. Hence,
validity can be defined as a process that corresponds to how well a test
evaluates what needs to be tested. While validity is the writing ability
evaluation accurately represents the writing ability being assessed. Instrument
reliability is critical to ensuring that the instrument will perform
consistently at other times. As a result, the instrument as a test is
trustworthy.
Our result would be invalid if we are not truly
evaluating what we claim to be evaluating. We have used The Fornell and Larcker
criterion strategy to investigate discriminant validity in our research. The
construct of factor loadings must be more than 0.65 to generate convergent
validity. Besides, the average variance extracted (AVE) and composite
reliability or known as CR are admissible if the values are greater than 0.5,
and the AVE remains smaller than the construct's CR (Fornell.and Larcker,.1981). Table 3 depicts the values
that have all the criteria. As we can see, the column of factor loadings;for all the components are
higher than 0.65. The sixth column also shows the Cronbach’s Alpha’s
reliability where the value is in the range of 0.81 to 1.00, which indicates it
is considered as very reliable.
Other measurements, such as
the hetero-monotrait property ratio (HTMT), were used to examine the validity
of discrimination. HTMT is also known as a newly proposed multi
thread-multimethod matrix measurement. As stated by Henseler, Ringle, &
Sarstedt (2015), this method is better than the method established by Fornell
and Larcker in 1981 and cross-section loading method. According to the
requirements, if the HTMT level is higher than the threshold, we could notice
the lack of discriminating in validities. According to the criterion, each
construct's HTMT ratio should be smaller than 0.85 to show that there is
convergent validity. Hence, as seen in Table 4, we managed to establish the
discriminant validity as all the components meet all the requirements suggested.
5.2 Hypotheses
testing
To
evaluate the hypotheses, we employed Partial Least Squares (PLS) structural
equation modelling which is useful for a multiple of reasons: (1) PLS is a
structural equation modelling approach for estimating composite factor models
that uses construct scores rather than sum scores. (2) PLS contains adequate
and necessary information to estimate various weights as well as assist in the
detection of a wide range of measurement model misspecifications. (3) When
other models fail, PLS can be used in a variety of situations involving tiny
samples. (4) For exploratory research, PLS can be a useful tool. Furthermore,
it is considered both the measurement model and the theoretical structural
model at the same time (Chin et al., 2003) . We have 6 hypotheses needed to be
tested in this research. We refer to Table 5 for the hypotheses testing.
First,
we looked at direct correlations. BI&A is favourably and strongly related
with data-driven insight (=0.695, p 0.001) and circular economy performance
(=0.513, p 0.001), according to the findings. H1 and H2 have been ruled out by
these results. The results also show a clear link between BDA and data-driven
insight ( =0.000, p 0.001) and CE performance ( =0.133, p 0.001), hence H3 is
accepted and H4 is rejected. Decision-making quality (= 0.003, p 0.001) is
favourably connected to data-driven insights, and decision-making value is
strongly related to CE performance (= 0.475, p 0.001). H5 has been supported by
these data, but not H6. Because the sample size is small and the p-value is
much less than the level of significance, we reject H1, H2, H4, H6.
6.0 Discussion
The impact of business intelligence
and analytics and big data analytics on CE performance by improving DMQ and DDI
was investigated using data from 30 respondents from Malaysian manufacturing
enterprises. The findings suggest that CE performance are not influence or
related by the capabilities of business intelligence and analytics and big data
analytics. These findings are different with other study which says they are
positively related. Furthermore, whereas
BDA capabilities are a better prognosticator of DDI and not of CE performance.
The findings of our research contradict (Ghasemaghaei & Calic, 2019)
assertion that BDA capabilities and BI & A have a favourable link with DDI
and improve DMQ. We also discovered that BDA is more highly linked to the
quality of decision-making while having no effect on CE performance. The need
of using suitable preference representations in LSDM for developing a
data-driven large-group decision-support system have highlighted in the
previous studies. (Ding et al., 2020). Consequently, in this study, LSDM
demonstrates the extent to which members of industrial clusters use BDA to
address the complex resource shortage. There are 6 out of 8 hypotheses that
have been tested. Based on the diagnostic testing of the variables, there are
only two hypotheses that can be accepted. First, it is proven that BDA
capabilities positively influence a firm’s DDI at H3= 0.000, p < 0.01. Then,
it also has been proven that DDI positively relate to a firm’s DMQ at H5=
0.003, p < 0.05. Lastly, all of the other hypotheses are rejected as the
p-values are greater than the significance level of 5%. The other two
hypotheses, H7 and H8 have been exempted from data analysis as the sample sizes
are too small.
6.1 Theoretical
contributions
This
research study adds to the literature in a variety of ways. The study's key
contribution is the development and testing of a conceptual model that
identifies the process that allows a company to improve its CE. First and
foremost, the purpose of this article is to investigate how BDA capabilities
impact data-driven insights and CE performance in manufacturing companies. The
efficiency and precision of new manufacturing industries that utilise BDA
capabilities may have a favourable influence across the entire organisation in
terms of process and product innovation, as well as operational efficiency
(O’Donovan, 2015). Our findings add to the body of knowledge by revealing that
the KBV is equally significant in improving decision-making quality. This research
establishes a relationship between the KBV, and the quality of decision-making
influenced by data-driven insights. The KBV lens might be used to explain our
conceptual model. We use the KBV to argue that learning is a key feature that
offers a fresh source of information and develops new knowledge, allowing
businesses to extract value from key knowledge sources. Our findings show that
businesses should improve the quality of their decision-making in order to
improve CE outcomes. Company leaders play a key role in accomplishing the aims
of BDA adoption since the use of analytics in real-time decision-making has
major positive implications on business operations and performance enhancement
(Adrian & Rusli & Yusmadi, 2017).
Second, according to this study,
decision-making may be achieved by developing data-driven insights that are
gathered within an organisation using BDA. Although BDA skill is necessary for
making quality decisions, there is little research on how to increase decision
quality. (Janssen et al., 2017). It also necessitates internal efforts and
particular competencies to assure the quality of big data-driven choices
(Janssen, van der Voort, & Wahyudi, 2017; Shamim, Zeng, Shariq, & Khan,
2019). As a result, we propose that manufacturing companies with higher
decision-making skills might transmit and internalise information to create
recycling goods.
Third, most of the studies outside
of Malaysia has proven that business intelligence and analytics work closely
with a firm’s driven insights. Business
analytics entails much more than just displaying data, numbers, and statistics
which means that the application of logic and mental processes is at the heart
of analytics methods for deducing meaning from data (Nerkar, 2016). In
Malaysia, on the other hand, there are few and inadequate empirical studies to
prove that these two has relations to each other. Subsequently, we found that
firm’s circular economy performance has nothing to do with business
intelligence and analytics. Previous studies have empirical findings that
emphasise the necessity of having a more holistic approach to BA development so
that businesses can better manage their CE implementation and ROC of their IT
portfolio, resulting in enhanced organisational performance and higher BA
investment returns (Kristoffersen & Mikalef & Blomsma & Li,2021).
Moreover, our findings found that
big data analytics capabilities have no influence toward a firm’s circular
economy process. In Malaysia manufacturing firms, many firms has no or little
awareness of the power and potential of BDA that could be to boost the circular
economy process, thus, they did not adopt the BDA as a tool in their
decision-making process and have no or little effect on the firm’s circular
economy. Lastly, based on our research, we found that big data decision-making
effectiveness has no effect onto circular economy performance. According to Del Giudice, Chierici,
Mazzucchelli, and Fiano (2020), while the importance of big data in successful
decision-making processes has been recognised, few empirical studies have
focused on how big data may be utilised to improve the environmental, social,
and economic performance of enterprises in the circular economy.
6.2 Managerial
relevance
Acknowledgement on the importance of
BDA in making decisions is rising, however, there is gap for research on its
ability to influence the process of making decisions (Janssen et al., 2017).
Therefore, this study is set to understand the effect of BI&A and BDA
ability to influence decision making processes. Apart from that, this research
can benefit the manager level executive with some idea of managing data-driven
insights in influencing decision-making process. Firstly, our research
positively shines lights upon the gaps in CE performance in terms of
predictive, perspective, and descriptive analytics. Established BDA
capabilities relates to better critical decision-making processes especially in
the circular economy. One transformation that managers can undertake is to
change the production method from linear to closed loop aided by the data
driven strategies. Next, from the previous section, it is understood that there
is an established BDA and DDI culture relationship (Duan et al., 2020).
However, little exploration has been made to understand the chain between
BI&A and data insights. The results in this research shed light on big data
decision-making. Managers are reminded not to fully trust and rely on BDA as it
may cause flawed and biased decision-making process.
Thirdly, this topic has been
actively discussed among academicians as the collective awareness on the
importance of BDA and BI&A. Previous research shown that there is relation
of relying on BDA capabilities and the ability to make better decision (Tang
and Liao, 2019). However, when it comes to empirical test to understand the
role of data driven insights, little research has been conducted. Based on our
results, we advised managers to carefully choose and select their best DDI in
order to design strategies will improve their circular economy standards. Firms
that aim to heighten their social impact standards are advised to implement
data-driven insights to understand their specific issues. From the literature
review, decision making involving big group must not fully rely on the
professionals (Emmerling and Rooders, 2020). The result has shown that it is
best to incorporate BI&A, and BDA capability as well. It is believed that
better CE performance is attributed to incorporation of artificial intelligence
such as BI&A, BDA and data-driven insights.
7.0 Conclusion
The impact of BDA capabilities on CE
performance was investigated in this study, which looked at the decision-making
quality of manufacturing enterprises in Malaysia. Our findings show that BDA
has a favourable impact on data-driven insights, but it does not lead to
improved CE performance. BDA is positively influenced the DDI and DMQ will
positively affected by data driven insights. This shows that BDA capabilities indirect
relationship with decision-making. Overall, we conclude that while BDA
capabilities are important and useful in the creation of DDI, the relationship
between decision-making and CE may not be shaped by them.
7.1 Limitations
and future research
This study faces limitations on the
sample size and the context of the study. Research is conducted with 30
respondents on data collection due to time constraints. The sample size is too
small and inaccurate to draw a conclusion about the population for this study.
Future studies could focus on a larger sample size within an appropriate period
in conducting the research. Besides, this study is conducted in the developing
country context, Malaysia in which these findings might suit only a single
economic zone. However, future research is needed to develop and boost the
generalizability of these findings which fits well with other economies such as
European countries and the United State.
In recommendation, future research
can focus on scopes other than in this study. For instance, in examining the
role of Circular Economy performance in mediating the environmental and
innovation performance with Big Data Analytics capability. We also suggest
future researchers test the data integration management capability in mediating
the decision-making and Big Data Analytics capability. The study needs to pay
attention to the investigation of how Big Data Analytics capabilities in
enhancing the Circular Economy performance across industries is also
recommended in the future. The stakeholders' diversion role is crucial in a
Circular Economy and future research may focus on the role of stakeholders in
robust Circular Economy performance.
References
Adrian,
C., Rusli Abdullah, Rodziah Atan, & Yusmadi Yah Jusoh. (2018). Conceptual
Model
Development of Big Data
Analytics Implementation Assessment Effect on
Decision-Making.
doi:10.9781/ijimai.2018.03.001
Akter,
S., Wamba, S.F., Gunasekaran, A., Dubey, R., Childe, S.J., 2016. How to
improve firm performance using
big data analytics capability and business
strategy alignment? Int. J.
Prod. Econ. 182, 113–131. https://doi.org/10.1016/j.
ijpe.2016.08.018.
Alavi, M., Leidner, D.E. (2001).
Knowledge management and knowledge management systems: conceptual foundations
and an agenda for research. MIS Q. 25 (1),
107–136. http://www.jstor.org/stable/3250961.
Babu, M. M., Rahman, M., Alam, A.,
& Dey, B. L. (2021). Exploring big data-driven innovation in the
manufacturing sector: evidence from UK firms. Annals of Operations Research, 1-28. Retrieved from https://link.springer.com/article/10.1007/s10479-021-04077-1
Berg,
H., Blévennec, K. L., Kristoffersen, E., Strée, B., Witomski, A., Stein, N., .
. .
Vrancken, K. (2020). Digital circular economy as a cornerstone of
a sustainable
European industry
transformation. Paper presented at the White Paper - ECERA
European Circular Economy
Research Alliance.https://www.era-min.eu/sites/default
/files/publications/201023_ecera_white_paper_on_digital_circular_economy.pdf
Blomsma,
F., & Tennant, M. (2020). Circular economy: Preserving materials or
products?
Introducing the Resource
States framework. Resources, Conservation
and Recycling,
156. doi:https://doi.org/10.1016/j.resconrec.2020.104698
Božič, K., & Dimovski, V. (2019). Business intelligence
and analytics use, innovation
ambidexterity, and firm
performance: A dynamic capabilities perspective. The
Journal of Strategic
Information Systems, 28(4),
101578.
https://doi.org/10.1016/j.jsis.2019.101578
Cagno,
E., Neri, A., Negri, M., Bassani, C. A., & Lampertico, T. (2021). The Role
of Digital
Technologies in
Operationalizing the Circular Economy Transition: A Systematic
Literature Review. Applied Sciences, 11(8), 3328. Retrieved
from
https://www.mdpi.com/2076-3417/11/8/3328
Chai,
W., Labbe, M., & Stedman, C. big data analytics. Retrieved from
https://www.techtarget.com/searchbusinessanalytics/definition/big-data-analytics
Chen,
H., Chiang, R.H.L., Storey, V.C, 2012. Business intelligence and analytics:
from big
data to big impact. MIS Q. 36 (4),
1165–1188. https://doi.org/10.1016/S0140-6736
Circle
Economy. (2020). The Circularity Gap
Report 2020. Circ. Econ. Retrieved from
Amsterdam, Netherlands:
Ding,
R.X., Palomares, I., Wang, X., Yang, G.R., Liu, B., Dong, Y., Herrera, F.
(2020).
Large-scale decision-making:
characterization, taxonomy, challenges and future
directions from an artificial
intelligence and applications perspective. Inf. Fus. 59, 84–
102. Retrieved from
https://doi.org/10.1016/j.inffus.2020.01.006.
Dubey,
R., Gunasekaran, A., Childe, S.J., Blome, C., Papadopoulos, T. (2019a). Big
data and
predictive analytics and
manufacturing performance: integrating institutional theory,
resource-based view and big
data culture. Br. J. Manag. 30 (2),
341–361.
Dubey,
R., Gunasekaran, A., Childe, S.J., Bryde, D.J., Giannakis, M., Foropon, C.,
Hazen, B.T., 2019b. Big data
analytics and artificial intelligence pathway to
operational performance under the
effects of entrepreneurial orientation and
environmental dynamism: a study of
manufacturing organisations. Int. J. Prod. Econ.
107599. Retrieved from https://doi.org/10.1016/j.ijpe.2019.107599
Economic Plan Unit. (2021). Malaysian Digital Economy Blueprint. Retrieved from
https://www.epu.gov.my/sites/default/files/2021-02/malaysia-digital-economy-b
lueprint.pdf
Elgendy,
N., & Elragal, A. (2014). Big Data
Analytics: A Literature Review Paper. Paper
presented at the ICDM 2014:
Advances in Data Mining. Applications and Theoretical
Aspects. Retrieved from https://link.springer.com/chapter/10.1007/978-3-319-08976-
8_16#citeas
Fornell, C., & Larcker, D. F. (1981). Evaluating
structural equation models with
unobservable variables and
measurement error. Journal of marketing research, 18(1),
39-50.
Ghasemaghaei,
M. (2019). Does data analytics use improve firm decision making quality?
The role of knowledge sharing
and data analytics competency. Decis. Support. Syst.
120, 14–24. Retrieved from https://doi.org/10.1016/j.dss.2019.03.004.
Ghasemaghaei, M., Calic, G., 2019. Does big data enhance
firm innovation
competency? The mediating role
of data-driven insights. J. Bus. Res. 104,
69–84. Retrieved from
https://doi.org/ 10.1016/j.jbusres.2019.07.006.
Giudice, M. D., Chierici, R., Mazzucchelli, A., & Fiano,
F. (2021). Supply chain
management in the era of
circular economy: the moderating effect of big data. The
International Journal of
Logistics Management, 32(2), 337-356. Retrieved from
https://doi.org/10.1108/IJLM-03-2020-0119
Gold, A.H., Malhotra, A., Segars, A.H., 2001. Knowledge
Management: An
Organisational Capabilities
Perspective. Journal of Management
Information
Systems, 18(1), 185-214. Retrieved from:
https://www.tandfonline.com/doi/abs/10.1080/07421222.2001.11045669
Grant,
R.M. (1996). Toward a knowledge-based theory of the firm. Strateg. Manag. J. 17
(S2), 109–122.
Gupta,
S., Chen, H., Hazen, B.T., Kaur, S., Santibanez ˜ Gonzalez, E.D.R. (2019).
Circular
economy and big data
analytics: a stakeholder perspective. Technol. Forecast. Soc.
Change 144, 466–474.
https://doi.org/10.1016/j.techfore.2018.06.030.
IBM.
Big data analytics. Retrieved from https://www.ibm.com/analytics/big-data-analytics
Kristoffersen,
E., Blomsma, F., Mikalef, P., & Li, J. (2020). The smart circular economy:
A
digital-enabled circular
strategies framework for manufacturing companies. Journal
of Business Research, 120,
241-261. Retrieved from
https://doi.org/10.1016/j.jbusres.2020.07.044
LaValle,
S., Lesser, E., Shockley, R., Hopkins, M. S., & Kruschwitz, N. (2011). Big
data,
analytics and the path from
insights to value. MIT sloan management review, 52(2),
21-32.
Lehrer,
C., Wieneke, A., vom Brocke, J.A.N., Jung, R., Seidel, S. (2018). How big data
analytics enables service
innovation: materiality, affordance, and the individualization
of service. J. Manag. Inf.
Syst. 35 (2), 424–460.
Liu,
B., Shen, Y., Chen, X., Chen, Y., Wang, X. (2014). A partial binary tree DEA-DA
cyclic
classification model for
decision makers in complex multi-attribute large-group
interval-valued
intuitionistic fuzzy decision-making problems. Inf. Fus. 18 (1), 119–
130. Retrieved from
https://doi.org/10.1016/j.inffus.2013.06.004.
Malaysia Digital Economy Blueprint:
MyDigital blueprint to drive Malaysia’s
high-income nation aspirations. (2021, March). The Edge Markets.
https://www.theedgemarkets.com/article/malaysia-digital-economy-blueprint
-mydigital-blueprint-drive-malaysias-highincome-nation
March, J. G. (1997). Understanding how decisions happen in
organisations. Organisational
decision making, 10, 9-32.
McAfee,
A., Brynjolfsson, E., Davenport, T.H., Patil, D.J., Barton, D. (2012). Big
data: the
management revolution. Harv.
Bus. Rev. 90 (10), 60–68.
Muijs,
D. (2004). Validity, reliability and generalisability. Doing quantitative
research in
education with SPSS, 64-84.
MyGOV - The Government of Malaysia’s
Official Portal. (2022).
Malaysia.gov.my.
https://www.malaysia.gov.my/portal/content/31187
Nerkar, A. D. (2016). Business
Analytics (BA): Core of Business Intelligence (BI). International Journal of Advanced Engineering, Management and Science
(IJAEMS) 2(12). Retrieved from https://media.neliti.com/media/publications/239736-business-analytics-ba-core-of-business-i-3e76db25.pdf
Palomares, I., Martinez, L.,
Herrera, F. (2013). A consensus model to detect and manage noncooperative
behaviors in large-scale group decision making. IEEE Trans. Fuzzy Syst. 22 (3), 516–530.
Planning,
P. (2015). Business model innovation in a circular economy reasons for
non-acceptance of circular
business models. Open J. Bus. Model
Innov., 1.
Puranam, P., & Swamy, M. (2016). How initial
representations shape coupled learning
processes. Organisation Science, 27(2), 323-335.
Raghunathan, S., 1999. Impact of information quality and
decision-maker quality on decision
quality: a theoretical model
and simulation analysis. Decis. Support Syst. 26 (4), 275–
286. Retrieved from https://doi.org/10.1016/S0167-9236(99)00060-3.
Rejikumar, G., Asokan, A. A., &
Sreedharan, V. R. (2018). Impact of data-driven decision-making in Lean Six
Sigma: an empirical analysis. Total
Quality Management & Business Excellence, 31(3/4), 279-296. Retrieved
from https://eds-p-ebscohost-com.ezproxy.um.edu.my/eds/detail/detail?vid=0&sid=850a3b0b-6e27-4bad-b883-9bf26b96a31e%40redis&bdata=JnNpdGU9ZWRzLWxpdmU%3d#AN=141097757&db=heh
Rialti, R., Marzi, G., Ciappei, C.,
Busso, D. (2019a). Big data and dynamic capabilities: a bibliometric analysis
and systematic literature review. Manag
Decis. 57 (8), 2052–2068. Retrieved from https://doi.org/10.1108/MD-07-2018-0821.
Shamim, S., Zeng, J., Khan, Z.,
& Zia, N. U. (2020). Big data analytics capability and decision making
performance in emerging market firms: The role of contractual and relational
governance mechanisms. Technological
Forecasting and Social Change, 161. Retrieved from https://doi.org/10.1016/j.techfore.2020.120315
Shamim, S., Zeng, J., Shafi Choksy, U., & Shariq, S. M.
(2020). Connecting big data
management capabilities with
employee ambidexterity in Chinese
multinational enterprises
through the mediation of big data value creation at
the employee level. International Business Review, 29(6), 101604. Retrieved from
https://doi.org/10.1016/j.ibusrev.2019.101604
Siddiqi, D. A., Abdullah, S.,
Dharma, V. K., Shah, M. T., Akhter, M. A., Habib, A., . . . Chandir, S. (2020).
Using a low-cost, real-time electronic immunisation registry in Pakistan to
demonstrate utility of data for immunisation programs and evidence-based
decision making to achieve SDG-3: Insights from analysis of Big Data on
vaccines. International Journal of
Medical Informatics, 149. Retrieved from https://doi.org/10.1016/j.ijmedinf.2021.104413
Song,
M., Cen, L., Zheng, Z., Fisher, R., Liang, X., Wang, Y., Huisingh, D. (2017).
How
would big data support societal
development and environmental sustainability? Insights
and practices. J. Clean. Prod.
142, 489–500. Retrieved from https://doi.org/10.1016/j.
Jclepro.2016.10.091.
Sousa-Zomer,
T. T., Magalhães, L., Zancul, E., & Cauchick-Miguel, P. A. (2018).
Exploring
the challenges for circular
business implementation in manufacturing companies: An
empirical investigation of a
pay-per-use service provider. Resources,
Conservation
and Recycling, 135, 3-13.
doi:10.1016/j.resconrec.2017.10.033
Wamba,
S.F., Gunasekaran, A., Akter, S., Ren, S.J., Dubey, R., Childe, S.J. ( 2017).
Big data
analytics and firm performance:
effects of dynamic capabilities. J. Bus. Res. 70, 356–
365.
Yu, S., QIng, Q., Zhang, C.,
Shehzad, A., Oatley, G., & Xia, F. (2022). Data-Driven Decision Making in COVID-19 Response: A Survey.
Retrieved from http://arxiv.org/abs/2202.11435
Zhao,
R., Liu, Y., Zhang, N., Huang, T. (2017). An optimization model for green
supply
chain management by using a big
data analytic approach. J. Clean. Prod. 142, 1085–
1097. Retrieved from https://doi.org/10.1016/j.jclepro.2016.03.006.
Appendix
Table 1: Section A: Demographic and information of the
respondents and businesses.
Table 2: Section B: Big Data Analytics (BDA)
Table 3: Reliability and convergent
validity.
|
Variable |
Items |
Factor loadings |
Average Variance Extracted (AVE) |
Composite Reliability |
Cronbach’s alpha |
|
BIA |
BIA1 BIA2 BIA3 BIA4 BIA5 BIA6 BIA7 BIA8 BIA9 |
0.917 0.916 0.942 0.957 0.894 0.954 0.953 0.948 0.915 |
0.871 |
0.984 |
0.981 |
|
BDA
|
BDA1 BDA2 BDA3 BDA4 BDA5 BDA6 BDA7 BDA8 |
0.965 0.959 0.971 0.922 0.908 0.942 0.942 0.922 |
0.887 |
0.984 |
0.982 |
|
Data-driven
insights |
DD1 DD2 DD3 DD4 DD5 DD6 DD7 DD8 DD9 |
0.968 0.960 0.959 0.952 0.959 0.958 0.948 0.966 0.957 |
0.919 |
0.990 |
0.989 |
|
Decision-making
quality |
DMQ1 DMQ2 DMQ3 DMQ4 DMQ5 DMQ6 DMQ7 DMQ8 |
0.977 0.967 0.958 0.956 0.954 0.947 0.968 0.943 |
0.919 |
0.989 |
0.987 |
|
Circular
economy performance |
CE1 CE2 CE3 CE4 CE5 |
0.979 0.976 0.956 0.954 0.953 |
0.928 |
0.985 |
0.981 |
Table 4: Heterotrait-Monotrait
ratio, skewness, and kurtosis.
|
|
Factors |
Skewness |
Kurtosis |
1 |
2 |
3 |
4 |
|
1 |
BDA |
-1.894 |
4.634 |
|
|
|
|
|
2 |
BIA |
-2.254 |
5.748 |
0.957 |
|
|
|
|
3 |
CE
performance |
-2.308 |
5.408 |
0.845 |
0.861 |
|
|
|
4 |
DMQ |
-1.577 |
2.904 |
0.981 |
0.929 |
0.899 |
|
|
5 |
DD |
-1.681 |
3.291 |
0.972 |
0.949 |
0.900 |
0.981 |
Table 5: Path Analysis and
Hypothesis Testing
|
Path |
Original Sample |
T Statistics |
P Values |
|
BIA
→ DD |
0.079 |
0.392 |
0.695 |
|
BIA
→ CE performance |
0.390 |
0.654 |
0.513 |
|
BDA
→ DD |
0.893 |
4.657 |
0.000 |
|
BDA
→ CE performance |
-0.865 |
1.506 |
0.133 |
|
DD
→ DMQ |
0.636 |
2.988 |
0.003 |
|
DMQ
→ CE performance |
0.423 |
0.715 |
0.475 |
Table 6: Hypotheses and the results
|
Hypothesis |
p-value |
Result |
|
H1: Business intelligence and
analytics positively relates to a firm’s data-driven insights. |
0.695 |
Rejected |
|
H2: Business intelligence and
analytics positively relates to a firm’s circular economy performance. |
0.513 |
Rejected |
|
H3: Big data analytics
capabilities positively relate to a firm’s data-driven insights. |
0.000 |
Accepted |
|
H4: Big data analytics capabilities
positively relate to a firm’s circular economy process. |
0.133 |
Rejected |
|
H5: Data-driven insights
positively relate to a firm’s decision-making quality. |
0.003 |
Accepted |
|
H6: Big data decision-making
effectiveness positively relates to circular economy performance. |
0.475 |
Rejected |
Appendix 5
Google Form (Questionnaire)
Comments
Post a Comment