In 2023, data science and artificial intelligence made headlines. This sharp increase in visibility was, of course, caused by the development of generative AI. What therefore could occur in the field in 2024 to maintain its prominence? And what practical effects can these changes have on businesses?
We’ve polled data and technology executives three times in the last few months. Two involved attendees of the Information Quality Symposium and the Chief Data Officer at MIT; one was sponsored by Thoughtworks (not yet published), and the other by Amazon Web Services (AWS). We’ve previously written on Wavestone’s annual surveys, which were formerly conducted by NewVantage Partners. This was their third survey. Over 500 top executives participated in all of the new questionnaires, however there may have been some overlap in who answered.
Get News On Using Data And AI To Lead
Get monthly insights on the implications of artificial intelligence for your business and your clientele.
While surveys cannot foretell the future, they can reveal the thoughts and actions of those most involved in a company’s data science and artificial intelligence goals and initiatives. These data executives have identified the following top five emerging concerns that demand your immediate attention:
1. Generative AI is exciting, but it needs to be useful. As we mentioned, generative AI is gaining a great deal of interest from consumers and businesses. However, does it actually bring any financial benefit to the companies who use it? According to the poll results, there is a lot of enthusiasm about the technology, but value has not yet been realized for the most part. Many respondents think generative AI has the potential to be a game-changer; in the AWS poll, 80% of participants indicated they thought it will change their businesses, and in the Wavestone survey, 64% said it was the most revolutionary technology in a generation. The vast majority of respondents to the poll are also making more investments in technology. Nonetheless, the majority of businesses are still only doing departmental or individual experiments. Just 6% of businesses in the AWS survey and 5% of businesses in the Wavestone survey, respectively, reported using generative AI in production.
According to surveys, there is a lot of enthusiasm surrounding generative AI, but the technology has not yet shown to be extremely useful.
Naturally, additional funding and organizational reform will be needed for generative AI production deployments—not simply experiments. Redesigning business processes and retraining employees—or, in a small number of circumstances, replacing them with generative AI systems—will be necessary. It will be necessary to incorporate the new AI capabilities into the current technological framework.
Data-related changes, such as integrating various sources, enhancing data quality, and curating unstructured content, may be the most significant ones. 93% of participants in the AWS study agreed that having a solid data strategy is essential to reaping the benefits of generative AI; however, 57% had not yet made any modifications to their data.
2. Industrial data science is replacing artisanal data science. Businesses feel pressure to produce data science models more quickly. An more industrialized activity that was formerly artisanal. To boost productivity and deployment rates, businesses are investing in platforms, feature stores, machine learning operations (MLOps) systems, processes and techniques, and other tools. MLOps systems track the performance of machine learning models and determine if they continue to make accurate predictions. The models may need to be retrained with fresh data if they aren’t.
The once-artisanal process of creating data models is becoming increasingly mechanized.
While some companies are already creating their own platforms, the majority of these capabilities are sourced from outside providers. Reusing existing data sets, features, or variables, even entire models, is likely the biggest contributor to data science productivity, even though automation (including automated machine learning tools, which we address below) is helping to boost productivity and enable wider participation in the field.
3. There will be two main versions of data products. Eighty percent of data and technology leaders in the Thoughtworks poll stated that their companies were either using or contemplating adopting data products and data product management. When we refer to a “data product,” we imply a software package that combines data, analytics, and AI for use by internal or external clients. Data product managers oversee it from inception to implementation (and continuous improvement). Systems that advise consumers on what to buy next and pricing optimization tools for sales teams are two examples of data products.
However, companies have two distinct perspectives on data products. Less than half of the respondents (48%) claimed that the idea of data goods incorporates analytics and AI capabilities. Thirty percent of people think that analytics and AI are different from data products, and they probably only use the word to refer to reusable data assets. Merely 16% of respondents said they never consider analytics and AI in relation to products.
Given that analytics and artificial intelligence (AI) are the means by which data is rendered useful, we lean towards characterizing data products in this way. However, the most important thing is that an organization defines and talks about data products consistently. A definition that maintains many of the beneficial elements of product management can also be used if a business chooses a mix of “data products” and “analytics and AI products.” However, if the description is unclear, corporations may not know exactly what product developers are expected to produce.
4. Data scientists won’t be as attractive. Because of their ability to make data science initiatives successful in every way, data scientists—dubbed “unicorns” and the owners of the “sexiest job of the 21st century”—have suffered a decline in their star power. Several developments in data science are giving rise to substitute methods for handling significant portions of the workload. The emergence of allied positions that can handle specific aspects of the data science problem is one such shift. Data product managers oversee the entire project, translators and connectors collaborate with business stakeholders, machine learning engineers scale and integrate the models, and data engineers maintain the data.
An further aspect diminishing the need for expert data scientists is the growth of citizen data science, in which astute businesspeople with a background in mathematics develop their own models or algorithms. Most of the labor-intensive work may be performed by these people using AutoML, or automated machine learning techniques. The Advanced Data Analysis modeling feature in ChatGPT is even more beneficial to residents. It can handle almost every step of the model development process and explain its operations with a brief prompt and an uploaded data set.
Naturally, there are still a lot of data science-related tasks that call for experts in the field. Tasks like creating completely new algorithms or deciphering the operation of intricate models, for instance, still exist. The role will still be important, but maybe not to the same extent or with the same level of glitz and power as before.
5. Leaders in data, analytics, and AI are growing less lonesome. Over the course of the previous year, we noticed that a growing number of companies were reducing the number of “chiefs” in charge of technology and data, such as chief data and analytics officers and occasionally chief artificial intelligence officers. Despite growing in popularity, the CDO/CDAO position has historically been associated with short tenure and a lack of clarity regarding duties. The roles played by data and analytics executives are not disappearing; rather, they are becoming more and more integrated into a larger range of data, technology, and digital transformation responsibilities overseen by a “Supertech leader” who typically answers to the CEO. Chief digital and technology officer, chief information officer, and chief information and technology officer are titles associated with this position; Sastry Durvasula at TIAA, Sean McCormack at First Group, and Mojgan Lefebvre at Travelers are a few real-world examples.
The ThoughtWorks survey’s main focus was on this evolution of C-suite roles. Data leaders and technology executives made up 87% of the respondents, who agreed that people in their organizations are either completely or somewhat confused about where to turn for data- and technology-oriented services and issues. A significant proportion of C-level executives reported that there is little to no collaboration between tech-focused leaders in their own companies, and 79% acknowledged that a lack of communication had previously hampered their organization.
We think there will be more of these broad tech executives in 2024, with the capacity to generate value from the data and tech experts under their direction. Because this is how businesses interpret data and use it to benefit both consumers and staff, they will still need to place a strong emphasis on analytics and artificial intelligence. Above all, these leaders will need to be extremely business-oriented, capable of debating strategy with their peers in senior management, and able to convert that strategy into the systems and insights that enable it to be implemented.