Machine learning methods could provide a better experience for video streaming services
09-20-2022
Is it possible to improve video streaming services without using paying-customers as test subjects?
Currently, when video streaming services want to develop app improvements, they must elect a large number of live users to beta-test before deciding whether it should be deployed to all users. This type of testing can impact the performance though the services are doing it to improve customer experience.
Bruno Ribeiro, an assistant professor in the Department of Computer Science at Purdue University, argues that rather than beta-testing on live users, “We could use machine learning methods on data logs of past sessions to predict the impact of new app improvements, rather than testing new designs on live users."
Counterfactual questions and ML tools
The process is not only less expensive but it could also improve user experience. However, this requires novel machine learning tools that can answer counterfactual questions, such as, "Would the service that occured yesterday be improved if we had used algorithm X rather than algorithm Y in the app?"
“The holy grail is to predict the performance of app changes in what-if scenarios, without the need to experiment on live users,” said Ribeiro.
In order to develop these new methods, Ribeiro and his collaborator, Professor Sanjay Rao in the Purdue's Elmore Family School of Electrical and Computer Engineering, are developing new causal machine learning methods for streaming applications, since existing machine learning tools are unable to answer such questions.
Industry-wide concern
“This is an industry-wide concern,” said Ribeiro. “Designers think, ‘What would happen if we had chosen a different algorithm? Could it have been better this way?’" He added, "If the only way to try new algorithms is experimenting on live users – which is expensive and impacts user satisfaction – the industry will become a conservative and stifling innovation.”
There is another advantage to this approach. These new methods would give developers a so-called, time machine. For example, a video streaming service might launch a new series that is going to be viewed by millions. This is a series with wide appeal and marketed extensively, but when the launch arrives, it doesn’t go so well – most users couldn’t access the stream and the users that could, experienced poor service. Developers ask, “What could we have done differently to improve service?” but currently there is no time machine available to them to test their ideas on live users (since the event is now in the past).
“Examples like this show that we can’t go back in time and ask, ‘What if the app used algorithms A rather than B?’” said Ribeiro.
Cutting costs with ML tools
“The streaming services are not going to launch another series (i.e., at astronomical costs) just to try a new algorithm B on their app," said Ribeiro. Instead, he believes the future is in developing counterfactual evaluation methods that allow developers to go back to the data that was collected and fix problems for user experience and service.
This work, Answering counterfactuals from offline data for video streaming by Ribeiro and his co-PI, Professor Sanjay Rao was funded by a 2021 Amazon Research Award.
“The work and research we are currently doing applies not only to video streaming, the principles can be applied in other networking domains as well,” said Ribeiro. “With most cloud services or apps, you can use this technique to improve services.”
“The main takeaway and benefit to people is the ability to have a service improve while avoiding user experimentation, and being able to significantly improve user experience in rare events," said Ribeiro. "This is why we developed this new concept and are thankful to Amazon for the support.”
Abstract
When deploying Internet video streaming algorithms, designers may often ask counterfactual questions such as “what would the performance have been if a new Adaptive Bit Rate algorithm were used”, or if the algorithm had optimized for a different Quality of Experience (QoE) metric, or if the users had been served from a server in a different location. Applying machine learning (ML) tools developed for observational tasks on passive trace data to answer counterfactual questions can result in wildly incorrect answers, as our preliminary work shows. A/B testing can impact the performance of live users, and its deployment cost can be prohibitive for iterative development. In this proposal we explore a novel approach, that instruments a deployed video streaming system to collect additional data in a manner that does not impact live users, yet can be used to answer counterfactual questions offline in a bias-free manner. While we focus on video streaming, the principles can be applied in other networking domains as well.
Bruno Ribeiro is an assistant professor in the Department of Computer Science at Purdue University. He obtained his PhD at the University of Massachusetts, Amherst and did his postdoctoral studies at Carnegie Mellon University. His research interests are in invariant & causal representation learning, with a focus on sampling and modeling relational and temporal data. He received an NSF CAREER award in 2020, an Amazon Research Award in 2022, and multiple best paper awards including the ACM SIGMETRICS 2016 best paper award.
About the Department of Computer Science at Purdue University
Founded in 1962, the Department of Computer Science was created to be an innovative base of knowledge in the emerging field of computing as the first degree-awarding program in the United States. The department continues to advance the computer science industry through research. US News & Reports ranks Purdue CS #20 and #16 overall in graduate and undergraduate programs respectively, seventh in cybersecurity, 10th in software engineering, 13th in programming languages, data analytics, and computer systems, and 19th in artificial intelligence. Graduates of the program are able to solve complex and challenging problems in many fields. Our consistent success in an ever-changing landscape is reflected in the record undergraduate enrollment, increased faculty hiring, innovative research projects, and the creation of new academic programs. The increasing centrality of computer science in academic disciplines and society, and new research activities - centered around data science, artificial intelligence, programming languages, theoretical computer science, machine learning, and cybersecurity - are the future focus of the department. cs.purdue.edu
Writer: Emily Kinsell, emily@purdue.edu
Source: Bruno Ribeiro, ribeiro@cs.purdue.edu