Podcast: When your face is your ticket
In the third installment in this latest series, Jennifer Strong and the team at MIT Technology Review jump on the pitch to unpack how much is changing.
- Donnie Scott, senior vice president of public safety, IDEMIA
- Michael D & # 39; Auria, vice president of business development, Second Spectrum
- Jason Gay, sports columnist, Wall Street Journal
- Rachel Goodger, director of business development, Fancam
- Rich Wang, director of analytics and fan engagement, Minnesota Vikings
This episode was reported and produced by Jennifer Strong, Anthony Green, Tate Ryan-Mosley, Emma Cillekens, and Karen Hao. We are edited by Michael Reilly and Gideon Lichfield.
Stark: I'm in the Queens neighborhood near a huge stadium complex called Citi Field. It's home to the New York Mets because it's off season for baseball. Right now everything is locked in and all you can really hear is the rush hour traffic.
However, if you look up at the edge of the stadium, where thousands of fans eventually return, you can see some of the hardware that supports the team's facial recognition. These cameras are designed to detect faces that have been banned from the site – people like Ticket Scalper, people who walked into the field, even committed crimes in the parking lot, and this system is powered by one of the biggest names in face recognition – NEC. It can measure things like ears – and it still works with people who wear masks, hats, and sunglasses.
And then when you get to the turnstiles there is another face system from a company known for airport security – Clear – and that for ticketless entry. Basically, you can use your face as a ticket. When you get inside there is a payment system in place on a concession – which means you can buy a beer with your face on if you want.
But when you get into your place, things get really interesting. Even before the pandemic, participation in baseball games was declining. In fact, this stadium has about 15,000 fewer seats than the one it replaced. On the one hand, the stadiums try to make the experience as safe and hassle-free as possible, but they also try to learn as much as possible about who these people are in the stands and so they also work with facial recognition. I'm Jennifer Strong, and in this latest installment in our miniseries, we'll look at how this and other tracking systems are transforming the sports experience in the stands and on the pitch.
(Sound of Chicago White Sox at Milwaukee Brewers (Anchor): Ok, we'll play ball again. Two out. 1st inning. No score. And the batsman will be Harold Baines with a 7-game series …)
(Sound of Chicago White Sox at Milwaukee Brewers: audience applause)
Strong: For decades, the hustle and bustle on television or radio was the first choice to do sports. Often times, it meant tuning in for hours like that 1984 Major League baseball game between the Chicago White Sox and the Milwaukee Brewers.
(Sound of Chicago White Sox at Milwaukee Brewers (Anchor): That's deep in midfield. Back. It could be out of here. Manning looks up. It's out of here! A home run for Harold Baines. The Soxs win 7 -6 in the longest Game of American League History.)
Stark: The game lasted eight hours and six minutes. And it had to be completed over two days. But sports watching looks very different today. A person's attention span is measured in seconds and shrinks. Millions of people still tune in to watch, but about a third are streaming it on mobile devices. And of those who still watch TV, 80 percent do so while using a second device to browse stats, collect live scores, message other fans, and watch related videos. The segment of fans who personally take part in games is now considered a high-quality customer. And that's another place that facial recognition comes in.
(Sound from CNBC Newspaper (Anchor): And if you've been upset about Facebook invading your privacy, you may not want to attend a major sporting event.)
(Sound from CNBC news program (Eric Chemi): New high-tech cameras can now take a high-resolution photo of anyone, anywhere, any minute of the game.)
Strong: Facial data collected from stadiums by companies like Fancam is now being used to provide insights into fan demographics such as age, gender, and race. Panoramic cameras are capable of capturing images in such detail that you can (from a bird's eye view of a stadium) zoom in on the stands and on a single person, and still see nuances like a smile and writing on their shirt, even the texture of their jacket. And now you can also quickly calculate the percentage of people wearing masks – as in the case of the NFL's Minnesota Vikings.
Wang: This is new to everyone. We're still trying to figure out exactly how to enforce these mask rules and how to monitor and track them.
Stark: Rich Wang is her Director of Analytics & Fan Engagement. He makes a zoom call and shows them how to use Computer Vision.
Wang: Even if you look at this graphic. The lowest point is that 87% of people wear their mask most of the time and most of the game. People you know are behaving and enforcing the mask rule. So these are really positive storylines that will continue to support our case of growing fans
Goodger: To be able to use those stats to reopen venues and bring fans back to the stadium. And then just as a protection, when the fans are back in the stadium and are using some of these metrics in addition to mask usage, they can also use the information about the section capacity.
Stark: And this is Rachel Goodger, the director of business development at Fancam.
Goodger: Obviously fans have been assigned a seat when they go back to the stadium and the fans are socially distant. But what happens when the fans move around the stadium and a section is overloaded? They know that we can notify employees in real time and that they can see that information and say, ok, we need to break this section up a bit. And then so the teams can look back after every single game and say, "Wow, we did a great job today." Or, "Wow, we really need to work more on using masks in the lower or upper objective of this section," and such Things. I think it is dates that will be very important not only to reopen these stages, but to keep them open in the future.
Strong: The company sells data back to the sports teams who use it to drive their marketing. This affects everything from the music played in the stadiums to the ads people see during and even after the game ends.
Scott: You're going to start seeing the data you want to share coupled with the identification technology to improve predictability.
Strong: Donnie Scott is Senior Vice President, Public Safety at IDEMIA. It develops AI-driven identity and security solutions for all types of companies.
Scott: And that would be anything from a digital driver's license on your phone to a physical driver's license, a credit card, to an electronic payment mechanism.
Strong: They also make biometric technology that recognizes faces, fingerprints or eyes and can be used to verify identity in sports stadiums or other places like airports and theaters.
Scott: So we would essentially embed the technology in their loyalty program, but we would add it, the ability to either link their biometrics – face, fingerprint, iris in some countries they prefer based on face covering and other things, or you mobile device on which you can share your biometric data or the fact that you are the holder of a season ticket with a device at the venue. And that's why you know, when you show up, they know, okay, Jennifer has tickets to this game. They are valid at this point in time. She can go through the gate.
Strong: your goal? Is to be invisible. Identity data is captured by cameras that are hidden like a normal turnstile. It's about creating what is called a frictionless experience.
Scott: Especially when it comes to theme parks, technology is evolving from being something that stands out from the crowd, becoming part of the normal flow and cue of the venue.
Stark: We already unlock smartphones with eyes, fingers and faces, and that has got us used to this idea of biometrics in our daily life. Scott believes this may be why the response to these services has been largely positive.
Scott: You know, I've seen my kids grow up by first opening an Apple device with their thumbprint and then moving on that they felt very abused for not being able to unlock it with their face. And we are all, you know, the last 15 years, 10 years, desensitized to the craziness of it. I think most of society is focused on how it makes my life easier.
Stark: And in a world where verifying your identity is as easy as unlocking a phone, your biometric information can become more important than a passport, car keys, or other physical item we carry with us.
Scott: I think people are really going to get used to technology, how to use it, how to interact with it and what to expect from it because I think we will see it in all walks of life. We'll see when we travel. We'll see when we do business with our government. We will see when we do business in grocery stores where you know sports and concert venues, as well as music parks. So, it becomes such a normal way of life that the access part becomes de facto normal. And then it happens next.
Stark: And what happens next could mean more personal experiences.
Scott: I think the next thing to come will be to enable the fan experience. But after that, how does the fan experience fit into your life? And, you know, this is a concept that is pretty big and broad, but one that, once the first two parts are made possible by technology and by user self-acceptance, are just natural things that come with an improved, mature one Use of go hand in hand with a technology. You could envision an amusement park, head, or character where kids can walk up to their favorite character and be recognized who they are and have an individual experience for them.
Stark: Which is likely to happen on a large scale.
Scott: You could see a future where you arrive at the airport or at the sporting event and that will take you to your parking lot by recognizing your car or telling the airport operator or the airport who you are from your phone, airline or the TSA itself You would have a known time to the gate, right. What is the ideal state in which to say I have a five o'clock flight today based on the predicted waiting times and where we are. I know it will take me 12 minutes to get from the front of the airport through the checkpoint to the gate. And you will have directions along the way, the same experience is done for sports venues and concert venues where you will be guided by the shortest line from parking, you know this line will go move fast because it is biometrically enabled , and then you can determine where to get my desired concessions, how long to have before I have to start walking so that I can sit in my seat beforehand I think these kinds of secondary benefits will come pretty quickly, though the venues are instrumented in order to recognize and identify people.
D’Auria: I think there is a great opportunity to make the kind of sports fan experience more engaging and impactful. And I just think where we are at the beginning of it. I'm Mike D & # 39; Auria and I'm Vice President, Business Development at Second Spectrum.
Stark: The company provides tracking data and analysis software for professional sports leagues such as the NBA and Major League Soccer. A range of cameras no larger than your standard security camera provide unparalleled machine understanding for any game.
D’Auria: The core of this technology is computer vision, which is carried out through these camera feeds. This is intended to track the movement of each player and the ball 25 times per second. So you can think about a typical NBA basketball game and take millions of data points that didn't exist before and use that to build a range of products or experiences on it that can change the way we see and interact with sport. really change.
Strong: These data points are quickly analyzed with AI, which allows predictions such as the probability that a player will pocket a three-pointer, while the game is still running. This data is also used to provide fans who watch TV with a more personal, interactive viewing experience.
D’Auria: In this last NBA final, we carried out what is known as a video expansion, essentially in real time. For example, you could use the shot probability model there. And while the game is playing, you can include a 3D shot probability bubble over the head of each offensive player in the video, which is updated in real time. We can represent the game that is being played. So if you are trying to learn a little about the game, have a tutorial or what it feels like to have a trainer sitting next to you. You know, or if you just want to have some fun or play a bit, you know every time someone hits the ball you can see a bolt of lightning on the backboard. As such, each of these experiences may not be for everyone, but I think we're going to venture into a world where live sports can really be personalized the way you want to see it.
Stark: And access to data has changed the way coaches train their players.
D’Auria: So if you take a step back and think about how data is traditionally collected in sport, people either sit in the stands or watch the game on TV and code manually. That was a shot. That was a pass. It was a pick and roll action. From this type of underlying tracking dataset, you can apply machine learning to automate the whole process.
Stark: This automation allows all of this data to be adapted to the feature film. Coaches, general managers, and analysts can then look through it with a software tool that works like a search engine.
D’Auria: And so you can ask very complicated questions for people who work on an NBA team, or ask very detailed questions about the game. And with a few keystrokes, a few clicks of the mouse, you get a very accurate answer in the data visualization and an automatically generated playlist from, for example, Anthony Davis, LeBron James, pick and roll from the right wing where the defense freezes and Anthony Davis rolls and someone rolls him off marked on the weak side. And so LeBron James takes a jump shot and does it. You know, every time this combo has happened over the course of these guys' NBA careers, you can get the very accurate sentence in seconds and then use it for your coaching purposes. And now someone at the team level can spend their time saying, well, I have this video or information. How can I help a coach incorporate this into their game plan? Or how can I help my players learn something new on the pitch? And so the workflow shifts to teaching and implementing versus data collection and manual work.
Stark: And he says that the roles of these machines in the game could shift from assistant coach to assistant referee in the next few years, adding context and nuance to difficult calls.
D’Auria: I mean, we've seen that in a couple of other places we work. We're giving the soccer example that you now have technology that helps with goal, not a goal call, right? You see this in tennis, when computer systems are used to judge whether a ball is over the line or, as you know, inside or out of bounds, and can do so with an accuracy that is frankly better than that of one Linesman or a referee who could have a really difficult angle to see if literally every inch of the ball missed. You are starting to see this with the offside line in football as well. So I think the first place this happens is to expand or support a referee's skills. So you can think about providing an arbitrator and an additional data source or, as you know, additional validation of one of your decisions.
Strong: Since the system can already identify players based on their jerseys, Second Spectrum does not need to use face mapping or face recognition. But it's useful for analysis. And that doesn't just apply to capturing faces. At the moment, players appear in the system as points on a map. And as their camera systems improve, those points could turn into full skeletons. Additional details such as the elbow angle in real time could lead to even more precise shot predictions. However, not everyone is on board.
Gay: You know, one sport I follow that I find fascinating is cycling, and cycling is a sport that has actually been talked about for a long time. Remove technology.
Strong: Jason Gay is a sports columnist for the Wall Street Journal.
Gay: Technology now in cycling can say okay, if you want to win this race or catch up with that person, you have to put in X effort for X minutes. And you actually have this data directly on an on-board computer, on a bike in front of you, which tells you exactly what to do. Now. This is such an amazing thing. But it's not particularly human either, is it? It seems a bit clinical and it's created what many people think is a bit of a dry racing style where people are data driven and use their heads too much as opposed to their hearts. The French have an expression of verve. They love to see races won with verve, which basically means our gut instinct. And so there was talk of what happens if we take these computers away from drivers and get them to use their heads in their hearts to ride a bike. Now there is a security consideration that comes with that, right? You actually want this information to often create a safer experience for a driver, but it's fascinating that in certain cases the technology has gotten so good as to maximize the effort or tell an athlete what effort it takes to get them it's I'm starting to withdraw from it.
Stark: And for sports that use this technology, the way the game is played changes.
Gay: Here's an example from baseball, and pretty often we see a manager come to a hill and remove a pitcher from a game, even though the pitcher throws very, very well that day. The reason he removes them is because the data shows that this pitcher has a tendency to break down at some point. It's almost like a car tire or something. And they just say, well, at this point in the game, that pitcher will historically stop playing at the high level we need from him. So we're going to take this step. We remove the courage to say there, well, it's rolling today, let's just let go of it. You rely on the numbers.
Strong: Data-driven game strategies are also changing the way teams recruit. Like basketball, where players who can take a three-point shot (once considered a gimmick by the NBA) are now considered extremely valuable.
Gay: The reason for this is because basketball teams found out from their numbers that a three-point shot is a more efficient shot. You'd rather do that three-point shot than a longer two-point jump. And so you prioritize the three pointers in a criminal offense. The most extreme example of this is the Houston Rockets, who have a perennial MVP candidate in James Harden who often takes three pointers after three pointers in a game because it's an efficient way for them to play.
(Sound of Houston Rockets at Los Angeles Clippers (announcer): Harden, no one around, sits all the time and nails the three-pointer! Steps back, open three, understood! James Harden steps back and puts up a three, jumps and falls through!)
Strong: Technology also plays an assistant role in places like the Dallas Mavericks locker room.
(Sound from Marc Cuban's video at Dallas Mavericks (Cuban): When a player walks in or someone walks in, we have facial recognition. It will take a picture of you and it will say & # 39; ok & # 39; here comes Marc or here comes Dirk & # 39;)
Strong: Marc Cuban is its owner.
(Audio from Marc Cuban's video at Dallas Mavericks (Cuban): And notes are made for coaches for each player or staff member: Here's what is expected of you and to tell you what's going on. For everyone We don't know that it will be ehh-ehh-ehh. ")
Stark: And it's not just basketball. The use of AI to find the most efficient pattern of play is growing in all sports. And there is also a role for face recognition. The same face mapping that appears when you look directly at your phone to unlock it can also help coaches see what players are focusing on during the game.
Gay: I mean, this is an incredibly important thing for a football quarterback. If somehow you could be able to render what a football quarterback sees or, more importantly, not see it, not see a downfield. Well, you could see that every quarterback, soccer team, is instantly useful. But it also applies to a point guard or someone playing left attack or someone who catches a baseball team. There are numerous games that have huge ramifications when you can see what an athlete sees or doesn't see on the court, which is probably the most essential.
Stark: In the next episode, we'll end our miniseries with a look at how face mapping is changing the shopping experience. And spoiler alert – it goes way beyond just identifying who's in the business
Guive Balooch: To really be able to try on make-up virtually virtually with augmented reality, you have to recognize where the eye is and where the eyebrows are. And um, it has to be so accurate that when the product is there it doesn't look like it's not right on your lip and people's lips can vary in shape, color between your skin tone and your lip, can also be very different. Hence, you need an algorithm that will recognize it and make sure it works.
Strong: This episode was reported and produced by myself, Anthony Green, Tate Ryan-Mosley, Emma Cillekens, and Karen Hao. We are edited by Michael Reilly and Gideon Lichfield. Thanks for listening, I'm Jennifer Strong.