Patrick Bajari Q&A

Tell us about your past, working at Amazon for 13 years.

Amazon gave me a call in 2010, and they said we’re looking for someone to build an economics team for us. I was looking to take a leave and I thought it would be more interesting working for Amazon than, for example, working in Washington, DC on policy or antitrust where you are doing analytics to support regulators and politicians. Helping to contribute to the future of e-commerce seemed much more exciting to me.

By about 2012, it was clear that the role of data in industry was going to change drastically. What we now call scientists (people with doctorates in Machine Learning, Stats, Computer Science, etc.) that were using data inside of code were being highly innovative. I could see that a wave was changing technology and I decided to take a chance, give up tenure and stick around Amazon as a tech VP.

Amazon was like a candy store of data-driven tech problems to work on. Everything ranging from supply chain to search to pricing and even legal or HR. The company was always looking to push boundaries in terms of using data in totally new applications (e.g. not many people were using AI to drive large-scale supply chains around 2015), which have huge implications for operating businesses but received relatively little attention.

I initially started out by building in economics and helped to form economics as a job family within tech that is still quite vibrant and growing today. About eight years into my career, I was asked to help lead a worldwide Machine Learning and Engineering team that we called Core AI. The team was composed of about 120 scientists in Machine Learning, Computer Vision, Natural Language Processing, Statistics, Operations Research and Economics. We also had a large number of software developers who were interested in learning to apply cloud computing to this boundary. We were allowed to invent solutions across the company. The team was incredibly bright, I learned a lot and it was an honor to lead.

I also served as a part of our executive team reporting to the CEO of Consumer. While I did not have the large budgets and managerial problems of my peers, I think my data, how industries and the economy are moving and just being fussy about numbers and science was helpful to the team.

How did you start out in your career?

I got my doctorate at 27. My first job was in the econ department at Harvard. I went to Stanford pretty quickly because my field was pretty new, and Stanford was a place that was getting momentum. For the next decade, I did the thing that most college professors work on — teaching, advising students, publishing papers, attending conferences and so forth.

After I got tenure, I went back home and taught at Minnesota. My family was in Minnesota since the 1860s. They were some of the first settlers of the state, and I have really deep roots there. I swore I’d never leave again because I liked living there.

Amazon offered me a position during a two-year leave of absence, but my intention was to go back. However, when I saw the data wave blowing up in tech, I knew I had to stay and be a part of it. I gave up tenure and dove all in.

When I gave up tenure to go into Amazon, people thought I was nuts. People didn’t give up tenure to take jobs in tech companies in economics. But I never like to follow the beaten path.

Describe your specialties, strengths and interests.

I’d describe myself as someone who likes to solve ‘last mile’ problems and tries to search for highly valuable but overlooked problems. In AI and ML, many people spend their time standing up new tools. However, what businesses need are workable solutions to specific problems such as ‘How should I plan my inventory, how should I set prices, how should I provide benefits to my employees?’ and so forth. I start by asking what is the customer facing or business problem I would like to solve. What functionality or utility would I like to give them? How would it be useful? After I form a crisp vision of the utility to a user, I then ask what data, engineering and scientific tools we need to build out that solution.

I do work on some problems that are more purely methodological, where the tech seems useful but we don’t quite know what to do with it yet. That is proper diversification of a portfolio, as they would say in finance. However, I wanted to always know at the end of the year my team was paying for itself with measurable value that would be verified by our finance partners.

One of the things I do is called ‘Hands off the Wheel.’ What we do is take a process path within the management of a company, such as your planning of capital investments. There is already a team that is working on this problem. I identify a set of ‘touch points’ at which key decisions are made. And at each touch point, I have a couple of scientists shadow the manager who is making that decision. They start by translating Excel into Python or a statistical package. We take the data that is already being used and put just a little bit more formal and rigorous modeling around it. We have the tech team work closely with the business partner, going back and forth, to make sure that the code they are writing assists in making decisions in a more standardized, data-driven and scientific manner. The business owners often really appreciate that help since a new set of eyes on the problem helps them to look at the data differently and make better decisions.

We then start to publish forecasts and decisions by pulling together the collection of models for the individual tech teams. We begin to not just publish, but recommend decisions. The business managers are given a rules and decision engine where they can accept or reject the recommendation of the model. If they reject the recommendation of the model, we ask for a reason code or explanation. These rejections are then a defect for the tech teams to reduce. They go to the owners, deep dive the reasons for the lack of acceptance and harden the models and recommendations. Over time, the acceptance rates get higher and higher. You may not fully automate, but you can automate a lot of tedious spreadsheet-driven decisions that managers did not enjoy anyway. They turn into auditors to the tech systems and have time to focus on more high value tasks such as improving the customer experience.

But in this process you need to be humble, learn the details, do your homework and put one foot in front of the other. If you do not, the management will not trust the tech and you will likely have a wasted investment.

This is what I call digital transformation. It’s hard work, but do it on a problem for a few years and it becomes transformational.

I think there’s a huge market for true AI consulting that helps firms digitally transform. This is a hard problem and firms fail at it more often than they succeed. I am excited by joining Keystone and helping to spread the use of data, science and engineering to do our work more rationally and productively.

Keystone is starting to do really large, enterprise-level contracts with multinationals of this form and I find it exciting to help lead these efforts.

Digital transformation, firms fail at it 92% of the time. You need to be humble and work backwards from real solutions to make it work.

What are some of the challenges and obstacles in digital transformation?

You need to get people with very different skills to collaborate as a team. This includes managers, salespeople, operations, software developers, scientists and many more. Business is a team sport. If we do not work together, we risk failing. And it is a new capability for most firms to help people with these very different skill sets to work together.

What drew you to Keystone and what niche are you filling here?

I want to help build a real AI consulting practice. These are very difficult technologies to deploy in the field. I have been a teacher in many ways in my life. I want to sketch a path to help solve a problem for a firm and how to staff a team that can help them make that solution their own and allow them to continuously iterate and improve on that solution. I helped to build hundreds of small teams around Amazon, some of which became much bigger teams. People are the best way to do real tech transfer. And we need to do the unselfish thing and get out of the way to let a team that we helped build own its own work.

What new developments do you see ahead in the world of digital transformation?

I am hoping that people will begin to get much broader about the set of problems to which they apply AI and how to have ‘humans in the loop’ in an appropriate manner. A lot of the most visible AI solutions are in fields related to advertising (e.g. the work of big platforms like Google or Facebook). But advertising is only 1% of GDP. We need to digitally transform agriculture, transportation and warehousing, health care and so many other industries that make up the other 99% percent of GDP.

I was teaching at Stanford from 1998-2003. At that time I saw a large investment in tech talent and then a movement of that talent to the center, east and south of the country in the so-called ICT wave. I am seeing a similar dynamic occur right now where West Coast tech workers with skills in Cloud and AI are moving into new industries and geography. Previously, a health care firm in Minnesota would not have been able to attract a large team of AI and Cloud people. One of my former Principal Engineers just moved to a very large health care company to do just that.

I think we will see more of this. And it will involve the movement of workers and technologies across industries and space. In many ways, we hope to be brokers and facilitators of that dynamic.

Do you have anything to add?

I really like to integrate across fields. In business you have to work together and a more diverse team often creates better solutions. I try to learn a little bit about a lot of fields, everything from CV, NLP, Edge Computing, the latest in the Cloud to Computer Vision, NLP, LLMs, Statistics, OR and Economics. I try to ask questions about what I don’t understand — no mortal can keep on top of that many topics at great depth. However, as a leader you can gain some basic fluency and help encourage the team to mix and match methods where appropriate to create diverse and more inventive approaches.

I also try to keep myself humble (although I fail occasionally). I’m from Minnesota. I was taught to keep your head down, work hard, be sincere and good things can happen with perseverance.