Uber is developing a payment platform for India that enables operations teams to more seamlessly collect and distribute cash and digital wallet payments to drivers. In this article, San Francisco-based software engineer Yijun Liu reflects on his experiences working with the Uber India Engineering team in Bangalore to architect this revamped payment system.
With tech offices around the world, Uber engineers are responsible for building new features and systems that improve rideshare, new mobility, food delivery, and other services enabled by our platform. Our Uber Engineering Blog highlights some of these efforts, giving technical explanations of our work that can serve as useful examples to the engineering community at large.
Throughout 2019, we published articles about front-end and back-end development, data science, applied machine learning, and cutting edge research in artificial intelligence. Some of our most popular articles introduced new open source projects originally developed at Uber, such as Kraken, Base Web, Ludwig, and AresDB. Likewise, we shared articles from Uber AI covering research projects such as POET, EvoGrad, LCA, and Plato, and original research on our new research publications site.
Along with our technical articles, we offer a look at what it’s like to work at Uber through interviews with engineers and profiles of offices and community building programs…
Styles are the conventions that govern our code. The term style is a bit of a misnomer, since these conventions cover far more than just source file formatting—gofmt handles that for us.
The goal of this guide is to manage this complexity by describing in detail the Dos and Don’ts of writing Go code at Uber. These rules exist to keep the code base manageable while still allowing engineers to use Go language features productively.
This documents idiomatic conventions in Go code that we follow at Uber. A lot of these are general guidelines for Go, while others extend upon external resources:
All code should be error-free when run through
go vet. We recommend setting up your editor to:
go vetto check for errors
You can find information in editor support for Go tools here: https://github.com/golang/go/wiki/IDEsAndTextEditorPlugins
This article is the first in a series covering how Uber’s mobile engineering team developed the newest version of our driver app, codenamed Carbon, a core component of our ridesharing business. Among other new features, the app lets our population of over three million driver-partners find fares, get directions, and track their earnings. We began designing the new app in conjunction with feedback from our driver-partners in 2017, and began rolling it out for production in September 2018.
In early 2017, Uber made the decision to rewrite our driver app. This is the sort of decision that Joel Spolsky, the CEO of StackOverflow, once called “the single worst strategic mistake that any software company can make.”
Rewrites are incredibly risky, resource-intensive, and take a long time to deliver a tangible benefit for users. For this particular rewrite, hundreds of engineers contributed in some capacity, not to mention designers, product managers, data scientists, operations, legal, and marketing. In practice, our rewrite took a year and a half to implement and roll out globally.
Our case is an extreme example of a question that engineers in all organizations face. If you are an engineer working for a start-up and are considering rewriting some code or a feature, you might ask, “How much of our runway are we burning?” If you are working on a small team in a large organization, you might ask, “Are these changes worth the features we are not building?” A good engineer and a good team will look at these broader questions before they take on the challenge of a rewrite.
So, while the rewrite process involved a number of important technical decisions (to be covered in future articles), the decision to rewrite involved a combination of both technical considerations and broader business concerns. While these questions are hard to answer, good answers to the above questions will help you justify a rewrite to your organization or team.
Ultimately, these decisions do not get made in a vacuum. We did not make the decision to rewrite the app as a result of theoretical architectural thinking (“our code might be better, if only we…”), but rather as a result of an intensive, three-month research process that involved hundreds of pages of documentation and broad, cross-organizational buy-in. In the following sections, we discuss our decision to rewrite the Uber driver app and what we discovered as a result of this process.
The early architecture of Uber consisted of a monolithic backend application written in Python that used Postgres for data persistence. Since that time, the architecture of Uber has changed significantly, to a model of microservices and new data platforms. Specifically, in many of the cases where we previously used Postgres, we now use Schemaless, a novel database sharding layer built on top of MySQL. In this article, we’ll explore some of the drawbacks we found with Postgres and explain the decision to build Schemaless and other backend services on top of MySQL.
The Architecture of Postgres
We encountered many Postgres limitations:
- Inefficient architecture for writes
- Inefficient data replication
- Issues with table corruption
- Poor replica MVCC support
- Difficulty upgrading to newer releases
Location and navigation using global positioning systems (GPS) is deeply embedded in our daily lives, and is particularly crucial to Uber’s services. To orchestrate quick, efficient pickups, our GPS technologies need to know the locations of matched riders and drivers, as well as provide navigation guidance from a driver’s current location to where the rider needs to be picked up, and then, to the rider’s chosen destination. For this process to work seamlessly, the location estimates for riders and drivers need to be as precise as possible.
Since the (literal!) launch of GPS in 1973, we have advanced our understanding of the world, experienced exponential growth in the computational power available to us, and developed powerful algorithms to model uncertainty from fields like robotics. While our lives have become increasingly dependent on GPS, the fundamentals of how GPS works have not changed that much, which leads to significant performance limitations. In our opinion, it is time to rethink some of the starting assumptions that were true in 1973 regarding where and how we use GPS, as well as the computational power and additional information we can bring to bear to improve it.
While GPS works well under clear skies, its location estimates can be wildly inaccurate (with a margin of error of 50 meters or more) when we need it the most: in densely populated and highly built-up urban areas, where many of our users are located. To overcome this challenge, we developed a software upgrade to GPS for Android which substantially improves location accuracy in urban environments via a client-server architecture that utilizes 3D maps and performs sophisticated probabilistic computations on GPS data available through Android’s GNSS APIs.
In this article, we discuss why GPS can perform poorly in urban environments and outline how we fix it using advanced signal processing algorithms deployed at scale on our server infrastructure.
Matt Ranney talks about the limits that some companies have encountered in their large microservices deployments and some non-microservices approaches to those same problems. He also talks about the non-microservices systems that Uber is building to maintain developer productivity with a large and growing engineering team.
Uber posted an article detailing how they built their “highest query per second service using Go”. The article is fairly short and is required reading to understand the motivation for this post. I have been doing some geospatial work in Golang lately and I was hoping that Uber would present some insightful approaches to working with geo data in Go. What I found fell short of my expectations to say the least…
Uber Engineering’s Schemaless storage system powers some of the biggest services at Uber, such as Mezzanine. Schemaless is a scalable and highly available datastore on top ofMySQL¹ clusters. Managing these clusters was fairly easy when we had 16 clusters. These days, we have more than 1,000 clusters containing more than 4,000 database servers, and that requires a different class of tooling.
Initially, all our clusters were managed by Puppet, a lot of ad hoc scripts, and manual operations that couldn’t scale at Uber’s pace. When we began looking for a better way to manage our increasing number of MySQL clusters, we had a few basic requirements:
- Run multiple database processes on each host
- Automate everything
- Have a single entry point to manage and monitor all clusters across all data center
The solution we came up with is a design called Schemadock. We run MySQL in Dockercontainers, which are managed by goal states that define cluster topologies in configuration files. Cluster topologies specify how MySQL clusters should look; for example, that there should be a Cluster A with 3 databases, and which one should be the master. Agents then apply those topologies on the individual databases. A centralized service maintains and monitors the goal state for each instance and reacts to any deviations.
Schemadock has many components, and Docker is a small but significant one. Switching to a more scalable solution has been a momentous effort, and this article explains how Docker helped us get here.
With the company valued at over $62 billion and running in over 523 cities around the world with plans to launch a fleet of autonomous cars, Uber is constantly looking to disrupt and innovate. And they are leading the pack when it comes to the use of machine learning in their strategy.
At the Kaizen Data Conference hosted at Galvanizes’ San Francisco campus, Uber’s Head of Machine Learning, Danny Lange, discussed past, present, and future advancements in his area of expertise. This compelling talk covers artificial intelligence’s central role in business disruption and innovation, from Uber’s self-driving cars to emerging new technologies. Take a look: