ML modeling teams at Twitter face a variety of uniquely hard yet fundamentally related machine learning problems. For example, tasks as different as ad serving, abuse detection, and user timeline construction all rely on powerful representations of user and content entities. In addition, because of Twitter’s real-time nature, entity data distributions are constantly in flux, so these representations must be frequently updated. By generating high-quality, up-to-date representations (embeddings) and sharing them broadly across teams, Twitter decreases duplication of efforts and multiplicatively increases cross-team modeling productivity. Abhishek Tayal offers insight into how Twitter’s ML platform team, Cortex, is making entity embeddings a first-class citizen within Twitter’s ML platform by commoditizing tools and pipelines that create high-quality, custom, regularly retrained, benchmarked, and centrally hosted embeddings. Abhishek also highlights various use cases of how teams at Twitter are using entity embeddings in their ML stack.
