Code Impact

This paper details the evolution of Google's Spanner, a globally-distributed database system, from a key-value store to a fully-fledged SQL system.  Key improvements  discussed include distributed query execution, handling of transient failures via query restarts, efficient range extraction for data retrieval, and the adoption of a common SQL dialect. The authors also  explain the transition  ...

The article explores Change Data Capture (CDC), a method for tracking database changes, highlighting its advantages over traditional daily snapshots. It details three CDC implementation approaches: using database triggers (e.g., in PostgreSQL), capturing API requests and using a message broker (e.g., Kafka), and leveraging change streams within a data warehouse (e.g., Snowflake). The article compa...

This research paper introduces DeepSeek-R1, a large language model enhanced for reasoning capabilities using reinforcement learning (RL). Two versions are presented: DeepSeek-R1-Zero, trained purely via RL without supervised fine-tuning, and DeepSeek-R1, which incorporates additional multi-stage training and cold-start data for improved readability and performance. DeepSeek-R1 achieves results com...

This Atlassian blog post details the migration of Jira Cloud's Issue Service from JSON to Protocol Buffers (Protobuf) to enhance performance. The switch involved a phased approach to minimise downtime, creating new endpoints and logic to handle both formats concurrently before a complete transition. The results showcased significant improvements: 75% less Memcached CPU usage, 80% smaller data...

This research paper introduces Hyaline, a novel family of memory reclamation schemes for lock-free data structures in unmanaged C/C++ code. Hyaline leverages reference counting, but only during reclamation, minimising overhead during object access and balancing workload across threads. The paper details Hyaline's design, including a scalable multi-list version and robust extensions to handle s...

This Atlassian blog post details Trello's migration from RabbitMQ to Kafka for its websocket architecture.  RabbitMQ's unreliability during network partitions and high costs associated with queue creation and deletion  prompted the switch. The article compares various queuing systems, highlighting Kafka's  superior failover capabilities and in-order message delivery . Trello implemente...

This podcast explores the field of reliability engineering, tracing its origins at Google with the development of Site Reliability Engineering (SRE). It differentiates reliability engineering from SRE, highlighting its broader applicability across various organisational structures. The podcast outlines four key promises of a successful reliability team: defining service levels (SLA/SLO/SLI), manag...

This podcast profiles Antithesis, a company developing a "multiverse debugger" for large, distributed systems. It traces the history of debugging tools, highlighting Antithesis's innovative approach using deterministic simulation testing (DST) to allow time travel debugging. The podcast includes a Q&A with Antithesis's co-founder, detailing the challenges of debugging large s...

This podcast details the creation of Shopify's interactive Black Friday/Cyber Monday live dashboard, nicknamed "Live Globe". The 2024 version, built by a six-person team in two months, features a spaceship-themed interface showcasing real-time sales data and boasts impressive technical specifications, including peak loads of nearly 30 million database reads per second. The design pro...

This podcast examines the contrasting "wartime" and "peacetime" operating modes in tech companies, drawing on the author's experiences at Uber and observations across the industry. It defines these modes in terms of leadership styles, employee behaviours, and organisational priorities, highlighting the differences in approaches to project management, performance reviews, an...

Jim McCormick's "The First Time Manager" offers a practical guide for new managers, covering essential aspects like communication, delegation, and conflict resolution. The book employs a clear and relatable style, using real-world examples and actionable advice to help readers build foundational leadership skills. While some advice may be general, its comprehensive approach to fundamental manageme...

This research paper details the development and implementation of efficient techniques for processing multiple, similar aggregate queries in data streaming systems. The authors address the challenges of scaling to handle hundreds of concurrent queries, each with potentially different time windows and selection predicates. Their proposed "on-the-fly" methods avoid computationally expensive static q...

The article details how Vercel's platform handles web requests, from initial user input to final response.  Vercel's Edge Network  directs requests to optimal data centres, minimising latency. A multi-layered  firewall system  protects against threats.  Advanced routing  features, including middleware, manage request flow. Finally,  Edge caching  and  Vercel Functions  optimise speed and scalabili...

This research paper introduces Monolith, a real-time recommendation system designed by Bytedance.  Addressing limitations of existing deep learning frameworks , Monolith uses a novel collisionless embedding table to efficiently handle sparse, dynamic features, significantly improving model quality and memory usage.  A key innovation is its online training architecture , enabling real-time model up...

This article reminisces on the history of the Postgres project, spearheaded by Michael Stonebraker at UC Berkeley from the mid-1980s to the mid-1990s. It details Stonebraker's design philosophy and the project's technical innovations, including support for abstract data types, active databases, and novel storage and recovery mechanisms. The article highlights Postgres's evolution into the open-sou...

This blog post details Yelp's in-place migration of their Yelp Reservations service database from PostgreSQL to MySQL. The migration, necessitated by maintenance and expertise limitations with PostgreSQL, involved significant code refactoring to address unsupported features and ensure data consistency. A gradual rollout strategy, employing multi-DB support and careful synchronisation, was implemen...

Meta's FBDetect system, detailed in this research paper, is a robust, in-production performance regression detection system. It identifies minuscule performance regressions (as small as 0.005%) across millions of servers and hundreds of services by monitoring hundreds of thousands of time series metrics. Key to FBDetect's success are advanced techniques for subroutine-level performance analysis, f...

This paper details the architecture and evolution of Amazon DynamoDB, a fully managed NoSQL database service.  Key features  highlighted include its scalability, predictable performance, high availability (achieved through multi-region replication and sophisticated failure handling), and strong durability (guaranteed by techniques like write-ahead logging and continuous data verification). The aut...

This blog post discusses the multifaceted definition of a senior software engineer.  Technical expertise  is crucial, encompassing a T-shaped skill profile and a deep understanding of software development principles. However,  soft skills , such as communication, leadership, and a growth mindset, are equally vital for moving projects and teams forward. The author suggests several strategies for pr...

Amazon Web Services (AWS) has launched Amazon S3 Tables, a new storage service optimised for analytical workloads. These tables, stored in a new type of S3 bucket, utilise the Apache Iceberg format for efficient querying with tools like Amazon Athena and Apache Spark. Offering significant performance improvements (up to 3x faster queries and 10x more transactions per second) over self-managed solu...

This transcript from an AWS re:Invent 2024 session details Amazon S3's architecture and engineering principles.  Two senior engineers  explain how S3's massive scale enables efficient data management, utilising techniques like  shuffle sharding  to distribute workloads across millions of drives. They discuss the  physics of data storage , showcasing how S3's design improves performance and reliabi...

This blog post by Marc Brooker, an AWS engineer, explains the write operations within Aurora DSQL, a scalable SQL database. It details how Aurora DSQL uses optimistic concurrency control (OCC) combined with multiversion concurrency control (MVCC) to achieve strong consistency and snapshot isolation. The system uses an "adjudicator" service to manage write conflicts and a "Journal" for durable, ord...

This research article details the development and benchmarking of a high-performance URL parser compliant with the WHATWG standard. The authors created a C++ implementation leveraging vectorisation techniques, resulting in a parser significantly faster than existing solutions like curl and rust-url. Their parser was integrated into Node.js, leading to substantial performance improvements in URL pr...

This article presents technology predictions for 2025 and beyond, focusing on several key themes.  Firstly , it highlights a growing mission-driven workforce prioritising positive societal impact over solely financial gain.  Secondly , it discusses the crucial role of technological innovation in addressing the global energy crisis, advocating for a blend of renewable and nuclear solutions alongsid...

This paper details the architecture and design of Amazon Aurora, a cloud-native relational database service.  Key design choices  focus on mitigating network bottlenecks inherent in high-throughput cloud systems by offloading redo processing to a separate, distributed storage service. This approach enhances  durability and availability  through a novel quorum model and segmented storage, significa...