Software Engineering Daily

Logs are the source of truth. If a company is sufficiently instrumented, the logging data that streams off of the internal infrastructure can be refined to tell a comprehensive story for what is changing across that infrastructure in real time. This includes logins, permissions changes, other events that could signal a potential security compromise. Datadog is a company that was built around log m...

Dev.to has become one of the most popular places for developers to write about engineering, programming languages, and everyday life. For those who have not seen it, DEV is like a cross between Twitter and Medium, but targeted at developers. The content on DEV ranges from serious to humorous to technically useful. DEV contains a set of features which appeal to a developer community, such as the ab...

Drug trials can lead to new therapeutics and preventative medications being discovered and placed on the market. Unfortunately, these drug trials typically require animal testing. This means animals are killed or harmed as a result of needing to verify that a drug will not kill humans. Animal testing is unavoidable, but the extent to which testing needs to occur can be reduced by inserting machine...

Across a company, there is a wide range of resources that employees need access to. Documents, S3 buckets, git repositories, and many others. As access to resources changes across the organization, a history of the changes to permissions can be useful for compliance and monitoring. Indent is a system for simplifying access management across infrastructure. Indent allows users within an organizatio...

Acquisitions are part of the technology industry. A successful corporation will often have an “exit”, either going public or becoming acquired. And with each of these corporations, there is a set of stories that narrate the company from beginning to end.  Acquired is a podcast that tells the stories of companies such as YouTube, Instagram, and PayPal. During each episode, the life of a company is...

Ray is a general purpose distributed computing framework. At a low level, Ray provides fault-tolerant primitives that support applications running across multiple processors. At a higher level, Ray supports scalable reinforcement learning, including the common problem of hyperparameter tuning. In a previous episode, we explored the primitives of Ray as well as Anyscale, the business built around R...

Pandas is a Python data analysis library, and an essential tool in data science. Pandas allows users to load large quantities of data into a data structure called a dataframe, over which the user can call mathematical operations. When the data fits entirely into memory this works well, but sometimes there is too much data for a single box. The Modin project scales Pandas workflows to multiple mach...

A large codebase cannot be searched with naive indexing algorithms. In order to search through a codebase the size of Uber’s it is necessary to build a much more sophisticated indexing system than simple pure text search. SourceGraph is a system for universal code search. It allows developers to more easily onboard to a new codebase, make large refactors, and perform other tasks. SourceGraph can i...

Users do not use web applications in the way that you might expect. And it is not easy to get the data that is necessary to get a full picture. But a newer API within browsers does make this more possible by capturing DOM mutations.  The change capture of these DOM mutations can be stored for replay in the future. After being stored, this change capture can be retrieved and replayed. That allows f...

Managing microservices becomes a challenge as the number of services within the organization grows. With that many services comes more interdependencies–downstream and upstream services that may be impacted by an update to your service.  One solution to this problem: a dashboard and newsfeed system that lets you see into the health and changes across your services. With this kind of system, you ca...

ADP has been around for more than 70 years, fulfilling payroll and other human resources services. Payroll processing is a complex business, involving the movement of money in accordance with regulatory and legal strictures.  From an engineering point of view, ADP has decades of software behind it, and a bright future of a platform company used by thousands of companies. Balancing the maintenance...

Software companies can be funded in a variety of ways: venture capital, self-funding, and debt, among others. In order to receive financing, a company is evaluated on its ability to generate future cash flows. After all, a valuation is a number that summarizes the present value of future cash flows. Determining that valuation number is a complicated, subjective process. If the valuation can be det...

GitHub has been a social network for developers for many years. Most social networks are centered around mobile applications, but GitHub sits squarely in a developer’s browser-based desktop workflow. As a result, the design of a mobile app for GitHub is less straightforward. GitHub did acquire a popular mobile client called GitHawk, which was developed by Ryan Nystrom. Since joining GitHub, Ryan h...

A service mesh provides routing, load balancing, policy management, and other features to a set of services that need to communicate with each other. The mesh can simplify operations across these different services by providing an interface to configure them.  There are lots of different vendors who offer service mesh technology: AWS has AppMesh, Google has Istio (which is open source), Buoyant ha...

Netflix runs all of its infrastructure on Amazon Web Services. This includes business logic, data infrastructure, and machine learning. By tightly coupling itself to AWS, Netflix has been able to move faster and have strong defaults about engineering decisions. And today, AWS has such an expanse of services that it can be used as a platform to build custom tools. Metaflow is an open source machine...

WordPress has been a dominant force in the world of online publishing for many years because of how battle-tested it is. WordPress is the definitive leader in CMS technology. But there have always been alternatives.  Drupal, Ghost, and other open source CMSes. More recently, there has been an emergence of the headless CMS, such as Contentful, which decouples the CMS backend from the frontend prese...

M3 is a scalable metrics database originally built to host Uber’s rapidly growing data storage from Prometheus. When Rob Skillington was at Uber, he helped design, implement, and deploy M3. Since leaving Uber, he has co-founded a company around a hosted version of M3 called Chronosphere. If you have access to a scalable metrics database, you might as well start accumulating as much data as possibl...

Developing machine learning models is not easy. From the perspective of the machine learning researcher, there is the iterative process of tuning hyperparameters and selecting relevant features. From the perspective of the operations engineer, there is a handoff from development to production, and the management of GPU clusters to parallelize model training. In the last five years, machine learnin...

AWS has over 150 different services. Databases, log management, edge computing, and lots of others. Instead of being overwhelmed by all of these products, an engineering team can simplify their workflow by focusing on a small subset of AWS services–the defaults. Daniel Vassalo is the author of The Good Parts of AWS. An excerpt from the book: “The cost of acquiring new information is high and the c...

The modern release workflow involves multiple stakeholders: engineers, management, designers, and product managers. It is a collaborative process that is often held together with brittle workflows. A developer deploys a new build to an ad hoc staging environment and pastes a link to that environment in Slack. Other stakeholders click on that link, then send messages to each other in Slack, or make...

Deepgram is an end-to-end deep learning platform for speech recognition. Unlike the general purpose APIs from Google or Amazon, Deepgram models are custom-trained for each customer. Whether the customer is a call center, a podcasting company, or a sales department, Deepgram can work with them to build something specific to their use case. Sound data is incredibly rich. Consider all the features in...

DynamoDB is a managed NoSQL database service from AWS. It is widely used as a transactional database to fulfill key-value and wide-column data models. In a previous show with Rick Houlihan, we explored how to build a data model and optimize the query patterns for a NoSQL database.  Today’s show is about DynamoDB specifically: partitioning, indexing, query semantics, normalization, table design, an...

As a user browses a webpage, that browser session generates events that need to be recorded, validated, enriched, and stored. This data is sometimes called customer data infrastructure, or CDI. This data requires a full stack of different tools: a system on the frontend to collect the data, middleware to transport the data, and backend systems for storing and loading that data into data warehouses...

A software company manages and interacts with hundreds of APIs. These APIs require testing, performance analysis, authorization management, and release management. In a word, APIs require collaboration. Postman is a system for API collaboration. It allows users to test APIs with collections of requests, monitor the API responses, and visualize the query results. Users of Postman can collaborate wi...

At a customer service center, thousands of hours of audio are generated. This audio provides a wealth of information to transcribe and analyze. With the additional data of the most successful customer service representatives, machine learning models can be trained to identify which speech patterns are associated with a successful worker. By identifying these speaking patterns, a customer service c...