Software Engineering Daily

This episode is different from the traditional interview format of Software Engineering Daily, and focuses on the history of Hadoop. Thanks to Marco Bonaci for allowing us to republish this in audio format. You can find the original post here: History of Hadoop

Operating a data center has been turned into a service. For most new companies, managing real-world servers is no longer an issue. But one exception is if your startup is a company offering the data center as a service. In this episode, Jeff and Sam talk about the engineering that goes on at DigitalOcean, and how the company differentiates itself from its competitors. Sam Kottler is an engineering...

Monoliths versus microservices – the architectural debate continues across the internet. SoundCloud is a popular music company that experienced the rapid development benefits of a monolith, as well as the long-term technical debt. That technical debt has since been relieved by a move towards microservices. In this episode of Software Engineering Daily, Lukasz joins Jeff to talk about the realities...

Stream processing engines are often the component of data engineering stacks with the most variety, with the big data ecosystem offering several compelling options. Assessing the differences between projects like Flink, Storm, and Spark Streaming is difficult without an agreed upon set of metrics to compare them on. Fortunately, the Yahoo engineering team created a set of benchmarks to do exactly...

Software engineering combines art, science, and philosophy. On the Developer Tea podcast, Jonathan Cutrell explores how engineers can improve their technical ability as well as their interpersonal skills and mental clarity. On this episode of Software Engineering Daily, Jeff and Jonathan about focus, career development, and software podcasting. Jonathan Cutrell is the founder and host of Developer...

Matplotlib is a python plotting library inspired by MATLAB. It allows developers to easily visualize their data using simple functions. In this episode, Jeff and Ben discuss why visualizing data is integral for the human experience, and how they it can make drawing conclusions from data more meaningful. Ben relates this to his experience studying atmospheric data in the wild, and how his work as a...

Keras is a minimalist, highly modular neural networks library, written in Python and capable of running on top of either TensorFlow or Theano. It was developed with a focus on enabling fast experimentation. In this episode, François discusses the state of deep learning, and explains why the field is experiencing a cambrian explosion that eventually may taper off. He explains the need for Keras and...

Software is changing every domain including Marketing, Healthcare, and even Agriculture. Government is one specific area where the potential for improvement is obvious. Governments are often defined by abstract rules, transactions, permissions and hierarchies, grounded on some fundamental principles – this is striking similar to the same tenets that software and computer science are based on. In t...

In this episode, Jeff and Derrick sit down to discuss the state of tech journalism, and how it has kept up with the rapid pace of technology innovation. Additionally, they discuss the evolution of the “big data” stack and where it is headed. The conversation focuses in on Apache Mesos and Mesosphere, the company building the Datacenter Operating System (DCOS). Derrick Harris is a long time tech jo...

Incarceration in the United States is often a punitive exercise rather than a rehabilitative one. As a consequence, recidivism is the norm – according to a 2012 report by the California Department of Corrections and Rehabilitation, just over 65% of those released from California’s prison system return within three years. San Quentin State Prison in California is challenging this issue by trying to...

Governments are often laggards in the adoption of new technology. Every aspect of life, from transportation to healthcare, are being affected by the convergence of several trends including connected mobile devices, cloud storage and distributed systems. Yet federal services are in many cases still incapable of delivering the best and most affordable services to citizens. The most visible instance...

Riak is a distributed NoSQL key value data store. It was created by Basho, taking inspiration from the Amazon Dynamo paper. In this episode, Jon and Jeff discuss the benefits and different implementations of non-relational distributed data stores, and why Riak is a compelling option. Jon Meredith is the chief architect at Basho.

Apache Spark has skyrocketed in popularity lately, arguably surpassing Hadoop as the hottest big data technology as of late. Even IBM has thrown its weight behind the framework, calling it the “most important new open source project in a decade”. In this episode, Holden joins Software Engineering Daily to discuss why Spark is growing in popularity and how developers can begin learning the framewor...

Apache Drill is a schema­-free SQL query engine for Hadoop, NoSQL, and cloud storage. Drill is unique in that it uses a JSON data model instead of a relational model like many traditional engines. Additionally, it supports on-the-fly schema discovery, which allows execution to begin without knowing the structure of the data. As a result, Drill can work with data that is constantly updating or chan...

Machine learning is something that many business are starting to tack onto their existing processes. Yet, to add machine learning capabilities after the fact is often a fool’s errand. Joshua argues that machine learning cannot be an afterthought, but rather must be custom developed to suit the specific problem or question that each company is trying to answer. His company, Wise.io, tackles this ch...

Data science has emerged as an integral discipline alongside software engineering. In this episode, Jeff and Srini discuss the rise of the field, what it means to be a data scientist, and how the field will evolve as more businesses adopt data driven decision making. Srini also explains his thoughts on the essential qualities of a data scientist, and how educating data scientists is a growing chal...

WebTorrent is a streaming torrent client for the browser that can be used without any additional plugins, extensions, or installations. It is written entirely in JavaScript, and uses WebRTC to handle peer-to-peer communication. There are some limitations to making BitTorrent work over the web using WebRTC, including the inability for a browser-based WebTorrent client or “web peer” to communicate d...

Django is an open source web development framework that was initially released in 2005, around the same time as Ruby on Rails. In many ways, the two frameworks are similar – they expedited web development by modularizing common components of web apps and enabling developers to quickly prototype and launch their products. They both encouraged the use of the MVC architectural pattern, with each resp...

Ruby on Rails is a web app framework released 10 years ago that influenced the way websites were being built. Rails skyrocketed to popularity in the late 2000’s and empowered many small companies to quickly build and maintain robust web apps. While it is still a mainstay in web development, it has been overshadowed as of late by JavaScript, and Node.js on the backend. David joins Software Engineer...

Marty Haught is a director of Ruby Central, a nonprofit dedicated to the support and advocacy of the worldwide ruby community. Ruby Central also hosts the RubyConf and RailsConf software conferences.

Michael Hartl is the author of The Ruby on Rails Tutorial, a widely acclaimed guide to learning how to build Ruby on Rails webapps.

Soylent is a meal replacement beverage, available in both liquid and powdered forms. Soylent is designed with the goal of meeting all nutritional requirements for an average adult. John Coogan is the CTO of Soylent.

Druid is a column-oriented distributed database that excels as a data warehousing solution for fast queries on large data sets. Fangjin Yang is a core contributor to Druid, and is currently co-founder and CEO of Imply, which helps build interactive analytics powered by Druid.

Brian Kernighan is a professor of computer science at Princeton University and the author of several books, including “The Go Programming Language” and “The C Programming Language”, a book more commonly referred to as K&R. Professor Kernighan also worked at Bell Labs alongside Unix creators Ken Thompson and Dennis Ritchie and contributed to the development of Unix.

Kitematic is an open source project built to simplify and streamline using Docker on a Mac or Windows machine. It allows users to get up and running on Docker with a single click, and provides a user interface that makes running and managing Docker files easy. Kitematic was acquired by Docker in March 2015. Sean Li is the co-founder and Chief Product Officer of Kitematic. He is now working on prod...