Software Engineering Daily

A data warehouse is a data management system that often contains large amounts of historical data and is used for business intelligence activities like analytics. It centralizes customer data from multiple sources to be an organization’s single source of truth. Getting the data from your data warehouse into the different applications used by your organization can be difficult.  The company Census...

IT infrastructure are the components required to operate IT environments, like networks, virtual machines or containers, an operating system, hardware, data storage, etc…. As companies build out different deployment environments with infrastructure configurations, they must maintain the different environments, replicate them, and update them. The management of infrastructure, often automated to so...

Decentralized applications, termed “dApps,” are applications that feel like normal apps but are actually deployed (mostly) on the Ethereum blockchain. This means dApps can’t be taken down, can’t be censored or blocked, typically use Ethereum accounts as identity, and would only experience downtime if Ethereum itself went down. There are a lot of things you can do with blockchain applications, part...

A ‘token’ can represent almost anything in Ethereum, according to Ethereum.org: Lottery tickets, points in an online platform, fiat currency, and much more. These tokens must follow a standard called ECR-20 to have the same type and value of any other token, and behave just like the ETH.   The platform Opyn lets users buy and trade decentralized finance (DeFi) options on ETH and ECR20s. “Options”...

In decentralized finance (DeFi) a liquidity pool is a collection of cryptocurrency funds created from the deposits of many users and usually multiple different currencies. There are 2 main types of pools: custodial and non-custodial. Custodial pools are controlled by a third party manager which contains information like the private keys and the funds. They are most commonly web-based. Non-custodia...

A smart contract contains the “terms” of a blockchain transaction between a buyer and a seller as well as the capabilities to execute those terms. In order for smart contracts to include outside data from the world, such as stock market data, weather, sports data, etc…, the contract needs a third party service called an oracle. The industry standard blockchain oracle is Chainlink. Chainlink is a d...

Using artificial intelligence and machine learning in a product or database is traditionally difficult because it involves a lot of manual setup, specialized training, and a clear understanding of the various ML models and algorithms. You need to develop the right ML model for your data, train the model, evaluate it, optimize it, analyze it for outliers and anomalies, assemble confidence ranges of...

The typical procedure many companies follow to reach production-level code is design the program, code and test it in different environments, and put it in a pipeline to deploy to production. Developers can make it pretty far into building their core features before inevitably breaking to include enterprise features and security standards like Single Sign On, Okta security and authorization for di...

A major change in the software industry is the expectation of automation. The infrastructure for deploying code, hosting it, and monitoring it is now being viewed as a fully automatable substrate. Equinix Metal has taken the bare metal servers that you would see in data centers and fitted them with supreme automation and repeatability. This movement to modern metal-as-a-service has brought with it...

Cryptocurrencies like Bitcoin and Dogecoin are electronic currencies with a complete transaction history stored on a blockchain. A cryptocurrency blockchain is a linear record of all the transactions between users for a given currency. This record is public and distributed across thousands of computers, which makes falsifying a transaction nearly impossible because the hacker would need to alter o...

Digital communities have exponentially grown in importance ever since most of the world went remote. Basically every popular online forum, message board, chat app, and other online social aggregators were created before this new normal. Many of these platforms lack sufficient organization or are just outdated for a fully remote environment. If society continues to work fully or hybrid remote, then...

Creation Labs is helping bring Europe 1 step closer to fully autonomous long haul trucking. They have developed an AI Driver Assistance System (AIDAS) that retrofits to any commercial vehicle, starting with VW Crafters and MAN TGE trucks. Their system uses camera hardware mounted to the vehicle to capture video data that is processed with computer vision to understand the context on the road. This...

Product teams sometimes double as data teams. They struggle through import errors, scrub long and complicated data sheets for consistency, and map spreadsheet fields on step 3 in a long instruction document. Data structuring and synchronization is a very real problem that product teams regularly overcome.  Flatfile uses AI-assisted data onboarding to eliminate repetitive work and make B2B data tra...

Right now, more than 10 million people use notebooks like Jupyter in their workflow. Notebooks are open-source tools for creating and sharing documents with live code, equations, visualizations and explanatory text. Notebooks like Jupyter have exploded in popularity the past 5 years to become the standard tool for data science teams. They became especially important in 2020 because they facilitate...

Many startups today begin their life as an open-source project. Open source projects allow early adopters of a technology to experiment, to contribute code and feedback, and to shape the evolution of the project in its early stages. When a “community maintainer” company emerges to provide service offerings based on that project, its early customer base often consists of contributors to the open so...

ELT, or “Extract, Load, and Transform,” is the process that modern data pipelines use to replicate data from a source and load it into a target system such as a cloud data warehouse. ELT is a more flexible evolution of the traditional “Extract, Load, Transform” workflow used in pre-cloud systems. The power of ELT relies on flexible integrations between data sources and their targets, called connec...

The complexity of building web applications seems to have grown exponentially in the last several years. This added complexity may bring power, but it can also make applications brittle, costly, and difficult to maintain. Suborbital is an open-source project with a goal of making web application development simple. Its flagship project is Atmo, a platform that integrates three underlying projects...

Observability is a key feature of a well-architected application. Because building an observability system for a cloud application can be challenging, especially at scale, many organizations elect to use third-party observability platforms rather than build internal tools. But these third-party provider contracts often charge by volume of data collected, which can be unpredictable and difficult to...

As the volume and scope of data collected by an organization grow, tasks such as data discovery and data management grow in complexity. Simply put, the more data there is, the harder it is for users such as data analysts to find what they’re looking for. A metadata hub helps manage Big Data by providing metadata search and discovery tools, and a centralized hub which presents a holistic view of th...

The shift to microservices architectures and distributed systems has been a challenge for systems using conventional security practices, such filtering IP addresses using network policies. In addition, the increasing intersection of development and operations exemplified by the DevOps methodology has expanded the scope responsibilities in implementing secure systems.  Part of CNCF, SPIFFE is a set...

Software-Defined Networking describes a category of technologies that separate the networking control plane from the forwarding plane. This enables more automated provisioning and policy-based management of network resources. Implementing software-defined networking is often the task of Site Reliability Engineers, or SREs. Site reliability engineers work at the intersection of development and oper...

Google Cloud, AWS, and Azure are the dominant cloud providers on the market today. But the market is still highly competitive, and there is significant overlap in the services offered by all three large providers. Since all three offer a broad range of services, developers looking to choose a platform for their application must focus on providers’ domains of relative excellence and how those align...

Vectors are the foundational mathematical building blocks of Machine Learning. Machine Learning models must transform input data into vectors to perform their operations, creating what is known as a vector embedding. Since data is not stored in vector form, an ML application must perform significant work to transform data in different formats into a form that ML models can understand. This can be...

Email has become such a routine feature of knowledge work that we often take it- and the email clients we use for it- for granted. While advancements such as intelligent spam filtering have improved the experience, many email clients retain the same basic structure and offer a largely similar experience. Superhuman is building a modern email client meant to reimagine email from the ground up. Supe...

WordPress is a free and open-source content management system, or CMS, written in PHP. Since its release in 2003, WordPress has become ubiquitous on the web. It is estimated that roughly 60 million websites use WordPress as a CMS. However, despite its popularity, Wordpress has limitations in its design. Wordpress sites are dynamic, and the front and back end are tightly coupled. A dynamic, full-st...