13 Best Platform Engineering Tools to Accelerate Software Delivery

Building sophisticated software requires coordinating many complex and interconnected systems. From provisioning infrastructure, to CI/CD pipelines, to monitoring and logging, the moving parts are endless. Trying to wire this all together manually results in a tangled mess slowing down delivery.

This is where platform engineering comes in. Platform engineering aims to tame the chaos by providing pre-built abstractions and pipelines tailored for software delivery. By codifying patterns and best practices, platform engineering powers organizations to ship better software faster.

In this comprehensive guide, we‘ll dive deep on platform engineering, including:

• Core concepts and benefits
• Architectural principles
• 13 must-have open source tools
• Implementation advice

If you want to transform how your organization delivers software, this guide is for you. Let‘s get started!

What is Platform Engineering?

Let‘s quickly level set on what platform engineering entails.

Platform engineering is the practice of streamlining software delivery by creating reusable abstractions on top of the underlying infrastructure. The goal is to remove undifferentiated heavy lifting from developers, so they can focus on writing business logic.

This is accomplished by having platform engineers handle tasks like:

• Provisioning infrastructure – VMs, networks, databases etc.
• Building deployment pipelines – CI/CD, testing etc.
• Managing config – No more checkins of config files!
• Providing shared services – Logging, monitoring, auth, notifications
• Defining policies and standards – Security, compliance, architecture

By codifying solutions to common requirements, engineers are freed from reinventing the wheel for every project. They can build on top of robust, secure, and scalable foundations provided by the platform.

Platform engineering has emerged as an essential capability as modern software has increased exponentially in complexity. Trends driving this include:

• Hybrid/multi-cloud – Apps spanning data centers, public cloud, edge
• Microservices – Distributed systems with dozens+ discrete services
• Polyglot persistence – Multi-model data from SQL, NoSQL, caches
• Orchestration – Rise of schedulers like Kubernetes and Nomad
• Increased scale – Large and variable traffic volumes
• Need for speed – Pressure to release innovations faster

Trying to tame this complexity at the individual application level is enormously challenging, time consuming, and error prone. Platform engineering introduces holistic solutions that improve the productivity of every engineer.

Benefits of Platform Engineering

Adopting platform engineering can yield tremendous benefits:

• Improved productivity – Abstract infrastructure and tools so developers spend time on differentiating work

• Increased velocity – Deploy MVPs faster, reduce time-to-market for experiments

• Enhanced reliability – Leverage proven patterns to reduce bugs and system failures

• Improved security – Implement policies to avoid anti-patterns that lead to breaches

• Lower costs – Standardization drives efficiency, automation reduces human overhead

• Compliance – Meet regulations more easily with security and access controls baked in

• Innovation – Free developers to focus on creating amazing user experiences

Based on real world results from companies like Netflix, Airbnb, Shopify, and Spotfy who pioneered platform engineering, development productivity and software delivery performance improves by 2-3x on average compared to traditional model.

These productivity gains multiply as engineering teams scale. Platforms are force multipliers that enable small teams to accomplish work previously requiring large teams.

For any software-driven organization, investing in platform engineering is one of the highest leverage ways to accelerate innovation.

Platform Engineering Principles

Effective platform engineering adheres to several key principles:

• Modular – Components are decoupled allowing independent evolution and reuse.

• Abstracted – Hide complexity and lower-level details from consumers.

• Documented – Platforms codify tribal knowledge into written artifacts.

• Automated – Platforms apply automation to remove human toil wherever possible.

• Secure – Zero trust model with least privilege access and encryption by default.

• Monitored – Inspectability at all layers – apps, network, infrastructure.

• Shared – Provide horizontal services leveraged across vertical applications.

• Standardized – Enforce uniform designs, patterns, and processes.

• Self-Service – Users can provision and manage platform resources themselves.

• Supportable – Engineer for operability from the start with log aggregation, metrics etc.

Well-designed platforms encapsulate complexity while providing guardrails. They empower autonomous teams by providing the tools teams need minus decisions teams shouldn‘t have to make.

Next let‘s explore top open source tools to operationalize these principles.

13 Must-Have Open Source Tools

Here are 13 open source tools essential for building a modern software delivery platform:

1. Terraform – Provision Infrastructure as Code

Terraform is the industry standard for provisioning infrastructure as code. It provides a simple, declarative language to describe any type of infrastructure – networks, instances, databases etc. Terraform supports hundreds of providers enabling multi-cloud infrastructure automation.

For platform engineering, Terraform provides the foundation for higher order abstractions by enabling automation of provisioning. Common infrastructure patterns become codified as reusable modules.

// Create reusable VPC module
module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  name = "my-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

} 

With Terraform, platform engineers can programmatically spin up preconfigured environments on demand for each project.

2. Crossplane – Unified Control Plane

Crossplane provides a Kubernetes native control plane to provision and manage infrastructure and services across environments. With Crossplane you can define abstractions like RDS or LoadBalancer once, and reuse them across AWS, GCP, Azure etc.

The Crossplane control plane architecture includes:

• Providers: Plugins integrating cloud provider APIs – S3, Route53, CosmosDB etc.

• Resources: Reusable abstractions – VPC, DB, Cache etc.

• Resource Claims: Declarative config of infrastructure parameters needed by an app.

Crossplane handles fulfilling claims by provisioning resources through providers. Complex multi-cloud apps can be easily built combining reusable building blocks.

For platform engineering, Crossplane enables true write-once, use anywhere abstractions. You encapsulate provisioning logic in resources and decouple it from applications through claims.

3. Harbor – Container Registry

Harbor is an enterprise-grade container registry for building, storing, and shipping container images. Harbor adds controls around image security, access control, and policy compliance missing from alternatives like DockerHub.

With Harbor, platform operators can centrally control responsibilities like:

• Policy compliance via image scanning
• RBAC to grant granular access to repos
• Auditing of all registry activity
• Replication of images across geographies
• High availability configurations

For platform engineering, Harbor gives teams a trusted base image registry. Images can be pre-scanned and certified to conform to security policies before promotion to production.

4. Kafka – Event Streaming

Kafka is a high performance event streaming platform. It serves as a central nervous system piping event data between services, databases, apps and external systems. Kafka provides durable storage, low latency delivery, and replication/failover.

For platform engineering, Kafka enables:

• Decoupling – Services interact by publishing events instead of point-to-point calls.

• Scalability – Kafka partitions and replicates topics across brokers.

• Replayability – Ability to rewind and replay events for debugging/audit.

• Integration – Connectors available for all major apps and infrastructure.

Kafka provides a critical substrate for service communication and data processing that underpins complex microservices architectures.

5. Consul – Service Networking

Consul delivers a full lifecycle service networking solution covering:

• Discovery – Services register and can find/connect to other services

• Configuration – Key value store syncs dynamic config to services

• Segmentation – Secure service communication with mTLS

• Observability – Proxy and ingest envoy data like traces

Consul provides cross-cutting networking, security, and observability capabilities decoupled from application code. It works across runtimes like VMs, containers, and serverless.

For platform engineering, Consul handles undifferentiated plumbing like networking and config management. Services built on Consul can focus purely on business logic.

6. Vault – Secrets Management

Vault is a secrets management tool for secure access to tokens, passwords, certificates, encryption keys required by applications and infrastructure. Features include:

• Dynamic Secrets – Secrets generated at runtime with short TTL

• Encryption – Data encrypted in transit and at rest

• Access Control – Granular path-based access policies

• Audit Logs – All access events recorded

• Replication – Active/passive clusters, multi-region

For platform engineering, Vault provides a centralized secrets backend to control access to security sensitive credentials. It also enables greater rotation and reduces hardcoded secrets.

7. Backstage – Developer Portal

Backstage is an open source developer portal that Spotify created to unify access to all its internal platforms and tools. Backstage provides a single pane of glass for things like:

• Code repositories
• CI/CD pipelines
• Application services
• Infrastructure topology
• Tech docs
• Access controls

For platform engineering, Backstage provides the experience layer developers operate within. The self-service UI/API vastly improves engineers‘ visibility into the platform and their productivity.

8. Steampipe – Infrastructure Querying

Steampipe allows you to query your infrastructure as code like a database using SQL. It taps into Terraform state, Kubernetes API, cloud provider APIs and synthesizes the results.

-- Find overprovisioned pods in prod
SELECT 
  namespace, 
  name,
  requests_cpu,
  limits_cpu
FROM kubernetes_pod
WHERE environment = ‘prod‘
  AND limits_cpu > 1
  AND limits_cpu > requests_cpu * 2

For platforms, Steampipe enables admins to enforce governance rules and compliance against provisioned infrastructure efficiently. Common checks can be packaged as reusable query packages.

9. Prometheus – Metrics Monitoring

Prometheus is a ubiquitous open source monitoring and alerting toolkit. It scrapes and stores metrics time series data from hosts and services. Prometheus supports powerful query language to analyze metrics and rich visualization options like Grafana.

For platform engineering, Prometheus is invaluable for end-to-end observability. Platform services expose base metrics for troubleshooting. Apps can build on rich metrics foundation enabling faster incident response.

10. Jaeger – Distributed Tracing

Jaeger provides open source distributed tracing for observing request flows across microservices architectures. Instrumentation like OpenTelemetry captures timing data per service. Jaeger aggregates and visualizes request journeys.

For platform engineering, Jaeger enables deep monitoring of requests traversing platform infrastructure. Performance bottlenecks and failures can be quickly isolated.

11. Linkerd – Service Mesh

Linkerd is a lightweight service mesh proxy that adds resilience, security, and observability to service communications without code changes. Key benefits:

• Retries and timeouts prevent cascading failures
• mTLS encrypts service traffic
• Distributed tracing provides request insights

For platforms, Linkerd provides baseline capabilities needed for service communications like fault tolerance and encryption. It can be installed mesh-wide vs. per service.

12. Jenkins – Automation Server

Jenkins is the leading open source automation engine. It supports continuous integration and delivery via pipelines codifying stages ranging from build to deployment.

For platform engineering, Jenkins provides the assembly line to orchestrate tasks into a repeatable process from commit -> Docker build -> scan -> deploy. Pipelines encode platform requirements like testing, security checks, and release governance.

13. Spinnaker – Multi-Cloud CD

Spinnaker is an open source multi-cloud continuous delivery platform. It provides release pipeline abstractions spanning deployment targets like Kubernetes, AWS, and on-prem data centers. Spinnaker pipelines natively support best practices like manual judgement gates and canary deployment strategies.

For platform engineering, Spinnaker provides cloud-agnostic pipelines to promote applications from environments like dev through staging, pre-prod, to production across regions and cloud providers.

Implementing Platform Engineering

Now that we‘ve covered powerful tools, let‘s discuss some key best practices for implementing platform engineering:

• Start small – Focus on high leverage areas first vs. boilling the ocean. Prioritize tooling for developers early on.

• Infrastructure first – Build from the bottom up. Focus initial abstraction on provisioning.

• Get feedback – Involve developers early and often. Dogfood the platform and refine based on feedback.

• Automate provisioning – Make environments self-service to provision.

• Incremental adoption – Support optional use and migration paths from old tools.

• Observability – Monitor platform logs, metrics, traces from day 1.

• Security first – Build zero trust model into foundation. Harden platforms before expansion.

• Loose coupling – Favor decoupled composable services over monoliths.

• Train developers – Clearly explain the why behind platform decisions and how to leverage it.

Done right, platform engineering powers an org towards radically improved productivity, reliability, and innovation velocity. The time invested in platform foundations pays exponential dividends over time as more apps are built leveraging shared services.

Conclusion

In this guide, we took a deep dive into platform engineering – its concepts, benefits, architecture, and tooling. The overarching goal of platform engineering is empowering engineers to focus on differentiating business logic instead of infrastructure and operations.

By codifying solutions to undifferentiated problems all apps face, platform engineering creates force multipliers enabling small focused teams to deliver outsized results. Companies who have invested in platform engineering like Netflix and Airbnb have seen engineering productivity skyrocket.

We explored 13 must-have open source tools covering provisioning, deployment, networking, security, and monitoring. Of course there are many other great options beyond those highlighted here when assembling your platform toolkit.

The most important thing is applying platform engineering principles – abstraction, automation, loose coupling etc. – across your technology stack. Done well, platform engineering can 2-3X your development velocity, reliability, and satisfaction.

Are you looking to optimize your software delivery process? Get in touch to learn how we can help assess your architecture and build a tailored platform engineering solution leveraging proven tools and practices. Let us know if we can be a resource in your platform engineering journey!