DevOps is not just a collection of practices but a culture that focuses on collaboration. Therefore, its main goal is to promote improved communication between the software engineer and the operations team. Integration of development and processes is a powerful innovation in advancing software development. If you're new to software engineering or want to improve your current processes, it can be hard to know which DevOps tools you need to know. So, in this article, I'm taking a look at my top five DevOps toolsets that every software engineer should know.

1. CI/CD Pipeline

Continuous integration and deployment are among the most important practices for software engineers. You can make incremental yet versioned and tested changes to code, making the whole process simpler and more accessible for everyone involved. CI/CD provides continuous feedback from customers and the DevOps team to increase transparency on all issues within and outside the team. This entire process ensures rapid product release. Failures are now detected and prevented much earlier in the process, saving time and money.

CircleCI

(https://circleci.com) CircleCI is quickly gaining popularity in the software engineer community and is used by over 1M engineers. It supports containerization, build automation across multiple environments, secure pipelines, cloud or self-hosted or on premise servers, and tons of tools and integrations. As a software engineer you can start for free and really dig in to its features and awesome interface, but know that it can easily scale up to your enterprise needs if necessary. I’ve seen this tool used across multiple languages/platforms and it really is a must-see.

GitHub Actions

(https://docs.github.com/actions) GitHub, now under the umbrella of Microsoft, is a mainstay in the version control and now CI/CD community. GitHub Actions helps you automate, customize, and execute your SDLC right in your GitHub repository (and others too). You can elect to run workflows on Linux, Windows, and macOS VMs. Or, you can run on your own servers. Through straightforward organization of workflows, events, jobs, actions, and runners, you have complete control of highly customizable YAML-based workflows. I am betting you have used or seen GitHub Actions already. If not, now is the time to check it out.

Azure DevOps

(https://azure.microsoft.com/en-us/products/devops/) Microsoft's Azure DevOps platform is the cloud-based and latest incarnation of what was once TFS but is now greatly improved and a truly wonderful CI/CD tool. In addition to the usual CI/CD pipeline capabilities, it also integrates just about everything else you need in an end-to-end software development lifecycle solution:

• Agile planning, item tracking, visualization and reporting tools
• Supports containerization and cloud-agnostic usage
• Provides integrated package management and supports Maven, npm, Python, and NuGet package feeds from public or private sources
• Integrates Azure Test Plans for planned and exploratory testing solution

Azure DevOps has come a long way since TFS, and with the addition of GitHub (Actions) to Microsoft’s acquisitions, we know there will be even more great things to come ;)

2. Monitoring

Infrastructure monitoring has always been important to “infrastructure” folks but with the advent of DevOps and now even the concept of NoOps it’s even more important that software engineers be aware of and be using strong infrastructure monitoring tools. Here are my picks for top monitoring tools that software engineers should be aware of and using if not using already.

Grafana

(https://grafana.com) Grafana makes its manta “dashboard anything. Observe everything.” With Grafana, you can query, visualize, and alert on your data no matter where it’s stored. It takes the unique approach of unifying your data into easy-to-use dashboards rather than ingesting your data into one database. Grafana connects to just about any data source you can think of, and it helps everyone in every type of role visualize that data in ways they probably didn’t think of. The dashboards are very customizable and continuously evolving to display complex data via heat maps, histograms, geo maps, and much more. Data visualization is one of Grafana’s key strengths for many users. With open source, cloud, and enterprise versions, and free forever plan, it’s a tool you simply must check out.

Honeycomb

(https://www.honeycomb.io) Honeycomb helps you reveal the truth about how users experience your code in complex and unpredictable environments. Within seconds, find patterns and outliers across billions of rows of data to definitively solve problems. Data can be sent to Honeycomb in many different ways. You can then trigger alerts based on service level objectives when bad things are happening and jump in immediately to diagnose the issue. Custom queries can be created to slice and dice through billions of rows of data. Outliers can be automatically bubbled up for review. Honeycomb is truly a great tool and useful both by software engineers during development, and in the DevOps capacity in production. Well worth checking out.

AWS Cloudwatch

(https://aws.amazon.com/cloudwatch) The Amazon Web Services (AWS) platform is the most comprehensive cloud platform in the world, with more than 200 full-featured services from data centers worldwide. Cloudwatch can be used to monitor and observe AWS resources both within and outside a specific AWS account, in the cloud and on premises. Cloudwatch can be used to:

• Collect, analyze, and visualize data
• Create alarms and automated actions set to activate at predetermined thresholds
• Integrate with more than 70 AWS services
• Troubleshoot pre-production and production problems via dashboards

Cloudwatch is one of the best and primary choices for monitoring in the AWS cloud environment.

Azure Monitor

(https://azure.microsoft.com/en-us/products/monitor) Azure Monitor is Azure’s native monitoring tool for collecting, analyzing, and acting on telemetry data from cloud and hybrid environments. It provides the same core features as the other big-3 (AWS, Azure, GCP) monitoring tools:

• Create customized monitoring experiences for particular architecture situations
• Use visualization tools to observe ingested data in one place
• Respond in near real time with alerts and the ability to autoscale resources when needed

If you are doing any DevOps in Azure, you are using Azure Monitor or you should be using it.

Google Cloud Monitoring

(https://cloud.google.com/monitoring) Cloud Monitoring is Google’s version of the big-3 monitoring tools. Like the other offerings, it provides for:

• SLO (service-level objectives) monitoring
• Metrics collection from many data sources including Kubernetes and virtual machines
• Specific strong support for its own platform (Google Cloud)- in the case of GCP their monitoring tool can discover and monitor ALL Google Cloud resources and integrate directly into the Google Cloud dashboard

Same message here: if you are doing DevOps on GCP then are using Google Cloud Monitoring or you should be using it.

3. Logging

Log management has historically been more an “Ops” thing in DevOps. But as tools got better at being real-time, and as DevOps became more “Dev”, logging toolsets became more important and valuable for software engineers to understand and use in their software development process. Here’s my pick of top logging tools from the software engineer perspective.

New Relic

(https://newrelic.com) The New Relic offering allows you to monitor, debug, and improve your entire stack. It packs over 15 tools into one suite and supports monitoring of application, infrastructure, containers, logs, browsers, mobile, network, and more. With a large coverage and pricing that starts with free and scales to very large capabilities, it’s a popular tool with software engineers that’s well worth a look if you’re not familiar with it already.

Splunk

(https://www.splunk.com) Splunk touts itself as “The Unified Security and Observability Platform”. Splunk has a large community, over 2K community-built apps, lots of customization options, and a large ecosystem of partners and members and experts to help larger organizations with their logging needs. A lot of organizations use Splunk so it makes a lot of sense as a software engineer to be familiar with this big logging tool provider.

Logstash

(https://www.elastic.co/logstash) Logstash is a server-side log data processing pipeline that dynamically ingests, transforms, and ships log data regardless of format or complexity.

Logstash is a free and open-source tool whose source code is freely available on GitHub. Logstash provides a variety of inputs, filters, and outputs that can be used to process and visualize different types of logs. Logstash is a popular open-source logging option that you should be familiar with. If it sounds familiar, it’s because it’s part of the classic ELK stack- Elasticsearch (search based on Lucene engine), Logstash (collect and transform log data from various sources), and Kibani (the visualization layer atop this stack).

4. Infrastructure as Code

Infrastructure as Code is the process whereby data center infrastructure can be managed and provisioned via machine readable definition files rather than physical hardware configuration or interactive configuration tools. IaC was historically more on the “Ops” side of DevOps. However more recently IaC has taken a stronger role in the software development lifecycle. Improvements have occurred around the concepts of pushing necessary hardware environment definitions along with code definitions across pre-production and production environments. So as a software engineer, you are likely already using tools in the list below- or your likely should be.

Terraform

(https://www.terraform.io) Terraform supports automating infrastructure on any cloud. It is large in the IaC community. Terraform uses its own flavor of a definition language as compared to other tools that use JSON or YAML. This is one of the few criticisms heard about Terraform. Plug-ins exist that wrap existing definition languages like Azure Bicep so they can be used. But regardless of the specific definition file language, Terraform is a heavy-weight when it comes to IaC, multi-cloud deployment, managing containerization and orchestration, managing virtual machines, and security.

Azure Bicep

(https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep) Azure Bicep is a unique domain-specific language (DSL) for declaratively deploying Azure resources. With Bicep, you define the infrastructure you want to deploy to Azure and then use that definition file to consistently deploy the same infrastructure. Bicep attempts to be an easier definition language syntax than say JSON. If you are doing any IaC in Azure, then you need to know Bicep and ARM templates.

AWS CloudFormation

(https://aws.amazon.com/cloudformation) AWS CloudFormation is an AWS service that utilizes template files to automate the configuration of AWS resources. AWS CloudFormation supports almost any service running on AWS, so that it can be described as infrastructure automation, infrastructure-as-code (IaC) tools, or cloud automation solutions. You can use CloudFormation to automate workload configuration on some of the most popular AWS services, such as EC2 compute services, S3 storage services, and IAM services for configuring access control. If you are doing any IaC on AWS, then you need to know Cloudformation.

Pulumi

(https://www.pulumi.com) Pulumi is an IaC platform for full-stack developers and cloud engineers interested in a more programmable solution for every cloud, every language, every architecture, and every builder. Unlike other IaC options, Pulumi caters more to the software engineer. It accomplishes this via a programmable cloud interface and reusable packages across many different languages including TypeScript, Python, Go, C#, Java, and YAML. Pulumi takes a very software-driven approach and really brings IaC into the code package process.

5. Feature switches

The first 4 DevOps toolsets a software engineer should know about were more “Ops” leaning when it came to DevOps. This #5 toolset - feature switches - goes the other direction. While feature switches used to be the domain of software engineers, the capabilities have now grown such that DevOps engineers and even product owners can now control feature switch configuration. Here’s my list of a few notable players in this toolset.

Split.io

(https://split.io) Split offers the ability to categorize any customer into attribute-based segments (called "splits"), integrate data from metrics services, and use logs to manage and measure flag behavior. Based on this data, Split allows users to create rules with basic alerts and kill-switch behavior. Feature flags provide engineering teams with powerful tools to turn code off and on in production. Split has really opened up the market and moved feature flag functionality forward for the better.

LaunchDarkly

(https://launchdarkly.com) LaunchDarkly is a developer SaaS platform for managing feature flags. By separating the deployment of functionality from the placement of code, LaunchDarkly enables software engineers to test their code in production, release features to user groups over time, and manage flags throughout their lifecycle. It allows developers to release better software with less risk. LaunchDarkly provides an SDK for software engineers in almost any language. It also supports workflows that allow users to schedule and approve feature flags. Feature flags can also be scheduled for delivery. And notifications can be done via integrations with Jira, Slack, and other tools.

Final Thoughts

Software engineering and DevOps continue to overlap more and more. As a software engineer, it’s important to have a firm grasp of the DevOps toolsets you may need to understand or even configure and implement. Doing so could be a real booster to your software engineer career.