DevOps Talks Singapore wrap up

After two very enjoyable editions of DevOps Talks Conference in Sydney and Auckland with Bikkies Daniel, Harry, Samatha and Samuel we just wrapped up this year’s DOTC in Singapore. Personally, I don’t need too much encouragement to visit Singapore, there’s the amazing food, the incredible climate and it’s the technology hub for the APAC region. I haven’t been to a tech event in Singapore since Red Dot Ruby in 2017 and what struck me the most about the community in Singapore after so long away, was their curiosity, commitment to innovation, and the large number of companies with historically low appetite for change (or perhaps it’s their lack of ability to change due to scale) undertaking very impressive DevOps and SRE transformations.

I had the pleasure of co-hosting the event with Anthony Rees, and presenting on the 2nd day, and there’s nothing like being in the midst of the action to appreciate the efforts of the organisers, the volunteers, and the presenters. The community in Singapore was friendly, eager to learn, and all keen to hear the stories and experiences in all 8 presentations; the catalyst for many great conversations.

There were some standout themes for me across the two days.

Innovation and transformation

We heard from UBS, and DBS; two large global banks that have undertaken substantial SRE transformations with flair and a tonne of great, practical solutions to maximise impact and reduce toil.

Sandeep Hooda spoke about the SRE transformation at DBS Bank, such a surprising and delightful presentation. The team's decisions were inspired by Axolotls, not a tool as I thought when I first saw the talk title, but the fascinating creatures that are:

• Adaptable: possessing both gills and lungs allowing it to breathe above land under water – the SRE team and their systems needed to be able to respond to change, and adapt and pivot with ease - not always easy in a large bank.
• Self-healing: they have the ability to regrow limbs, and regenerate brain tissue – systems needed to be self-healing.
• Flamboyant dancers: Axolotls attract a mate with showy, exuberant dance moves – the SRE team’s new tooling and workflows needed to be impressive enough to attract and retain new employees.

Ghouse M, Lead SRE at DBS then took us on a tour of their infrastructure management interface and workflows (with screenshots only - because security). The axolotl's influence was clear, in their adaptable, self-healing and user friendly systems.

Leslie Wang took us on the journey of UBS’s huge SRE transformation, a massive consolidation of tools and systems across their team. In an effort to reduce complexity and toil they whittled down the breadth of tools to a handful, and via this streamlined approach managed to reclaim focus and the ability to work on the things that mattered.

Reducing the number of tools across the team that do the same thing means not only reducing the upkeep of multiple systems, and management of multiple vendors, but also increases the teams ability to jump from team to team with the ability to hit the ground running without friction.

The team now have a form that engineers experiencing build related issues complete before a resolution is even attempted, it collects the essential information such as pipeline, build number, failure time and message, and is linted to rule out known issues and inactive or redundant instances and resources. Reducing repetitive, back-and-forth communication and having the necessary information upfront has seen an increase in focus for the team, and a significant reduction in Mean Time to Resolution (MTTR).

The opening keynote presenter was John Willis, DevOps veteran and prolific author, co-author of The DevOps Handbook and his latest book Investments Unlimited. His talk "Investments Unlimited - A Novel" was about the catalyst for his new novel, based on The Pheonix Project focused on an investment bank dealing with DevOps, DevSecOps, and IT Risk.

“If we can get our Developers thinking about brand problems relating to Latency then we’ve won!”

John Willis

He enlightened and inspired the audience, with a tonne of truth bombs, backed by statistics, and encouraged us to consider new ways of thinking about DevOps, and mitigating risk by shifting security left, including actionable ways to both automate and surface auditing and compliance requirements via a human readable and machine readable DSL or YAML.

I loved PagerDuty’s Heath Newburn’s "Automation-First, People-Centric AIOps" talk, he unsurprisingly introduced us to the concept of AIOps, with a challenge to consider leaning on AI to manage resolutions and reduce the noise from non-critical incidents by setting them to auto-resolve. Because noisy alerts, and lack of focus can result in burnout.

Gartner's definition of AIOps (Artificial Intelligence for IT Operations) states:

AIOps combines big data and machine learning to automate IT operations processes, including event correlation, anomaly detection and causality determination.

To address burnout, and to have the work to chip away at the noise prioritized Heath says:

it's essential to talk about SRE problems so that your CEO cares, to speak their language.

For example:

• we decreased employee attrition rates by 30% 📉👩‍💻
• we increased uptime by X% and conversion by $X as a result 📈💰

Talk in their terms, make it about money, "don't be the moles and trolls in the basement!".

Finally, he stressed that when adopting AIOps, it's important to understand while it probably won't be perfect on the first interation, or perhaps even the third, we should work with machines as we would junior engineers.

"Think of the machines as you would a junior engineer, you wouldn’t fire them when then make a mistake, you’d use that as an opportunity to teach them."

Heath Newburn, Distinguished Field Engineer at PagerDuty

Gabriel Mesquida-Masana’s talk "Twinning air traffic control – from the monolith to Kubernetes" was a highlight for everyone, if you are as excited by Aviation as a DevOps transformation in historically slow moving, and rigid environment (because safety!), this talk is 100% on point.

The team at Thales are working on a research project with the Singapore Government to enable experimentation with Air Traffic Control data in an attempt to impact a whole host of areas. They’ve built a “Digital Twin” that replicates an on-prem, super secure, 10 year old monolith, that hardly changes. This digitial twin is hosted in the cloud and uses Open Source tooling & Kubernetes to allow them to swiftly experiment, using historical flight plan data – to do things like predict the most efficient landing trajectories to significantly reduce fuel (& the carbon footprint) of landing. I can’t wait to see where the future of aviation goes with experimentation and research like this.

DevSecOps

A lot of the talks focused on the “Sec” in DevSecOps, and the ever increasing need to shift security left – to ensure that we’re working on the security of our application code and CI/CD processes and infrastructure earlier in our development lifecycles. And to encourage us to automate our compliance and security checks by generating Software Bills of Materials (SBOMs), so that we have a register of the software in our software and so that we can work to fix CVEs swiftly and without the effort of scrambling to find them across 400 micro-services, owned by…well…who knows?

“With the rise of containers you need visibility into what’s in them, vulnerability scans, logging and alerts! Save yourself from ending up on the front page of tomorrow’s newspaper! Shift security left everyone needs runtime visibility of these things!”

Anthony Rees, Sr. Solutions Engineer / Cloud Alliances Manager, Lacework

People first

And finally, and true to what’s at the heart of DevOps philosophy (yes folks, it’s still very much alive and kicking) there was an emphasis on the importance of fostering healthy human ecosystems and communication channels – and the understanding that these are still far more important to success than tooling or automation alone! Automation was a tool to save people from burn out, and to help teams maintain focus on more important, and enjoyable work - as it should!

And that's a wrap

That’s it for DevOps Talks in 2022 – we’ve made a lot of new friends and even caught up with some old friends in Sydney, Auckland and Singapore.

If you’d like more inspiring content from innovative teams spanning; security, containers, kubernetes, GitOps, dynamic build pipelines and Infrastructure as Code, join us online for our in December for Unblock – our very own 2 day virtual CI/CD conference.

Following Unblock Tessa Bradbury and I will be speaking at SREcon Asia Pacific in Sydney, after which we'll all take a well earned break, like I hope you do too – see you somewhere in the world in 2023.