Distinguished Engineer of Comcast Cable
As a global leader in media and technology, Comcast is the parent organisation of three primary businesses: Comcast Cable, NBCUniversal, and Sky. Comcast has more than 55 million subscribers, with Sky renowned as one of Europe’s leading entertainment companies operating in seven territories and Comcast Cable recognised as one of the biggest cable TV, high-speed internet, and phone providers in the United States. Sitting down in the new Comcast Technology Centre at its headquarters in Philadelphia, Pennsylvania, Larry Maccherone, Distinguished Engineer of Comcast Cable, shared how the company is uniquely positioned for success in their agile approach to achieving a DevSecOps cultural transformation.
Maccherone’s professional background heavily revolves around data analytics and Lean-Agile, and he started his first business while still an undergraduate at university. “I’ve been a serial entrepreneur throughout my entire career. My first business had 80 employees and made US$20mn annually in sales,” explains Maccherone. “We were writing software that controlled a large portion of the world’s power generation, and it meant that if hackers exploited a vulnerability in the software, then it potentially brought down the world’s power grid. We got really skilled at writing software that didn’t have exploitable vulnerabilities.”
Upon joining Comcast in June 2016, Maccherone became responsible for overseeing the company’s DevSecOps transformation. “I have a love/hate relationship with the term DevSecOps. I believe that if you’re doing DevOps right, then the security part is automatically included,” he explains. “You don't call it DevTestOps or DevPlanningOps, it’s just DevOps. However, what I do like about DevSecOps is the emphasis on security. My definition of DevOps and DevSecOps is essentially the same. I define both as empowered engineering teams taking ownership of how their products perform in production, including security. When you get development teams owning the problem, you get a fundamental difference in decision making.”
Since its creation over a decade ago, DevOps has become a vital component of how companies operate. Building upon the foundations of the agile movement, DevOps leverages automation, for quality and security testing as well as for formerly manual deployment and operations activities, in a bid to introduce software into production at speed. The primary goal of any DevSecOps initiative is to enable development teams to change their mindset and adopt security practices into their daily activities.
However, Maccherone believes it’s impossible without healthy collaboration and mutual trust. In order to achieve that level of trust, Maccherone introduced a trust algorithm. “The trust formula has three terms combined in the numerator: credibility + reliability + empathy which are all divided by apparent self-interest,” he explains. “It’s important that the apparent self-interest is as small as possible, with an emphasis on shared interests.” Maccherone believes that understanding and embracing each pillar of the trust algorithm is vital to success in DevSecOps. “Credibility means that you know what you’re talking about and it’s important that you’re not just saying things for the sake of it or repeating something you’ve read,” explains Maccherone. “Writing code has changed a lot in five years. DevOps was in its early stages back then and it’s fundamentally different now. If you come into a meeting with those old mindsets, make assumptions and use outdated terminology, then the development team will pick up on that and you’ll lose credibility. Reliability is the same regardless of the context; it’s the old business expectation of making and meeting commitments. It’s important to follow through and do what you say you are going to do. Finally, empathy is all about how much compassion you show, and the awareness of how challenging something is.”
Following the foundation of the trust algorithm, Maccherone believes that it has successfully allowed for increased efficiency and has ultimately meant better decisions. “Lots of security groups at other large companies spend an inordinate amount of time cajoling development teams to do things,” he says. “The reason they have to spend such a considerable amount of time policing is due to a lack of trust. Showing empathy is crucial and it’s important to acknowledge how difficult something is to do. However, it’s also fundamental to explain why you’re trying to make the case that this risk supersedes all of those challenges and give the reasons why. It’s vital that you aren’t dictating them.” The importance of coaching rather than policing is a key aspect of Comcast’s strategy. The company also has a programme in place that provides immediate feedback to the development team while also providing aggregated metrics to guide coaching efforts. “We created a workshop where we sit down with the development team, walk through the trust formula and the company’s DevSecOps practices and give them a chance to internalise what that practice means,” explains Maccherone. “When someone feels like they’re being forced into out-of-context practices, their natural reaction is to avoid them. That isn’t what we want; we want them to reach out and partner with us.”
Change management is a key driver to Maccherone and Comcast’s strategy. “The traditional way of gathering a response was to produce surveys. However, we found that the behaviour didn't change,” he says. “We decided on a framework that we can coach from and enable the developers to reflect on whether or not they meet the criteria. If we send an email to them then we get almost no response. However, if we sit with them and allow them to ask questions directly then they instantly start changing their behaviour.” With any successful transformation comes the challenge of recruiting and retaining top talent, and Maccherone believes it’s the most challenging part of any business. “It’s the key to any tech company,” affirms Maccherone. “The HR department that we have at Comcast is fantastic. They really understand the importance of exceptional talent. Candidates want to have work that is interesting, fun and challenging, in addition to working with peers they respect.”
In a bid to achieve mutual success, Comcast Cable has established a number of key partnerships, such as with WhiteSource, Vulcan Cyber, Checkmarx, Go2Group, Contrast Security, Synopsys, Bugcrowd and Veracode. Maccherone recognises the value of forming strategic, business relationships in order to realise long-term success. “We’re at the forefront of DevSecOps, and lots of our vendors see that,” says Maccherone. “We’re constantly searching for vendors that are trying to design their products to fit in with the direction we’re going.” Maccherone believes that without developing such robust and long-standing partnerships, the challenge of reaching the level of success Comcast has achieved would have been significantly harder. “Our vendors are a key to our success and we’re extremely excited and happy with the current set we have,” beams Maccherone. “They align well with our values and that’s been the differentiator to finding ways to reduce our security risk.”
DevSecOps has become a hot topic in the technology space in recent years and Maccherone has observed its rapid rise first-hand. “Three years ago, I started a Google alert on DevSecOps and would get one hit a week or even a month,” he says. “Now, I get 10-20 every day and we’re not even at the steepest part of the adoption curve for DevSecOps yet.” In 2019, Comcast’s goal was to scale the DevSecOps programme, the tech giant achieved that by tripling the number of teams onboarded to the programme. “By the end of 2020, we aim to double that number again, and I expect that will get us close to the saturation point of all the teams at Comcast. We’ve gone from essentially launching the programme to evolving, optimising and scaling it to the point of saturation. After we reach that saturation point, I anticipate that we’ll add more capability, tools and practices over the next few years.”
When it gets difficult, this is what really gets me excited. I believe that such transformation scenarios can only be solved and addressed. But you have to do this as a team - you cannot do it as a one-man show.