Introducing DevSecOps at scale
02
Comcast: Introducing DevSecOps at scale
03
w w w.c o mc a stc o rpo ra t i on . com
COMCAST
Larry Maccherone, Distinguished Engineer of Comcast, discusses DevSecOps’ growing influence on one of the world’s biggest telecommunications companies
A
s a global leader in media and technology, Comcast is the parent organisation
of three primary businesses: Comcast Cable, NBCUniversal, and Sky. Comcast has more than 04
55 million subscribers, with Sky renowned as one of Europe’s leading entertainment companies operating in seven territories and Comcast Cable recognised as one of the biggest cable TV, high-speed internet, and phone providers in the United States. Sitting down in the new Comcast Technology Centre at its headquarters in Philadelphia, Pennsylvania, Larry Maccherone, Distinguished Engineer of Comcast Cable, shared how the company is uniquely positioned for success in their agile approach to achieving a DevSecOps cultural transformation. Maccherone’s professional background heavily revolves around data analytics and Lean-Agile, and he started his first business while still an undergraduate at university. “I’ve been a serial
05
w w w.c o mc a stc o rpo ra t i on . com
Stop managing. Start fixing. The vulnerability management platform built for remediation
Handing you a solution Vulcan Cyber curates the best solution for any given vulnerability, whether it be a patch, configuration change or workaround
Initiating and orchestrating cross-team remediation workflows Vulcan Cyber helps establish cross-team alignment by managing and tracking all the steps in workflow from initial discovery to a successful remediation
Highlighting the organization’s biggest risks By prioritizing the vulnerabilities in your organization according to the true business-risk they pose to your environment, the platform minimizes workloads and lowers costs
Automating the grunt work
By orchestrating the cyber hygiene lifecycle, Vulcan Cyber makes the repetitive, most time consuming tasks automatic
Learn More
COMCAST
“ I believe that if you’re doing DevOps right, then the security part is just automatically included” entrepreneur throughout my entire — Larry Maccherone, Distinguished Engineer, Comcast
career. My first business had 80 employees and made US$20mn annually in sales,” explains Maccherone. “We were writing software that controlled a large portion of the world’s power generation,
08
and it meant that if hackers exploited a vulnerability in the software, then it potentially brought down the world’s power grid. We got really skilled at writing software that didn’t have exploitable vulnerabilities.”
DevSec CLICK TO WATCH
|
2:07
09 Upon joining Comcast in June
as empowered engineering teams
2016, Maccherone became responsi-
taking ownership of how their prod-
ble for overseeing the company’s
ucts perform in production, including
DevSecOps transformation. “I have
security. When you get development
a love/hate relationship with the
teams owning the problem, you
term DevSecOps. I believe that if
get a fundamental difference in
you’re doing DevOps right, then
decision making.”
the security part is automatically
Since its creation over a decade
included,” he explains. “You don’t
ago, DevOps has become a vital
call it DevTestOps or DevPlanningOps,
component of how companies oper-
it’s just DevOps. However, what
ate. Building upon the foundations
I do like about DevSecOps is the
of the agile movement, DevOps lev-
emphasis on security. My definition
erages automation, for quality and
of DevOps and DevSecOps is
security testing as well as for for-
essentially the same. I define both
merly manual deployment and w w w.c o mc a stc o rpo ra t i on . com
You change the world, we’ll secure it. Secure sof tware is crucial in our digital world. And we believe in focusing on your security.
So you can conf idently create the sof tware that helps move our planet for ward.
So you can focus on your business.
So you can be the hero of your stor y.
Cox Automotive, one of the world’s largest automotive service providers, is pushing its industry forward with software. Cox Automotive’s vision -- to transform the way the world buys, sells, owns and uses cars -- relies
Learn More
on software, and DevSecOps provides speed and security for its applications. By working with Veracode, Cox Automotive is realizing its vision, without being held back by security.
COMCAST
operations activities, in a bid to intro-
to achieve that level of trust,
duce software into production at
Maccherone introduced a trust algo-
speed. The primary goal of any
rithm. “The trust formula has three
DevSecOps initiative is to enable
terms combined in the numerator:
development teams to change their
credibility + reliability + empathy
mindset and adopt security practices
which are all divided by apparent
into their daily activities.
self-interest,” he explains. “It’s impor-
However, Maccherone believes
tant that the apparent self-interest
it’s impossible without healthy collab-
is as small as possible, with an
oration and mutual trust. In order
emphasis on shared interests.”
E X ECU T I VE P RO FI LE 12
Larry Maccherone Larry Maccherone is a Distinguished Engineer at Comcast where he currently leads the DevSecOps transformation initiative. Previously, Larry served as the Insights Product Line Director at Rally, where he published the largest ever study correlating development team practices with performance. Before that, Larry worked at Carnegie Mellon with the Software Engineering Institute (SEI) and CyLab conducting research on cybersecurity and software engineering. While there, he co-led the launch of the BuildSecurity-In initiative. He has also served as Principal Investigator for the NSA’s Code Assessment Methodology Project, on the Advisory Board for IARPA’s STONESOUP program, and as the Department of Energy’s Los Alamos National Labs Fellow. He speaks and publishes regularly on DevSecOps, Lean/Agile, and analytics and he is the primary author of a dozen open source projects, one of which gets 400,000 downloads per month.
13
Maccherone believes that under-
in its early stages back then and
standing and embracing each pillar
it’s fundamentally different now.
of the trust algorithm is vital to suc-
If you come into a meeting with those
cess in DevSecOps. “Credibility
old mindsets, make assumptions
means that you know what you’re
and use outdated terminology, then
talking about and it’s important that
the development team will pick up
you’re not just saying things for the
on that and you’ll lose credibility.
sake of it or repeating something you’ve read,” explains Maccherone. “Writing code has
Reliability is the same regardless of the context; it’s the old business expectation of making and meeting commitments. It’s important to follow
changed a lot
through and do what you say you
in five years.
are going to do. Finally, empathy
DevOps was
is all about how much compassion w w w.c o mc a stc o rpo ra t i on . com
Time is precious.
Automate Vulnerability Remediation
IDENTIFY COMPONENT
DETECT
VULNERABILITY
ENFORCE POLICY
AUTOREMEDIATE
Learn more about us at whitesourcesoftware.com
“ If you come into a meeting with those old mindsets, make assumptions and use outdated terminology, then the development team will pick up on that and you’ll lose credibility” — Larry Maccherone, Distinguished Engineer, Comcast
supersedes all of those challenges and give the reasons why. It’s vital that you aren’t dictating them.” The
you show, and the awareness of
importance of coaching rather than
how challenging something is.”
policing is a key aspect of Comcast’s
Following the foundation of the
strategy. The company also has
trust algorithm, Maccherone believes that it has successfully allowed for increased efficiency and has ultimately meant better decisions. “Lots of security groups at other large companies spend an inordinate amount of time cajoling development teams to do things,” he says. “The reason they have to spend such a considerable amount of time policing is due to a lack of trust. Showing empathy is crucial and it’s important to acknowledge how difficult something is to do. However, it’s also fundamental to explain why you’re trying to make the case that this risk w w w.c o mc a stc o rpo ra t i on . com
15
50m lines of code driving the kids home The more powerful you make software, the more inseparable DevOps and security need to be.
See why Checkmarx leads the industry in delivering automated security scanning for DevOps.
Checkmarx.com
WORKING@COMCAST CLICK TO WATCH
|
1:08
17 a programme in place that provides
what we want; we want them to reach
immediate feedback to the develop-
out and partner with us.”
ment team while also providing
Change management is a key
aggregated metrics to guide coach-
driver to Maccherone and Comcast’s
ing efforts. “We created a workshop
strategy. “The traditional way of
where we sit down with the develop-
gathering a response was to produce
ment team, walk through the trust
surveys. However, we found that
formula and the company’s
the behaviour didn’t change,” he says.
DevSecOps practices and give
“We decided on a framework that
them a chance to internalise what
we can coach from and enable the
that practice means,” explains
developers to reflect on whether
Maccherone. “When someone feels
or not they meet the criteria. If we
like they’re being forced into out-of-
send an email to them then we get
context practices, their natural
almost no response. However, if
reaction is to avoid them. That isn’t
we sit with them and allow them w w w.c o mc a stc o rpo ra t i on . com
to ask questions directly then they instantly start changing their behaviour.” With any successful transformation comes the challenge of recruiting and retaining top talent, and Maccherone believes it’s the most challenging part of any business. “It’s the key to any tech company,”
1963
Year founded
$108.9bn Revenue in US dollars (2019)
affirms Maccherone. “The HR department that we have at Comcast is fantastic. They really understand the importance of exceptional talent. Candidates want to have work that is interesting, fun and challenging,
190,000 Number of employees
w w w.c o mc a stc o rpo ra t i on . com
19
COMCAST
20
“ Three years ago, I started a Google alert on DevSecOps and would get one hit a week or even a month. Now, I get 10-20 every day” — Larry Maccherone, Distinguished Engineer, Comcast
21
w w w.c o mc a stc o rpo ra t i on . com
LEARN MORE
Comcast Partners CLICK TO WATCH
|
1:01
23 in addition to working with peers
are trying to design their products
they respect.”
to fit in with the direction we’re going.”
In a bid to achieve mutual success,
Maccherone believes that without
Comcast Cable has established
developing such robust and long-
a number of key partnerships, such
standing partnerships, the challenge
as with WhiteSource, Vulcan Cyber,
of reaching the level of success
Checkmarx, Go2Group, Contrast
Comcast has achieved would have
Security, Synopsys, Bugcrowd and
been significantly harder. “Our vendors
Veracode. Maccherone recognises
are a key to our success and we’re
the value of forming strategic, business
extremely excited and happy with
relationships in order to realise long-
the current set we have,” beams
term success. “We’re at the forefront
Maccherone. “They align well with
of DevSecOps, and lots of our vendors
our values and that’s been the differen-
see that,” says Maccherone. “We’re
tiator to finding ways to reduce our
constantly searching for vendors that
security risk.” w w w.c o mc a stc o rpo ra t i on . com
Integrate security into your DevOps environment code
deploy
test
se ea re l
Ops
operate
build
an
pl
Dev
m o n ito r
Automate application security testing with the development and deployment tools you use today. Read why Gartner selected us as a leader for Application Security Testing
“ When someone feels like they’re being policed, their natural reaction is to avoid the police. That isn’t what we want; we want them to reach out and partner with us” — Larry Maccherone, Distinguished Engineer, Comcast DevSecOps has become a 25
hot topic in the technology space in recent years and Maccherone has observed its rapid rise first-hand. “Three years ago, I started a Google alert on DevSecOps and would get one hit a week or even a month,” he says. “Now, I get 10-20 every day and we’re not even at the steepest part of the adoption curve for DevSecOps yet.” In 2019, Comcast’s goal was to scale the DevSecOps programme, the tech giant achieved that by tripling the number w w w.c o mc a stc o rpo ra t i on . com
27
of teams onboarded to the pro-
we reach that saturation point,
gramme. “By the end of 2020,
I anticipate that we’ll add more
we aim to double that number again,
capability, tools and practices over
and I expect that will get us close
the next few years.”
to the saturation point of all the teams at Comcast. We’ve gone from essentially launching the programme to evolving, optimising and scaling it to the point of saturation. After w w w.c o mc a stc o rpo ra t i on . com
1701 John F Kennedy Blvd. Philadelphia PA USA ComcastCorporation.com