Lessons and Wisdom in building Security Champions

In the Security Champion Program Success Guide blog, I invite industry professionals to offer their opinions, thoughts, experiences, and viewpoints about Security Champion programs in order to expand our collective knowledge and encourage a healthy conversation.  Here are some insights in building Security Champion programs through the eyes of highly respected AppSec Elder Diplomat, Brook Schoenfield.  Enjoy! — Dustin Lehr


Using Champions to Scale Security Architecture

In 2002-3 at Cisco, Michelle Koblas and I came to the conclusion that we would never fulfill our security architect staffing through hiring. I evangelized our then rather nascent, and perhaps just a bit naive, concept for federating security architecture by drawing from the architects in each partner organization to each partner (who were generally all-in), HR (who were suspicious), and Enterprise Architecture. Enterprise Architecture told us to wait time and again, for a year and a half. In 2004, I got another, “Just wait; we’re not ready” from Enterprise Architecture to my mounting frustration. Ferris Jabri was my Program Manager at the time. I’ll never forget it: Ferris said, “Brook, we’re just going to do it”, so we started. Within a year, nearly every organization, including Enterprise Architecture, were asking us how we did it, the program had such demonstrable results. The program carries on to this day.

Challenges with Process Control

Before we federated security architecture at Cisco, Infosec’s Web Architecture and Infrastructure team dutifully posted assessments, risk ratings, requirements, and exceptions from each engagement to a central repository. We could munge our collected data fairly easily for metrics. There were very few misses; every team member was thorough in documenting what they’d done. After empowering (and training!) our partner security architects, they really didn’t have time to perform this extra step. Our satellite champion program lost the ability to archive the results of every security engagement. Logging results of engagements became spotty at best. It was a powerful lesson that federation with empowerment comes at a cost of loss of control over the process. We had to invent new ways of staying current that hadn’t existed when it was just the central team.

Learn the Culture to Build a Community

Existing culture matters, sometimes a lot. After having built and technically led three software security programs including fostering a shared sense of community, I thought I was pretty good at this game. Then I got to McAfee, where its many acquisitions over many years had been kept in relative product-team isolation. Plus, before the current management at the time I arrived, let’s just say that corporate culture had not been particularly supportive of publicly identifying challenges (with all due respect to the efforts of the team of execs who, at that time, had been trying hard to shift to collaboration.) Unlike previous efforts at other organizations, I felt like I was pulling teeth at every meeting: champions just would not take any chances; it was clear to me that speaking up wasn’t safe. I felt like a complete failure, frankly. I just couldn’t get folks to believe that they could safely engage. What finally shifted things was addressing externally reported vulnerabilities within the community. I believe it was the shared sense of responsibility that finally made it safe enough for folks. It took 18 months to get there. None of my previous communities of practice had taken nearly so long to come together. I learned that shifting cultures can take a long time and a lot of effort. But it was worth it. That community could move mountains together.

— Brook Schoenfield


Want to share your own viewpoint or case study about security champions? Contact me on LinkedIn and let’s chat! — Dustin Lehr