May 26 2011

Software Development Lifecycle – Training

Lately I’ve been working on incorporating a Software Development Lifecycle (SDL) in the development processes of a smart grid vendor for their smart grid products.  It’s no secret that everyone (from the vendors to the utilities to the public utility commissions to NERC and FERC) are concerned about the rush to deploy smart grid or – more aptly – advanced metering infrastructure (AMI) systems.  There are many issues that need to be considered when doing an advanced metering infrastructure – internal security at the utility; securing the endpoint devices; the security of the connecting network.  All of these are things to rightly be concerned about.  However, very few smart grid vendors have focused on the builtin security of their software.  I’m not talking about all of the bells and whistles that they provide to secure the AMI infrastructure…I’m talking about the quality of their code.  It’s all well and good to have lots of security features that your customers can turn on and off…but what lurks under the engine?  Buffer overflows? Heap overflows? Cross-site scripting?  Cross-site request forgery? I could go on and on.  To deal with these concerns and potential vulnerabilities I’ve been working on implementing Microsoft’s Security Development Lifecycle (SDL) in our product development groups.  This has been a real challenge given that we previously didn’t worry about such issues since meters (electric, gas, and water) were isolated, electro-mechanical devices that didn’t have two-way (or in some cases even one-way) communication capabilities.  I plan to post updates with implementing an SDL in this blog in hopes that others learn from our experience.

One of the primary components of an SDL is a software security training program.  Developers and development management tend to focus on one thing primarily – writing code and getting it working as fast as possible.  In many cases security is not even an afterthought and even if it is given some consideration many developers don’t have the experience in writing code with security in mind.  This is where a software security training program is essential.  It needs to cover a wide variety of topics such as an overview of the SDL process, secure coding in C/C++/Java/.NET, threat modeling, and secure software architecture to name a few.  In today’s market there are two options in software security training for an organization that is looking to stand up an SDL:

  1. Do it yourself
  2. Outsource

From a “do it yourself” perspective one of the hardest parts is finding people skilled at secure coding within an organization that is already possibly behind the curve on software security.  All content would be developed internally – and there’s the Catch-22 situation: how can you develop the content when your staff doesn’t have the skills necessary to write the content which needs to be taught? In addition to that you will need to setup a learning management system (LMS) in order to track developers as they go through the training which is internally developed (or perhaps bought).

In many cases the only viable alternative is to outsource.  Outsourcing should leverage both instructor-led training (ILT) and online classes.  The only thing to decide is The question is which philosophy do you subscribe to with regards to training: ILT training first with online classes as a reinforcement/refresher or online classes first followed by ILT.  I’ll try and explain both approaches below:

Leveraging ILT before going proceeding to online training is based on the idea of getting the most, in-depth training upfront is the key component of the training and the online classes are just there for reinforcement of the material learned in the ILT classes.  In addition the online classes can be used as refresher classes after some specified period of time – say, approximately a year – after the initial ILT/online classes have been taken.  The trick is that the online class content needs to be updated during that time…otherwise it becomes stale and loses value for the developers.  The big benefit here is that you put a lot of effort upfront to get your developers trained and can leverage that training as soon as possible.

Flipping the sequence around has the online training occurring before the ILT classes.  The philosophy here is that the developers get a broad knowledge of the SDL and its various components and then you’re able to focus the ILT more effectively to provide the attendees a class that explores the content more completely.  One of the big benefits to this approach is that the developers get a broad education in what an SDL is and what steps are part of the overall process.  This allows you to provide some training to all of your developers (of course that depends on how many seats you buy for the e-learning system) and to take those who are key and provide them the ILT first.

It’s hard to say which is the better approach – too many factors to consider: cost and schedule being the primary ones.  It is my belief that both approaches are equally valid.  I would also stress that it depends on how big your developer population is and how quickly you need to get some training started.  From my own perspective I think the idea of starting the e-learning first and then moving to an ILT is more effective – it allows for your developers to all start at the same knowledge level before having them go through the ILT.  It also doesn’t prevent you from using the e-learning later as a refresher for the material that the learned in the ILT.  I’d be interested in hearing other’s thoughts as well on this.

2 responses so far

2 Responses to “Software Development Lifecycle – Training”

  1. Andre Girondaon 26 May 2011 at 5:59 pm

    I don’t think that online training works at all.

    Look at the “industry best” solutions for secure appdev training:
    1) PluralSight
    2) Security Innovation
    3) SafeLight Security
    4) ThreadStrong.com

    Go ahead and check them out. I think you will be surprised. At how bad they are. Even after me saying this, you will be further surprised at how bad they are compared to what I’m even alluding to. They are that bad. You can think “wow, there has to be more than just an explanation about parameterized queries” but really there is not. They don’t explain how the Command and Plugin patterns can screw everything all up. They don’t mention looking for string concatenation operations in third-party components or the callbacks to the frameworks themselves. They don’t even mention the basics laid out in the book, “The Art of Software Security Assessment”.

    Developers don’t learn how to do their job from Youtube or even KhanAcademy.org. Using the “theory of multiple intelligences” from Howard Gardner, we can qualitatively decide that application developers are not Spatial/Visual learners by trade. Perhaps some are, and perhaps some are to some degree.

    App developers also tend to work on projects alone. It’s way more common than most would have you believe. Sure, in most Enterprises and large organizations, they’ll say that it’s a team of developers. They’ll say a lot of things in Enterprises and large organizations (see: Dilbert).

    In my experience, every “training solution” must be customized to each application developer and his or her specific learning, process, and technology needs. So, this isn’t training-level instruction… this is tutoring.

    In reality — you don’t “train” people to do their jobs. They learn how to do their jobs on their own. Especially if they are writing public-facing software components for a web application that hosts millions of users’ private information such as their payment card information or their banking or healthcare details — let alone their email and pictures.

    What security professionals need to do is give application developers the tools that they need to do their jobs in the best way — without making any significant changes. Is the use of a security-focused static analyzer in every IDE or build server a significant change? Yes.

    Developers need to know what they need to do their jobs. So it’s the application security professional’s job to help them write their code-construction checklists (e.g. self-desk-checks) and design/API wikis (e.g. Confluence, Mediawiki, etc). This is where tutoring can take place.

    You probably don’t need one application security buddy for every application developer. But you do need a ratio of some kind.

  2. idubrawskyon 27 May 2011 at 2:50 pm

    Good points Andre. I agree with you that e-learning by itself is no good…and that’s where follow-up Instructor Led Training (or even the reverse: ILT without reinforcement) is critical otherwise the e-learning is a colossal waste of time. I’ve looked at some of the vendors you mentioned and realized that without ILT there was no way that the developers would be able to internalize anything and we would be wasting money. I also agree that you can’t “train” people to do their jobs – which is why we’re approaching this effort as education rather than training. In our case we’re mixing both training (online and ILT) along with periodic internal classes as well as other components of the SDL. From our perspective we have to start somewhere and build the capability from zero. It’s something that the developers must internalize over time – which means we will have to have refresher “classes” to ensure that we reinforce the knowledge that the developers have learned – otherwise they will simply go back to old habits. I’m not disagreeing with you but I see this as a necessary evil that we have to use until the SDL is fully internalized across the entire development lifecycle.

Trackback URI | Comments RSS

Leave a Reply