CTO Rules for software product development teams

Software product teams that work well can be amazingly productive. The following rules have helped me grow high performance teams – most of the time they are all appropriate.

1 – Building the right thing is more important than building things well – but building things well is also important.

2 – Customers almost always ask for incremental improvements to what already exists. Your job is to figure out what will wow them – what they didn’t even know they wanted.

3 – Every team member should be able to do a reasonable demo of their product. This tends to get engineers more involved in design from the end users/business perspective.

4 – Product release cycle time <= 90 days. Faster is generally better. Software product development is an attempt to predict the future. The quicker to market, the shorter timeframe you have to predict.

4a – Products/projects going more than 1 year without a release have a dramatically higher probability of cancelation – total waste. Release often so you can learn/re-vector faster.

5 – Talent, not resources. Resources are things like servers, software, networks and money. People are different – they can figure out how to solve problems and do things better. That is talent at work. Instead of a project resource plan, build a talent plan.

6 – Every team member is treated with respect. We tend to attract folks to product development that are, shall we say, socially challenged. However, these socially challenged engineers tend to be really good at things that are difficult to get done. Keep them focused, and make them feel like valuable team members.

7 – Engineers get the best gear. Things like laptops/tablets/software/phones have become status symbols in many organizations – executives get the best gear. If you want better team productivity, then engineers get the best gear.

8 – Most decisions should be made by teams – not dictated by the CTO, except for the rare occasions when it really matters that the CTO should override the team. There are usually a number a ways to get something done. Let the teams be creative where your standards allow.

9 – Team roles are defined out of necessity, but every team member is responsible for their total product – its never not my job.

10 – Automation often improves team throughput more than adding team members. Make sure you spend enough time and money on speeding up the team.



Hiring Great Talent – Passion for Technology

Many organizations look more for specific skills rather than broad experience and ability to rapidly learn new technologies when evaluating technology job candidates.  The assumption is that the ramp time will be short because the candidate already has the needed skills.   In many cases this can be very short sighted.  When things change you may have the wrong technologist.  Plus it often takes more time to learn your specific systems than a particular technology.  I look for candidates that invest heavily in personal research and development – pick up new technologies on their own because they have a passion for technology.  They constantly learn new technologies on their own time.  Maybe even use vacation time to go to a technology conference.  To them it’s more than a job, it’s what they do.  Look to see how many different technologies with which a candidate can have an intelligent conversation about.  And which technologies cause a candidate to really perk up.

A job candidate also needs to have a passion for the use of technology in product and business areas, particularly if they would be working closely with those teams.  Can the candidate carry on an intelligent conversation with product and business folks?  Are they inquisitive or passive in these areas?  A candidate that is just interested in technology can become a real issue if product and business folks are looking for a technology partner.

Resume scanners can be effective at identifying potential skill sets.  But you are going to have to dig much deeper to determine the breadth of a candidate’s experience, or how rapidly they learn new technologies.  Team based interviewing (including product and business partners) can be a very effective technique to determine depth of experience and passion for technology.  In team interviews, everyone gets to hear the candidate’s responses at the same time, rather than giving a candidate an opportunity to change their responses to fit the specific interviewer.

Using a consultant can be effective for a specific project (with very specific skills).   The candidate review process can be shorter, and there are no expectations of a continued assignment after the project.  Plus if the consultant turns out to be a super star, you can hire them later.

Looking for a broad set of technology skills (and passion for technology) is critical if you are going to build teams that can adapt to changing direction.  Otherwise you may spin your wheels getting new talent up to speed.

Vulnerability Creep

“Why can’t we do (xyz) for this project – just this once.  I know it does not follow our security rules but everything else is so locked down it should be fine”.  Seems like I have this conversation a number of times in every project.  “Just this once… we are behind schedule…  please?”

Seems safe enough.  The team wants to download some open source components to speed up the project.  It’s only an informational site, so how risky can that be?  We use (top name brand) firewalls and just had a penetration test done by a well known company.  Plus all the real processing and information is in different security zones in the production cloud.  That means they are protected, and we don’t need to worry about the front end servers, right?

It is rather difficult to have complete knowledge of the environment we are working on these days.  We think we are locked down.  But a bad guy only needs to find one entry vector, out of potentially thousands of things we need to do right.

So the component is downloaded and it goes into production.  The project is a big success.  It’s used as a company example of how to do things fast.  But.. 6 months later the database administrator notices unusual activity in your member database.   Oops.

Another case of vulnerability creep.  How did this happen?

Suppose your downloaded component has a small bug that allows malware to be installed on your informational site – giving a bad guy root access to that server.  Perhaps from that server he has limited access to a cloud of application and database servers.   And perhaps the admin passwords for those servers are all the same, so he systematically guesses a password only once for each server, say once per month so as not to set off any alerts.  If there are 100 servers, that’s 100 attempts.  Per month.   So the password eventually falls.

In addition, the database connection strings are in clear text on your informational server.  What, clear text?  Tthat was required by one of the third party components.  Oops.  Totally forgot about that one.  So the bad guy only needs access to the informational server in order to get to your member database.

Other controls could be in play here as well.  Such as how does the bad guy get the data back out past our firewall?  Bad guys usually have this one figured out as well.   Stealth back channels over common ports do happen.

Security is everyone’s responsibility.  Let’s assume that a bad guy will penetrate our application or infrastructure at some point, and that our component needs to operate in a compromised environment.    When we ask for an exception to a security rule/policy, make sure we know what vulnerabilities we are potentially opening up.  We may need a compensating control.

Some day you will probably end up responding to a security incident.  But if you are thorough in development and implementation, maybe it won’t be with your system component.

Software Development Cost Drivers

How many times have you negotiated low hourly rates only to find that you go way over budget in a release?  What metrics do you use for development organizations?  Schedule might be a more effective metric to minimize than hourly rate.

I have had the privilege to lead teams that have been benchmarked (by outside entities) at up to seven times normal (whatever that is) productivity.  How did the auditors know?  They usually looked at the total number of components, amount of code and features developed.  And then looked at how many developers were on the team.  It was quite entertaining.  “Where is the rest of your staff?  This team could not have possibly built this – they couldn’t have work enough hours”.  But we hired top performers.  Implemented lean processes.  And made capital investments to keep teams really moving.

The range in productivity between top and low performers is often quoted at 10 to 1.  I have found that it is often much more than that – like 20 to 1 or greater.  Top performers are exceptionally good at their craft.  It’s fascinating that you don’t have to pay 10 times more.  It’s usually much less than 50 percent more.  This would be an awesome deal if….  If you could actually hire these folks and figure out how to use them effectively.

How do you know someone is a top performer?  Not necessarily because they have vast technical knowledge.  They get the right things done.  They know when they need to put in the time to get the right things done.  They also have a knack for figuring out what the right things are.  And for working well with their team.  And they seem to learn new technologies extremely fast.

The number one cost driver in most releases (or projects) is usually schedule.  You commit talent to a project or release, and most folks stay on the project for the duration.  You then have an established run rate.  And schedules usually slip.

How do you stay on schedule?   Remove obstacles that get in your team’s path.  Fund the right tools (that they pick) to get them productive.  Make sure decisions are made well in advance of when the team needs them to be made.  Absolutely minimize wait time.  No waiting for servers, software, bandwidth – invest for productivity.  I have often spent 10-20k per developer on resources to help productivity.

If you do many of the same kind of projects, or plan multiple software releases, focus even more on architecture and automation.   Measure how well you do.  With some effort you could even get to continuous development.  And dramatic waste reduction.  It may seem counter intuitive, but have your best developers build architectural components and development automation tools.  Don’t leave out testing tools.  Automated testing is super critical to accelerating software product release schedules.  Many organizations say they do automated testing, but do they ONLY do automated testing?

So…  you might be wondering if you can get away on spending money for productivity tools with cheaper talent.  They may make a difference in productivity to a point.  But productivity tools often increase complexity.  So you may need top folks to figure out how to get many of the tools to really work together.

It may be difficult to convince the compensation committee that you are paying more to reduce the cost of a release or project.  I’ll leave that one to you.

Talent! Not Resources.

I am amazed that most organizations use the term resources to describe their technology staff.  Their staff may even use resources to refer to themselves.  If your organization is suffering from low energy level this may be part of the problem.

Resources…  hmm…  Stuff that I can buy or rent to get things done.  Resources do specific things.  Resources don’t need benefits.  Resources don’t have a life outside of work.  Resources don’t need pep talks when things aren’t going well.  And you don’t feel awful when you have to let the resources go.

But resources are not very adaptable.  Resources aren’t creative.  Resources don’t get behind a vision of the future.  Resources don’t put in super human effort when things are not going well.  Resources can’t continuously improve your business through innovation.  And resources can’t grow into leadership positions.

How would you do things differently if you managed talent instead of resources?  Start by calling them talent.  Then treat them like they are really part of your team – trusted colleagues.   Provide information on how things are really going, what’s coming up.  Don’t keep them in the dark.  Even if things are not going well.  Let them know what is important to you and to your organization.

Get to know team members.  Their hopes and dreams.  Personal challenges they face.  What they really get excited about.

Give them room to figure things out for themselves.  Even though you know how to do every job in your organization, give them some room to be creative.  They will undoubtedly surprise you at some point.  Focus on objectives, not the details.  Talent will learn faster by figuring it out rather than following orders.  And then they will be more valuable to the organization.  But don’t let them fail big.  If you have been there before, don’t confront them head on.  Guide them to a reasonable path.  Small failures will help their learning.  But big failures can be devastating.

Help them grow personally.  Each team member will have different ambitions.   Their ambition may be to grow along a technical path.  Or perhaps into management.  Or even a pivot into a totally different role.  Sometimes this may also mean they will need to leave the organization to follow their dreams.  Make a deal with them – you will help them get there if they find and train their replacement.

Give them a variety of experiences – technologies, projects, processes, and team interactions.  Having 10 years of the same 1 year of experience is not very valuable to an organization.

Keep teams together so they can develop personal relationships.  Let teams hire their new team members.  They will be able to figure out who can help get a difficult release out the door, who will watch their back in a crisis.

Recognize teams and team members when they go above and beyond.  Or when they build something incredibly cool and useful.   Salary compensation is just a maintenance factor for many IT professionals.

Start calling them talent.  Give them objectives and guidelines.  What things they can change, and what things need to be leveraged.  Paint a vision of the future that they can get behind.   Check in with them regularly.  Course correct when you need to.  But let them run with it.

You are leading talent, not resources.  Emotions matter.  Motivation matters.  It’s up to you to lead your teams to win.

The Zero Downtime Release

One of our financial providers (one of the top 3) released a new version of their online web/mobile banking a while back.  They were up and down for much of 24 hours.  Yes, they notified us a day in advance that this was going to happen, but I find it really hard to believe that a major business takes that long to cutover to a new release in today’s online world.  24 hours.  Wow.

It’s actually possible to install/cutover a new release while production is up.  Yes, even with non-trivial sites.  I’m surprised at how many folks don’t believe this.

But don’t most sites go offline for new product releases?  Unfortunately, many sites do.  Perhaps the release process is risky, and everything needs to be verified.  Or perhaps the site is still in rapid feature build mode, and can’t afford to take time to automate internal processes.  If the site has to go down for a release, then the tendency is to put more features into a release.  This makes the release even more complex and costly.  Often on weekends, with all hands present to quickly resolve issues.  Great way to spend a weekend.

Automating development processes, however, can make the release process very fast, routine and inexpensive.  Perhaps even a push button release.  What kind of competitive advantage would that give your business?  You would not have to wait for a specific release window to push features required for a new customer.  What if, instead of a release every quarter, you could do a release every week?  Or every day?  Or even several times a day?  Your product team might be free to test new features in production, instead of offline.  If the features are popular, leave them in.  If not, take them out.

It does take work to get to the point where you can install a new release while production is up.  Even more work with complex core systems.  You may need to find and cut out activities that are not really adding much value – but that you may believe are necessary because you have always done them.  It also requires automating more of your development/release value chain.  Automating things like software builds, (complete) regression testing, and the actual cutover to production.

It may take a while to get to the point where you can cutover to a new release while production is live.  There is a learning curve.  You will need to get your teams on board.   It may be best to set some short term objectives – knock down the hurdles so you can make progress towards this goal.

Hold your team accountable for uptime.  Make sure you include new release cut over time in your uptime reporting.  Your customers do.