Notes on SoftwareEngineering 1998

Last updated September 27, 2021

Here are some of my notes on Software Engineering and random thoughts from a lecture I give in 1998. Although this is pre-Extreme Programming (XP) the concepts are still valid.

1. Background

   Why is software so hard to write?

   1. Initial requirements for the project are rarely well understood even by the business unit
   2. Requirements change as the business climate changes sometimes within the same month
   3. Environment for running the software is constantly changing
   4. Software really is complex
   5. Employee turnover
   6. Locality Problem
2. Why is it called engineering?
   [Bridge example]
3. Recommended reading:
   The Mythical Man-Month by Frederick P. Brooks Jr (1975)
   Details work on the operating system for the IBM 360 (rumored to cost more than the Manhattan project).

   Some of his thoughts:

   1. Adding more people to a late project only makes it later
   2. Two Man-months done my one person is not the same as two people doing one month of work in software development
   3. Last 10% of a project may take 50% of the resources.
   4. Programmers vary greatly in talent (up to 10 times)
   5. Architecture should be defined by one or two people
   6. Beware the "Second System Effect". This is temptation to add too many features to the second system built.
   7. "Never go to sea with two chronometers; take one or three." In documentation always have either the formal definition or the prose definition dominant.
   8. With regard to working systems: "Plan to throw away one away; you will anyway".
   9. Software entropy goes up. "All repairs tend to destroy the structure, to increase the entropy and disorder of the system. Less and less effort is spent on fixing original design flaws; more and more is spent on fixing flaws introduced by earlier fixes. As time passes, the system becomes less and less well-ordered. Sooner or later the fixing ceases to gain any ground... A band-new, from-the-ground-up redesign is necessary."
   10. That is the key to history. Terrific energy is expended - civilizations are built up - excellent institutions devised; but each time something goes wrong. Some fatal flaw always brings the selfish and cruel people to the top, and then it all slides back into misery and ruin. In fact, the machine conks. It seems to start up all right and runs a few yards, and then it breaks down. -C.S. Lewis Mere Christianity p54
   11. To control the schedule one must have a schedule with specific milestones.
   12. There is no single development in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity.
   13. "I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation."
   14. "I am, after all, a programmer by background, and optimism is an occupational disease of our craft."
   15. Conceptual integrity is the most important consideration in system design.
   16. There is a cost-optimum schedule time to first shipment, T=2.5(ManMonths)^1/3. Hardly any project succeed in less than 3/4 of the calculated optimum schedule, regardless of the number of people applied!
   17. "The quality of the team is by far the largest factor of success, indeed four times more potent than the next largest factor."

   In 1986 he wrote a paper called "No Silver Bullet" in which he said that software was hard to write and no miracle cure for faster development would be found.

4. Software Engineering - what is it?

   Software Engineering is the establishment and use of sound engineering principles in order to economically obtain software that is reliable and works efficiently on real machines.

   Fritz Bauer (1969)
5. A Brief History

   At the dawn of the computer age, the predominate Software methodology was the "Waterfall Model". (The Build-Test-Fix model did not work well in large systems). In the "Waterfall Model" all the steps in software development are done sequentially. The steps are roughly as follows:

   1. Requirements
   	2. Specification
   		3. Planning
   			4. Design
   				5. Implementation
   					6. Testing
   						7. Operation

   The Waterfall method works reasonably well for large, static, well known systems. But it has several disadvantages:

   1. Changes in the system are hard to implement
   2. Risk management is difficult
   3. Projects must be well defined at the beginning
   4. Cost of fixing errors in requirements increases by 10X at each stage

   The next generation of methodologies emphasized Rapid Prototyping and customer feedback. Barry Boehm developed the "Spiral Model" in the mid 80s. Another Rapid Prototype methodology today is called "RUP", the Rational Unified Process.

   Rapid Prototyping's basic premise is that software should be created by doing small self-contained systems which are reviewed by the customer in a series of prototypes. It's somewhat like a series of waterfall models with the scope increasing each time and risk assessment done at the beginning of each series. (The riskier parts of the project should be investigated sooner rather than later). The customer can review the prototype and give feedback on the requirements.

   When all the series are complete, the Prototype code should be thrown away. The entire system should be redesigned, and rewritten from the ground up now that the requirements are well understood. (In real life the prototype is usually FedExed straight to the customer the next morning).

6. Engineering: Functionality, Quality, Cost, and Time.
   My definition:
   The art of engineering is building systems in a timely, cost efficient manner, which conforms to requirements.

   To build a software system well, the trade-offs in Functionality, Quality, Cost, and Time should be made at the beginning and well understood by all. A project is a compromise of these four items.

   consider,

   Functionality + Quality = Cost + Time

   Is it true? If so, what does the above imply?
   (Please note that the solution to the function above is not continuous. Even with enormous sums of money, most projects cannot be built in 3 hours)

7. Economics: Time vs. Functionality.

   Civil Engineering Example: A water pipeline is needed to bring water to a growing new subdivision. At the end of 20 years the estimated water need is 300 ft^3/min (cfm). At the end of 40 years the need for water is expected to grow to 600 ft^3/min.

   Two alternatives are proposed:

   1. Build a 2 foot diameter pipe now that supplies 300 cfm at a cost of $1,000,000. In 20 years build an additional 2 foot pipeline parallel to the original at a cost of $1,000,000.
   2. Build a 3 foot diameter pipe that would immediately supply 600 cfm at a cost of only $1,350,000.

   Which is the best alternative?

   How does this apply to Software Development?

8. The SwampRoad Analogy
9. This is an analogy that I've devised to visualize the software development process.

   Background: Two Roman cities in 50 BC are separated by a wide wetland. Trade is conducted using rafts, but its tedious. The wealthy merchants and townspeople decide to build a causeway over the marsh for trade. What is the best way to construct the path through a swamp to connect two cities? Two architects have devised two different construction plans.
   Notes: The wetland has pockets of quick sand. It is estimated to take 6 months to build it and will be designed for the width of two ox carts.
   
   Figure 1. Cities Alpha and Beta
   
   Figure 2. Proposed Causeway
   
   Figure 3. Marcus Broadus's Plan For Building the Causeway
   
   Figure 4. Brutus Isthmus's Plan For Building the Causeway
   
   What are the advantages and disadvantages of each plan?

   Lessons to be learned:

   1. Concentrate on getting the code written that has to be there for the minimal use. Delay writing the frill stuff till last. Perhaps the user really doesn't need the frills, or more likely after the initial review, she needs something else desperately, right now instead.
   2. Not all components have to be initially built with all functionality. If you can delay full functionality till later (and it doesn't cost you too much in retrofitting), consider delaying full implementation later.
10. Misc.
    1. The Software Engineering Institute at Carnegie-Mellon University defined a Capability Maturity Model (CMM) in 1986 with 5 levels of maturity.
       

       Level	Description
       1. Initial	The software process is characterized as ad hoc, and occasionally even chaotic. Few process are defined, and success depends on individual effort.
       2. Repeatable	Basic project management processes are established to track cost, schedule, and functionality. The necessary process discipline is in place to repeat earlier successes on projects with similar applications.
       3. Defined	The software process for both management and engineering activities is documented, standardized, and integrated into a standard software process for the organization. All projects use an approved, tailored version of the organization's standard software process for developing and maintaining software.
       4. Managed	Detailed measures of the software process and product quality are collected. Both the software process and products are quantitatively understood and controlled.
       5. Optimizing	Continuous process improvement is enabled by quantitative feedback from the process and from piloting innovative ideas and technologies.

    Organizations start at level 1 and through sustained effort move upwards. In addition to having some good ideas, SEI rating is useful for work with the government.
11. Mitch's personal tips and experience
    1. Make sure the functionality of your current build is what the customer really wants.
    2. Don't write your own basic components.
    3. Its way cool to write your own specialized, highly-optimized linked list code, but its much cheaper and faster to use a reputable one from a commercial package or a respected source on the net. The real cost in writing your own components comes, not from the time in coding, but from testing and the danger if they fail. ("I never thought about the case where the list was empty and I try to remove an item...")
    4. Remember: the complexity of a product is proportional to the square of the number of features.
12. Misc topics related to Software Engineering
13. 1. Metrics
    2. Function Points
    3. Locality Problem

    PS: Never release a product in December - you don't want to miss Christmas, and your customers are too busy to receive the product.

"There is nothing so useless as doing efficiently that which should not be done at all" -Peter Drucker

This document is located at www.fincher.org/tips/General/SoftwareEngineering/SoftwareEngineering.shtml