Software Engineering for Everyone?


Dennis E. Hamilton, System Architect

Revision History:

This is draft 0.05 of 2000-05-26


1. Who Needs SE4E?

2. Accountability: What Happened?

3. Collegiality: Community Accomplishment

4. Predictability: Make It So, Keep it So

5. Practice, Practice, Practice: Never Stop

6. References and Resources


Change History


1. Who Needs SE4E?

I'm delighted by the Computer Programming for Everyone (CP4E) project.  I dream of computer technology and programming being accessible to anyone: using the technology itself to support people in exploring and mastering as much as they choose; using the technology to be less inscrutable and more open to investigation and understanding by computer users.  I'm particularly taken by the idea of having support for programming be available for anyone to partake of whenever it's needed, for as much as it's needed.

Discussions about CP4E include proposals for reforming school mathematics (or not) and for encouraging computer-application programming as an exploratory experience made available to all young people [Edu-SIG].  The Python language is featured in discussion of programming environments appropriate for CP4E initiatives.  

It's been asked whether software engineering has any role in a CP4E syllabus:  Is there such a thing as Software Engineering for Everyone, and what does it have to do with CP4E?

I share the view that it is essential for CP4E to foster playful discovery and experimentation, especially in opening up neophyte experiences of self-directed exploration [Hacker].    Let's emphasize programming as an avenue for creative expression and improvisation without artificial constraints on early practice.

At the same time, CP4E provides a wonderful opportunity for the curious to appreciate the discipline involved in engineering.  There is, in CP4E, an unparalleled avenue for people to experience the level of attention that it takes to successfully delivery complex systems into an imperfect world, whether for commercial air travel, distributing electrical power, operating the postal service, visiting a comet, or giving depositors reliable access to their own accounts via automated teller machines anywhere in the world.  

There are simple everyday practices, not specific to software engineering as such, that are valuable to incorporate in CP4E, especially group projects of any kind.  It's useful to know how engineers win at their game, and there are elements that apply to any purposeful activity.  I am thinking of three as the basis for SE4E: accountability, collegiality, and predictability.

2. Accountability: What Happened When?

My first computer-related job was in 1958 as an engineering aide.  I had dropped out of college after two quarters, and the calculus plus high-school drafting classes got me the work.  My engineers wrote everything down. Everything.  I spent all of my time creating graphs and charts from the models we were exercising using programs for an IBM 701 computer system.  Everything was checked, signed, bound, and filed. 

I resented the tediousness of the work, yet I've never forgotten the care that was taken in having complete, accurate, and reproducible records.  I never feared riding in a Boeing 720 when they were later produced.  

In the course of this job, the lead engineer and I began teaching ourselves FORTRAN.  This was my first contact with a power user, and it is clear to me now what motivated him to learn to develop his own programs.  He wanted a way to have the program outputs be in the exact form that he demanded for the analyses and engineering reports we were producing, rather than suffering with the difficult-to-digest output that the programming staff was satisfied to produce.

In Watts Humphrey's Introduction to the Personal Software Process, students record their time and keep a log book and journal very early [PSP].  This becomes a progressive historical account of actual effort, including the ideas that were tried, the problems that were discovered, and how the course of a project evolves over time.   It also provides insight to the engineer, scientist, or student about where activity is spent and how long it takes to accomplish things in a genuine, not imagined, workday.  I recommend that notebooks be encouraged very early.

It is easy to introduce practices of accountability while young people are working on their first projects and flexing their abilities to accomplish something by exploration and creation. Providing accounts for what was done and what was involved leaves an useful sense of the responsibility with which software engineering is undertaken. It applies in science and engineering generally, and I would want young people left with a sense of that, even if they never pursue a technical career.  

Notebook records are very powerful for situations that do not turn out as intended.  Something goes wrong.   And computer programming provides ample opportunity for that experience.  This is also where ones tolerance for error and attention to its removal is exercised.  A notebook is a place for being systematic, making sure all bases are covered, and use in review of problem-removal actions.

It's taken me years to use notebooks consistently.  Nowadays, one 100-page spiral notebook lasts about two months.  I write a lot more than that every day, along with reading as many as 200 e-mail messages, yet my notebook has a distinct separate use.  It is rarely far from reach, especially if I am about to start something dangerous with my computer.  My notebooks are lifesavers for backtracking and reconstructing changes to my computer systems after problems materialize.  I carry my physical notebook on a bus, in the car, into meetings and seminars, and on air trips.  Checklists for projects, deliverables, and trips will be started and completed in my notebooks.  Test sequences and defect discoveries are recorded immediately in my notebook, well before being transcribed into more formal records or being communicated on-line.  

I carefully do not keep any working notebook on-line, though I also keep on-line notes in my work, along with other on-line notes about my work.  I'm obsessive about leaving on-line comments for myself in programs, in keeping backup and historical copies of programs and documents, and also keeping collections of e-mail exchanges in which decisions are proposed and important technical questions raised.  

My point is that basic accountability -- keeping tangible records of what was done, what happened, and ideas that came up -- builds awareness of how ones energies are expended.  It reflects how accomplishment occurs over time.  It leaves behind an accounting of key decisions and also what the consequences were later found to be.  

3. Collegiality: Community Accomplishment

I have been looking for a better word than collegiality. I am giving up that search.  Teamwork is part of it.  Scholarship is another.  I mean being a willing, self-conscious participant in a worthy activity that engages a community of participants separated in space and across time.

I grew up on a steady diet of pre-Sputnik science fiction. I like to say that Robert Heinlein taught me to read. In all of that reading, I thought of being on the moon or going to Mars as something that would be a personal, individual act. The reality of space flight and the magnitude of the enterprise that it took to have human beings walk on the moon and return just wasn't the way I dreamt it would be. Today I can be moved to tears by the magnificence of that undertaking and the contributions that so many people made in so many ways to bring space flight to reality. I see that there isn't anyone who participated in that effort whose contribution did not matter.

I want to see collegiality in the way SE4E is introduced, especially in schools.  Here are the central elements: seeking review and inspection from others, learning from the best contributions already available, and then giving it all back.

Human activities of any scale are cooperative activities. As a young software developer, I had this conceit that I could do it all myself, relying on my innate creativity, and if those other jerks would get out of my way it would all be perfect. I realize that is hogwash, yet I'm still caught wasting time before consulting someone else for assistance and to be my sounding board for ideas I am struggling with.  

In my first dedicated programming job (as a "Clerk Typist A" since they hadn't invented student programmer positions at that university yet), the faculty member I worked for was creating a handbook of software.  He had a grant for collecting programs that were available in his field and republishing them with the documentation needed for making them useful. We were also cleaning up the implementations so that the programs and routines could be used together as part of a coherent body of work. I loved it.  I was also  constantly frustrated in my struggles to write intelligible documentation.  I learned what I have heard repeatedly since: the way to learn to write is to start writing.  And that includes sharing ones writings as imperfect work in-progress.

Nowadays I incorporate documentation as an inseparable part of the design of programs.  Assuring that a program is explainable is one of my tests for conceptual economy of the program itself.  Even when I am not building software for anyone else to use, I preserve the hard-won habit of documenting what I am doing as if it is intended for others to be able to use without having written it themselves.  Truthfully, I don't ever think otherwise, because that someone else is often my forgetful self at a later time.  

It is important for neophytes to understand that work doesn't have to be perfect before it can be shared with others.  And the constructive observations of others will provide focus on essentials that are easily overlooked by someone immersed in a project.  Explaining a program design to someone else provides insights into what I am doing that I wouldn't have had, even when the person I am having a walkthrough with doesn't say a word.  I don't know why that is.  It works so often that I simply trust in it.  

My wife is a potter, not a programmer.  I discussed collegiality with her and she pointed out an important experience for her: mentoring another revealed to her what she had mastered in her own craft.  This is part of collegiality: recognizing how much we learn from others and being able to support the development of newcomers as part of our accomplishment.

Programming skills are mostly learned by first working from existing programs and samples. It becomes important to have good examples. At some point, the practice of refinement and adaptation begins to show qualities of craftsmanship, and our programs can be increasingly diverse from one to the next. 

Progressive refinement toward mastery of programming as a problem-solving skill is not a solitary activity, no matter how much we go through it individually. For it to work, we must be willing to submit our work to the adaptation and refinement of others. Sooner rather than later. When I first met Donald Knuth, he spoke about some of the most beautifully-crafted programs he had ever read and that inspired work he would later be renowned for.  Early, hand-crafted software became the inspirations for our generation of programmers, and it was possible because the early programmers made their work available to read and adapt. 

In my first moonlight hacking project, three student programmers modified a piece of open-source software. It was the SOAP II Assembler for the IBM Model 650 Computer System.  We were privileged that the source code and documentation for the assembler were printed in the back of the manual. By inspecting the code and finding ways to tighten it here and there, we eked out enough free memory to add the goodies we wanted.  I later met that software's author, a structural engineer by training and interest.  He acknowledged that it was quite a struggle to have the program published as part of the documentation.  I am grateful that he persisted and prevailed.

In the context of SE4E, the simple practice of sharing ones work, learning to inspect the work of others, and having our own work checked is essential. The open-source approach is also valuable here, as well as being today's great source of worked examples.  Seeing cooperative activities as a desirable, let alone permissible way to produce an effective result is an important experience to be exposed to when young. As is the practice of being accountable for what is original and what isn't, acknowledging the support obtained from earlier work. These are practices that are important to demonstrate in a practical, simple way.

4. Predictability: Make It So, Keep it So

It is something to reach the point where one can actually specify something and design it without having ever built it first. It takes practice and experimentation and some sort of conceptual shift after which design starts being a reliable activity. It's a big leap from the first days of having no idea what line of code to write first.

The first principle of predictability is divide-and-conquer. Creating interface agreements and sharing modules in a larger project or team is important to allowing parallel activity while assuring that the different paths will converge without surprise. Having stable interfaces around which everything else may change independently is a big technical insight.  Then one can learn that some interfaces work better than others, and begin to see why.

For activities involving many participants, even if each has a well-defined separate component to work on, there must be a way to keep an eye on the overall activity and to choreograph how separate results are brought together into the final result. For this, I have used different project management tools, and consulted a variety of books on project management.  The simplest are the best and easily accessible for introduction with SE4E.

These days I recommend Project Management: From Idea to Implementation [Project].  It isn't about software development at all, it is about projects of any kind.  The treatment is straightforward and the separate video provides a delightful taste of how projects are accomplished in the worlds of work or school.

I would look for examples in "real life," like getting homework or papers done on time, including working from outlines first, something that I remember as a challenge when I was a high-school student. Or managing ones training for a sporting event or team activity. Or producing a community-service project, putting together the school play, organizing an art exhibition, or managing an athletic tournament. 

Predictability in software development goes beyond this. It has to do with applying quantitative methods. It has to do with refinement of requirements into suitable design and then construction of an integrated result. It is certainly tied to accountability and collegiality. It adds being able to predict effort and the time it will take to have a completed result. 

In models for levels of software-engineering capability in organizations, predictability is the first rung of the ladder to superior software-delivery power.  Predictability is not easily achieved, nor is it automatically sustainable.  It also can't be obtained without also mastering the content of the work.  It is important for SE4E that the value of predictability be understood, that there be a sense of how it can be made possible, but it not be put in the way of early personal programming experience.

Predictability has been the stickiest part of my becoming an accomplished developer and software architect.  The key problem is that you have to fail, repeatedly, in order to learn how to predict effort and results.  It is not something that we already know.  So the first results we obtain will demonstrate that lack.  In SE4E, it is of immeasurable value to appreciate how incompetent one must be willing to be in order to build the experience on which predictability is based.  This fundamental of predictability is the area where I am constantly practicing and still have at best an erratic pattern of results.

Most software lives through many cycles of alteration and refinement.  Preserving predictability across change is almost always important.  Again, open-source projects provide visible activities where one can observe the actual process of managing predictability in a dispersed, multiple-participant community.  How different aspects of predictability are preserved (or not) is worked out in plain sight.  This is a valuable model for the level of informality that fits a classroom, too.

I recommend small-group projects in open-source style for revealing all of the ways that predictability is important.  Not much is needed by way of tools.  The first level of skill is more about adopting reliable practices and preserving communication among the participants.  For individuals wanting to practice farther, I can't think of any better activity than participation in one of the open-source software projects being carried out around the world.  Projects designed to produce instruments for use in simple software-engineering activities and in CP4E itself will be accessible, especially those focused on providing introductory tools and information.

5. Practice, Practice, Practice: Never Stop

        [Wrap it up here.]

6. References and Resources

Computer Programming for Everyone. Complete this reference to the CP4E space reachable on, with the project description and links.
Complete reference to and key to accessing this discussion list.
Raymond, Eric SHow to Become A Hacker.  Published on the Web as a Frequently-Asked-Questions file.  Updated periodically.
Haynes, Marion E.  Project Management: From Idea to Implementation,  ed.2.  A Fifty-Minute Book. Crisp Publications (Menlo Park, CA: 1989, 1998).  ISBN 1-58052-418-9.
Humphrey, Watts.  Introduction to Personal Software Productivity.  Find and include the Addison-Wesley Reference.  It will be on fatbrain or
Figgins, Stephen.  Hackers and Trackers: CP4E.  On-line article.  O'Reilly Network.  January 31, 2000.
Vixie, Paul.  Software Engineering.  In Open Sources: Voices from the Open Source Revolution. O'Reilly (Sebastopol, CA: 1999).  ISBN 1-56592-582-3.
Ward, Greg.  Re: Software Engineering for everyone?  Posted to Edu-SIG, March 27, 2000.  Create link to the archive where this note lives.
Find reference to "The Psychology of Computer Programming."  Or remove this if there isn't any need for it.


This article arose from discussions on [Edu-SIG].  Greg Ward took a position that had me notice my own experience in gradually recognizing the value of software engineering.  I  responded on March 27, 2000.  Create link to the archive of that too.

Change History

draft 0.05 2000-05-26 by orcmid
Work over section the predictability section and use SE4E as more than the title.  Another thing to do: point out that if one is to teach SE4E, the teacher must practice SE4E and be willing to share that, even rely on the students for mutual support.  Eek!  I'm now up to 3298 words.  Post as replacement editorial checkpoint with serious cutting in a new version under a different file name.
draft 0.04 2000-05-25 by orcmid
Review, complete and double-check correct citations to some reference material.  Reworked first two sections.  Struggle with the collegiality section to make it concise and also leave people with sense of what becomes available.  The new word count is 2947, including overhead such as this section.  Two more things yet to do: Summarize the key ideas in sidebars with each section, then boil each section down to directly support the sidebar.  Post as an editorial checkpoint.
draft 0.03 2000-05-03 it (+0200) by orcmid.
Smooth the coverage in each topic and begin pruning to the intended size. Get the key points and suggestions made more authoritatively and toss illustrations that don't get right to an important point.  Submit to editorial checkpoint.  Remember to run spelling checker first.  There are 3191 words in this version.  I want around 1200.
draft 0.02 2000-05-02 ist (+0200) by orcmid.
Developed basic context and theme.  Pasted in and organized the original e-mail notes and clippings that provided the initial thinking on this topic..
draft 0.01 2000-04-05 pdt (-0700) by dh
Initial draft set up as web page using an existing document for boilerplate.

$$Author: Orcmid $
$$Date: 02-10-13 15:26 $
$$Revision: 5 $

End of Document