Orcmid's Lair status 


Open-Standards Implementation, I: Celebrating Massachusetts

[This is the first of a likely three-part series on Open-Standards Implementation featuring the progression that is taking place in Massachusetts.  As usual, there is always more to look at and comprehend when I sit down to write what I think will be a straightforward analysis.] 

I have strong sympathies for the use of public, open formats for documents in civil administration, governmental records, and in the preservation of our electronic and pre-electronic heritage.  I absolutely favor any efforts that remove impedance mismatches from the interactions of citizens with [e]Government in all of its flavors.

I have some concerns about the extent to which offers of solutions are surrounded by too many claims of hypothetical benefits (which always look superior to the practical reality) and by poorly-disguised attempts to distort markets with a seasoning of appealing snake oil: asserted distopian consequences from all protagonists and very untested utopian-salvation promises. I am also pained by the hasty and thoughtless efforts to legislate technology that seems to be the popular substitute for the deliberate department-initiated efforts in Massachusetts.  I think legislation of technology, and the double-speak that accompanies it (also from all protagonists) is misguided; most of all, I am concerned that the citizens and their civil agencies and public institutions are going to suffer for it.

Experimentation and trial of alternatives is going to be important in finding successful approaches.  Some of the experiments will point to important course correction; most of all, they will add reality to our understanding of what it takes. 

The poster child for adoption of open-document formats in the United States is the Commonwealth of Massachusetts Information Technology Division (ITD), part of the Executive Office for Administration and Finance in the Commonwealth.  I think it is time to pay homage to the efforts that were undertaken by ITD and the Commonwealth CIO in establishing its Open Initiatives.  My goal is to recognize the practical, deliberate efforts in the ITD and the pragmatic realism that is displayed in the public documents on the ITD web site.  This is in marked contrast to the way these efforts are often portrayed.  I will indicate where I have some concerns; my overall purpose is to recognize the accomplishment and how it is being managed in a professional and highly-responsible manner.  It is also a noisy human effort; that is not an indictment.

Open-Initiative Policies: The Stake in the Ground

The current Open Initiatives page has this plain-language statement:

“The goal of the Commonwealth's open initiatives is to ensure that investments in information technology result in systems that are sufficiently interoperable to meet the business requirements of its agencies and to effectively serve its constituencies.”

It is important to understand that this is a commitment around the information-technology operations of an executive branch of state government: the policy is focused on technology used primarily in the internal operations of governmental administration.  This is consistent with the ITD department’s accountability.  The policies can certainly impact inter-governmental operations and coordination with vendors and the public in various collaborative exchanges, despite possibly-disingenuous claims to the contrary [3].  But the focus is on efficiency and delivery of services to agency constituencies.  This statement has its origin in the current Open Standards Policy, effective since January 13, 2004:

“The Commonwealth must ensure that its investments in information technology result in systems that are sufficiently interoperable to meet the business requirements of its agencies and to effectively serve its constituencies.  This policy addresses the importance of open standards compliance for IT investments in the Commonwealth.”

The policy statement goes on to define what is meant by an open standard:

“Open Standards: Specifications for systems that are publicly available and are developed by an open community and affirmed by a standards body. Hypertext Markup Language (HTML) is an example of an open standard. Open standards imply that multiple vendors can compete directly based on the features and performance of their products. It also implies that the existing information technology solution is portable and that it can be removed and replaced with that of another vendor with minimal effort and without major interruption (see current version of the Enterprise Technical Reference Model).”

It is useful to notice a number of features of this statement:

  • In January 2004, and certainly for the period of time when this statement was under development and review, there was no approved OASIS OpenDocument (ODF) specification although there might have been anticipation of such an event [1]. 
  • The illustrative choice of HTML is interesting for several reasons:
    • The W3C is not a standards body in the usual sense.  Nevertheless, its efforts as an international consortium have been extremely effective in the creation of widely-adopted specifications.  These are standards by virtue of their voluntary acceptance and widespread implementation.
    • The latest official HTML specification, that for HTML 4.01, was approved on 1999–12–24.    There is a certain continuity and maturity reflected in the progression to this level since 1995 [2].  HTML is a matured format, and it is still subject to development and implementation difficulties (as well as debates over what is “standard” behavior and what is not).
    • It is awkward to consider HTML as a basis for implying that “multiple vendors can compete directly based on the features and performance of their products.”  There is something to it, but it is not so clear exactly what the competition is among and what the strength of the implication is.  There is a complex relationship to the role of HTTP and the TCP/IP protocol stack (that is, protocols as well as formats all the way up from the remote application/service to the interchange of data units with local application software).  Are we talking about competition among browsers and if so, how does that impact ITD operations?  Or are we talking about web sites operated by ITD and how the de facto standardization on HTML, among other things, supports that?
      I think the awkwardness is in generalization to an a priori conclusion that the inference holds in some assured way.  There is a bone-yard of dead and still-born standards out there, some of them adopted as procurement requirements by government units at one time or another.  And, whether or not multiple vendors can compete, do they compete?  Are there meaningful substitutes and alternatives in what ITD finds itself doing with HTML?
  • I am suspicious of the implicit generalization that one can count on existence of an open standard as assurance that an “information technology solution is portable and that it can be removed and replaced … with minimal effort and without major interruption.”  These out-of-context generalizations are probably necessary in communicating the policy, but they leave a great deal under the rug.  I don’t know that there is any serious over-reaching her;  it probably attracts my attention because I have committed this particular transgression often enough myself.

There are great round words in the policy; they speak to the attributes sought in the adoption of technologies for the purpose of the ITD Open Initiative.  Still, I am wary.  The speculative notion that the mere existence of open standards implies some sort of sure thing need to be regarded critically and with healthy skepticism.  I also think it is necessary to consider more carefully how much the context of usage and local practices will impact conditions of “portability” as the term is used here.  The notion that affordable substitution becomes automatically available (with affordability including all costs of change-over, not just the acquisition of any products involved) is also something that takes more diligence than the easy inference recognizes.

I am concerned about over-reaching and over-justifying.  I don’t want to see unwelcome surprises for those paying the bills and that might besmirch the laudable objectives of the initiative.  I also concede that I am nit-picking and there is now actual experience in Massachusetts that is a better gauge on how progress is being made along these not-so-clearly-lit avenues.  Going forward, it is constructive to examine how these policies are made actionable and how the benefits are secured in a form that can be measured and compared.

With those caveats in mind, let’s follow along with the foundation that ITD established.

Open-Standards Policy: The Line in the Sand?

These are the teeth in the 2004 Open-Standards Policy:


“Agencies within the Executive Department and vendors providing information technology goods and services to these agencies must comply with this policy.


“Policy Statement

  • “All prospective IT investments will comply with open standards referenced in the current version of the Enterprise Technology Reference Model.
  • “Existing IT systems will be reviewed for open standards compatibility and will be enhanced to achieve open standards compatibility where appropriate. Open standards solutions will be selected when existing systems are to be retired or need major enhancements.”

The enactment is a progressive one that is established in an evolving “Enterprise Technology Reference Model.”  It is very revealing to examine that model and its progressive refinements.

Affirming the Focus

There is one more component to the Open-Standards Policy that deserves extra scrutiny [with my bracketed comments]:

Commonwealth's Position

  • “Effective and efficient government service delivery requires system integration and data sharing.
    [This is a conventional motherhood statement.  It makes sense and we can easily nod our heads over it.  There are some practical factors about how this is evolved over time, and whether that requires consolidated location or not.  There are costs as well as benefits, and we should look to see how their reconciliation is managed over time.]
  • “Technology investments must be made based on total cost of ownership and best value to the Commonwealth. Component-based software development based on open standards allows for a more cost-effective "build once, use many times" approach.
    [Here we find a prescription followed by a position on the benefits of component-based software development.  This most-clearly applies to internal application-software development and integration by ITD and its suppliers.  It is not entirely clear how open standards match up with this case, although the Technology Reference Model places great emphasis on Web Services standards, including the WS-I (Interoperability) harmonizations.  Although one might think office-productivity software is of negligible presence here, there is use of office-document software as instruments and tools in work processes that may be expected to support component-based integration for smooth operation.  Something to look for is an evolved, in-place measurement process that accounts for how well an investment in reusable components is returned in terms of actual benefits attributable to reuse.  Expect inclusion of recognition of direct software-engineering benefits of componentization as well as costs of standardization independent of any reuse.]
  • “Open systems and specifications are often less costly to acquire, develop and maintain and do not result in vendor lock-in.”
    [This is the most speculative and also the most difficult to parse.  There are a great number of preconditions on this position being realized, along with its satisfying appeal to an ill-defined and questionably-absolute assurance about (vendor) lock-in.  I think it is a weakness here to fail to consider that there is also a standardization tax and a non-lock-in avoidance/portability-assurance tax as well.  I am not clear how much ITD is involved in the acquisition, development, and maintenance of open specifications of any kind.  Going forward, look for some same-eggs-counted measurement of the benefits to acquisition, development, and maintenance.
    I thought this was applicable to open-document format adoption, but I probably read to much into the reference to lock-in.  This position may be far more about open-system building where open protocols and formats have great bearing and office-application software document formats have a relatively-minor rôle.]

Mainly, this is a policy created by an IT organization for how it sees itself as satisfying its mission by reliance on open standards.   The statement must satisfy the department’s patrons as well.  The position tells us about organizational sensibilities and expresses some advocacy from the IT perspective.  Measures and audits would be great for calibrating and validating these positions over time.  

Now we can follow along to see how these policies are applied and what the experiences and refinements are.

Risk Management and IT Policies

We have been slowly learning that one of the greatest risks in Information Technology endeavors is failure to manage and mitigate risk properly.  We see evidence of it in many ways, including the remarkable failure rate of IT projects that could have been useful if they’d failed much sooner.  Fail often and sooner is, in one perspective, the mark of an aggressive but healthy innovative organization.  That is sometimes a luxury, especially for undertakings with accountability to the public.  Shareholders rarely get much insight into the dazzling IT failures in commercial firms, but the employees can tell you.

I am not suggesting that this policy initiative is any sort of IT train-wreck.  On the contrary.  What I am suggesting is that the more abstract and speculative a policy-motivated approach is, the more aggressively one must look to risk management practices to see how practical safeguards and corrections are arrived at in the face of increasing experience.  In the case of a public agency, it is also a responsible way to account to its constituency.

So when I write about what I’m looking for, or what one would expect to find, it is from the perspective of being able to understand how risk has been managed and how the policy-set course is being navigated in a grounded, measurable way.   It is not January 2004, so it is easier to say this from the hindsight of April 2007.  In fact, at this point I only have provisional impressions gained as an outsider after cursory examination of public materials.   I will look more closely, and I will be careful to qualify my observations.

Coming In Part II: The ODF Fandango

[1] The OASIS Open Document TC was initiated on 2002–11–04.  The OASIS OpenDocument v1.0 Specification was approved on 2005–05–01 with a Second Edition approved on 2006–07–19.  ISO/IEC 26300:2006 the Open Document Format for Office Applications (OpenDocument v1.0) was available as an ISO Published Standard on 2006–11–30.  The OpenDocument v1.1 Specification was approved on 2007–02–02 and we are all anxious to know what the defined format for spreadsheet formulas will be and whether it appears in the now-in-progress v1.2.  We will then get to see how migration is accomplished from existing ODF-based spreadsheet software approaches that were taken in the absence of definitive specification in ODF v1.0–1.1.

[2] XHTML 1.0, the reformulation of HTML 4 into XML (although HTML 4 is not a subset), was approved on 2000–01–26 and revised on 2002–08–01.  The popular HTML 4 predecessor specification, HTML 3.2, was approved on 1997–01–14.  The early HTML 2.0 is described in the November, 1995, never-standardized IETF RFC 1866 (obsoleted by the Informational IETF RFC 2854 in June, 2000).  HTML 2.0 is described as an application of SGML, ISO 8879:1986.  SGML is based on the Generalized Markup Language (GML) devised by Charles Goldfarb at IBM in the preceding decade.  For whatever reasons, SGML has not seen the widespread awareness and adoption that its descendants HTML and XML have enjoyed. 

[3] In responding to issues raised by legislators in a review of its OpenDocument Format mandate, ITD created the 2005–09–21 “Final ETRM Version 3.5 Open Document Format Standard: Frequently Asked Questions” material.  We will see that this is also a very useful document for tracking the initiative.  In spots, the responses seemed a little defensive: less than forthright, including a few ingenuous claims such as “the Final ETRM applies only to documents that agencies create and save, not to documents they receive from third parties” and “The Open Document Format has been in use for several years … .”  Recall that ODF v1.0 had been approved for less than five months at that time, and the first and most-complete open-source implementation, OpenOffice.org 2.0 with default ODF support, was still in beta.  I see a few other howlers like those, although, overall, this is a very useful and responsive document.

You might find it interesting to do some research on how ITD was doing before they put together this initiative. It certainly was not a case of waking up one morning and saying, "Everything is fine, let's change!". They had some serious pain they were dealing with and the ETRM was their attempt to remove themselves from that pain. This is the type of decisions that private companies make every day. But when you are a public agency, this comes with public scrutiny.

(Also, a side point, didn't ISO HTML already exist at this point in time? So whether ones thinks that the W3C is a traditional standards body or not is moot.)
Rob: I can well believe that the Initiative came out of a lengthy struggle in ITD, and the ETRM is a big deal. (There's something similar at the Federal level, and my initial impression is that the ETRM is superior.) Thanks for emphasizing that.

Concerning ISO HTML, you're right. Who knew? There was ISO 15545:2000 and it is evidently a strict subset of HTML 4 (strict). I don't believe that many knew that or that anybody cared at the US Domestic level, when the W3C specs are what people use just fine and there's a lot more than ISO HTML being used.

I love the way the W3C works and what they've accomplished, second only to IETF. OASIS is a close third for me.

The bigger surprise to me is that there does not seem to be any ISO specification for XML, although there are several ISO Standards related to XML. Did I miss it?
SGML is an ISO standard, and XML is a specialization of SGML. Sending XML to ISO was discussed back around 2000, but nothing came of it.
So, I guess XML was contradictory with SGML? [I know, I couldn't resist. It is funny how one-sided these rationalizations are, though.]
Certainly C++/CLI was defeated in ISO for extending C++ in a way that allegedly caused confusion for users of C++. But no one alleged the same for XML to users of SGML. Part of this may also be due to the widespread vendor acceptance and promotion of XML, versus Microsoft's attempt to "go it alone" with C++/CLI. This clearly can make a difference.
I'm grateful that C++/CLI exists (it makes native implementation of .NET classes and methods much tidier than with JNI for Java).

How it is handled in VC++ 2005 is a nightmare for beginners, and really difficult to explain at that level, when the novice doesn't understand what the platform is, let alone distinguishing from the 3-4 that may be targetted from VC++.

That's not about the standardization effort though, although I assume it was a big lesson for Microsoft.

And, for those playing along at home, lets come back to what I find amusing about all of this. It is (1) that it doesn't matter that W3C is not a conventional standards body because there is an ISO HTML that is at best some meaningless check-off item contrasted with (2) it doesn't matter that there is no ISO XML because of the useless to most humans fact that it has a relationship to ISO SGML. (The relationship of HTML to SGML is even closer, wouldn't you say.)

I think we are reasoning backwards from the desired conclusion here, and the results are bizarre. I will watch myself on this score and I hope you and other keen-eyed folk will point it out when I slip down that slope.
What I find the most interesting in this narrative is that the pain and difficult experiences that led to adopting an open document format policy is basically hidden. It's held in the minds of those who went through it. And, as you point out, the pain and work practice adjustments that need to be made to adopt new document technologies (formats and the applications that let us use them) are never mentioned when talking about the future. But this is the kind of information that would be most helpful to others who will need to take this road (as well as illuminating to historians of technology and innovation). We desperately need to invent and deploy technologies and tools that give us easy document sharing and interchange, as well as permanent access to these document based records.

We're going to have to be honest with ourselves about the costs of technology innovation and stabilization. We know the costs when we undertake other infrastructure projects -- just think roadway or light rail construction. We can do it with technology. So let's hope that Massachusetts can be an example of how to keep notes.

Way to go, orcmid. I can't wait for parts 2 and 3.
Post a Comment
Construction Zone (Hard Hat Area) You are navigating Orcmid's Lair.

template created 2002-10-28-07:25 -0800 (pst) by orcmid
$$Author: Orcmid $
$$Date: 07-02-17 11:08 $
$$Revision: 26 $