Welcome to Orcmid's Lair, the playground for family connections, pastimes, and scholarly vocation -- the collected professional and recreational work of Dennis E. Hamilton
I was startled to see the level of passion in Alex Brown’s 2010-03-31 post, Microsoft Fails the Standards Test. Alex has two concerns: (1) dwindling OOXML standards-maintenance attention and resources; and (2) Microsoft silence with regard to implemented support for the strict level of IS 29500 and any retirement of the transitional level as the only level supported in Microsoft implementations of OOXML.
For me, the most peculiar aspect of the reactions I see is not that Alex has the concerns he announced, but that others treat his expression of concern for the future as a declaration of the actual present. Furthermore, these observers who proudly pontificate that there is no action, there will be no action, and there was never going to be any action, excitedly congratulate Alex on having awakened from being hood-winked. It is as if nothing has happened since 1998 and the book is closed on Microsoft forever.
Expectations Against Observable Reality
I want to look at just one part of this situation: expectations around IS 29500 implementation in Microsoft products. The desire to have Microsoft abandon to-be-deprecated transitional provisions of IS 29500 in favor of producing only documents in the strict IS 29500 format is tied into that expectation.
I am eminently qualified to address this topic. I have no information on what Microsoft is actually doing to incorporate support for IS 29500 in its products. I have no idea what strategy Microsoft has, if any, with regard to the retirement of support for IS 29500 compliant transitional documents. Microsoft doesn’t tell me anything about product efforts and I am happy to keep it that way. Microsoft doesn’t seek my advice on the matter either. So I am perfectly positioned to speculate, with my standing as a standards, interoperability, and architectural armchair astronaut unblemished.
What I am going to report is my observation of the simple state of affairs and how difficult it is to erase the past and jump to implementations that only produce what are called strict IS 29500 documents. Expecting that to have been achieved in two years is about as unobservant as belief that all Microsoft needed to have done was adopt ODF as its native format in the first place.
Can I Has Have Me Strict Now Please?
The ISO/IEC International Standard for OOXML, IS 29500:2008, has two major levels that can be implemented. There is a strict IS 29500 format that is the subject of the main part of the specification. There is also a transitional IS 29500 format that mainly includes everything allowed in the strict format along with some other provisions that it was agreed would be retired but retained now for compatibility purposes. The idea was that there would become a time when consumers might accept transitional documents but the routinely-expected output would be a strict IS 29500 document. The strict-transitional differentiation and the notion that production of non-strict documents would be discouraged for new documents was a creation of the DIS 29500 Ballot Resolution meeting in February 2008. It was ratified in the approval of DIS 29500 as an International Standard on April 2, 2008.
In IS 29500:2008 as it was first published in November, 2008, the set of strict documents is essentially a subset of the transitional documents. In addition, it is the transitional documents that include the most provisions of ECMA-376:2006, the specification supported by the already-distributed Microsoft Office System 2007 (and any Office 2003 configuration, like mine, to which the OOXML Compatibility Pack has been added). Various translators also accept or emit ECMA-376 documents. The .docx, .xlsx, and .pptx documents that are growing in numbers every day in production use satisfy the provisions of ECMA-376:2006 and tend to use transitional provisions of IS 29500 rather than preferable strict counterparts.
There is already a legacy situation with OOXML concerning the need for products to support ECMA-376:2006 in the documents that are accepted and produced.
Well, Not So Fast, Sparky
In addition, SC34 WG4, the standards-development working group that maintains IS 29500, has now created a situation in which there is more than one strict IS 29500.
As part of the early maintenance work, some found it disturbing that there was no way to differentiate provisions of ECMA-376:2006 from IS 29500 transitional and of IS 29500 transitional from IS 29500 strict. The same namespaces were used for all of them. This issue was being addressed before the ink was dry on IS 29500:2008. In 2009, the SC34 WG4 arrived at a set of amendments, most of which were designed to separate the namespace used for the strict provisions of IS 29500 from the namespace used for transitional documents with their additional/alternative too-be-deprecated provisions.
If Microsoft had already implemented a way to produce strict IS 29500:2008 documents as defined before these amendments, those documents would now be considered transitional documents. Their XML parts would employ the transitional namespaces, not the recently-adopted ones for strict IS 29500.
It has been a few months since the amendment solidified, and one would hope that Microsoft is looking at how to enact the production of strict IS 29500 documents in some customer-respectful manner. Whether that is something that could possibly appear in the soon-to-be-released Microsoft Office 2010 products is not something I will even guess about. Microsoft’s commitment to support IS 29500 is not specific on this topic, and there may be residual difficulties in how the separation of strict out from under transitional has been executed in the amendments and any anticipatory implementation work.
I can see ways to work through a gradual migration where the expected output is strict IS 29500 documents. But I would expect transitional/ECMA-676 documents to be accepted for a long time and to be produced at least as long as “Save As … 97-2003 Document” also exists.
Appendix: Arriving at Separated Strict and Transitional Namespaces: The Progression
Although it has been two years since there was agreement on what IS 29500:2008 would be, it was not until November 25 that the specifications were publicly available. In less than one year after that, amendments creating a substantial difference in the separation of strict OOXML documents from transitional OOXML documents was formulated and put out for ballot. The official Corrigenda and Addenda carrying those and other changes are not yet available to the public. Here is the progression.
ISO/IEC International Standard 29500:2008 Office Open XML File Formats. First edition 2008-11-15. This is the first official publication of IS 29500 for OOXML after the 2007 balloting and a subsequent Ballot Resolution Meeting in early 2008. The specification is in four parts:
Part 1: Fundamentals and Markup Language Reference. This is where the strict provisions are specified.
Part 2: Open Packaging Conventions. There are some transitional considerations in packaging. These are being treated by separate defect reports. The OPC are adaptable for non-OOXML usage.
Part 3: Markup Compatibility and Extensibility. A set of independent features by which extensions can be added to an XML Document using a set of specific conventions that allow for graceful degradation when the extensions are not understood.
Part 4: Transitional Migration Features. Additional features that are not included in the strict provisions. As originally formulated, the strict provisions were to be a subset of the transitional OOXML documents and all of ECMA-376 was embraced by transitional OOXML with a few deviations.
SC34 N 1246 ISO/IEC 29500-1:2008/FPDAM1, Part 1: Fundamentals and Markup Language Reference – AMENDMENT 1, 2009-08-04 available as a public 1.56MB downloadable Zip file consisting of the proposed draft amendment and the corrected (RNG and W3C) schemas.
This public document was taken to a four-month SC34 FPDAM Ballot that closed 2009-12-04.
SC34 N 1251 ISO/IEC 29500-4:2008/FPDAM1, Part 4: Transitional Features – AMENDMENT 1, 2009-08-04 available as a public 1.59MB downloadable Zip file consisting of the proposed draft amendment and the corrected (RNG and W3C) schemas.
This public document was taken to a four-month SC34 FPDAM Ballot that closed 2009-12-04.
SC34 N 1253 IS 2950099:2008 Defect Report Log [At Closure of the DCOR1 and FDAM1 Sets], 2009-08-04 edition available as a public 3.91MB downloadable PDF file.
Defect Item 08-0012 was submitted by Ms. Ruth Schneider of the National Body (SNV) for Switzerland. It was circulated to SC34 on 2008-11-03. The defect involves the inability to distinguish between ECMA-376:2006 schemas and those of IS 29500 because the same namespaces are used. From the log on this item we can see that there were many ways to deal with explicit versioning and that no reasonable solution stood out. Finally, at a meeting in Prague on 2009-03-24, a small break-out group returned with proposed requirements for any versioning solution:
To push producers gently towards strict conformance [emphasis mine, orcmid:]
To improve interoperability
To do no evil to the 29500 ecosystem (i.e., files, end users, and implementations)
The option to change the namespace for strict schemas and only strict schemas emerged in subsequent discussions. It took until the 2009-06-22/24 meeting in Copenhagen to finish wading through the considerations and agree that the only change would be new namespaces for strict schemas and the associated narratives. Along the way there were was stumbling around versioning, conformance attributes, and features of transitional that had no existence in ECMA-376:2006 and were there now merely to preserve transitional as the superset. Although the final changes needed to accomplish thisthe namespace separations were contributed by Microsoft expert Shawn Villaron of the ECMA delegation (and Microsoft), the issues and inter-dependencies were heavily discussed among all of the WG4 participants.
In the end, resolution of defect item 08-0012 involved over 300amendment entries in N1246 for IS 29500 Part 1 and over 200 in N1251 for Part 4. This is about 75% of all of the amendment entries in the FPDAM1 set, all on behalf of this single defect and its simply-stated disposition.
Brad Smith’s 2009-12-16 Microsoft Statement on European Commission Decision. This declaration from Microsoft establishes a “public undertaking” with regard to interoperability. Part of this undertaking includes a Warranty Agreement covered in a 2009-12-16 Annex (downloadable Microsoft Office Word .doc file). The timing of this announcement has nothing to do with events at ISO/IEC JTC1 SC34 (and some of the “undertaking” documents were first uploaded with 2009-10-06, dates). The “undertaking” is in the spirit of the Interoperability Principles and other agreements with regulatory authorities (c.f., my 2008-02-26 Interoperability by Design post). It also puts some serious teeth into Microsoft commitments by asserting that “Microsoft will make available legally-binding warranties that will be offered to third parties.”
In my reading of the available on-line documents, legally-binding warranties are provided for an unstated fee to parties that intend to provide interoperability with Microsoft implementations of various protocols and industry standards. Under such warranties, there is assurance that successors to Microsoft Office Word 2007, Excel 2007, and PowerPoint 2007 will support IS 29500 rather than ECMA-376. Nothing in the “undertaking” distinguishes between strict and transitional provisions of IS 29500.
Subsequent Creation of Documents. There are no more-recent public documents, as of 2010-04-03. On 2010-03-05, documents SC34 N 1382 and SC34 N 1384 appeared as final FDAM1 amendment texts on the SC34 document repository. I presume that these reflect the disposition of FPDAM1 ballot comments and perhaps other feedback on N 1246 and N 1251, respectively.
New Defects from the Namespace Change. I can’t imagine that the massive changes involved in separation of strict into its own namespaces would not be accompanied by new defects, especially inconsidering how the dependent the transitional provisions were on the formerly-strict-but-same-namespace provisions. This seems to be borne out in the Outstanding Action Items 7-9 of the 2010-02-18 SC34 WG6 Teleconference minutes (available as a public 574kB downloadable PDF file). Although there will need to be those, perhaps other, repairs, there is probably little that would further delay an implementation being able to recognize its supported strict features under either namespace so long as the provisions are identical. Preventing the clashing of unique-to-strict and unique-to-transitional provisions will require care in all cases. I am blissfully ignorant of how the mixing of features might complicate the internal, “in-memory” document model of Microsoft and other existing products. The added complexity of testing and building a relevant document corpus for all of the use cases strikes me as seriously daunting.
[update 2010-04-05T23:41Z And I am finally scrapping all of that useless white space that was on the end of my blog text. update 2010-04-05T23:11Z I misquoted Andy Updegrove’s blog title quoting Alex Brown and have fixed it, thanks to the good eye of Rob Weir. update 2010-04-05T16:49Z Too be more careful when there are twoo many words to get right the first time. Sometimes, I just can’t bear it. update 2010-04-05T15:11Z I took inspiration from Alex Brown’s comment to tidy up some wording and add an after-thought about the unseen but material impacts of a simple namespace change in a world of comingled strict and transitional features.]
Thank you for both this refreshingly sane assessment of my piece, and for providing some important technical detail.
Can expectations trump reality? In this particular case they have, since the National Bodies standardising OOXML made it plain that they weren't content to deal only in reality by creating OOXML Strict in the first place. This may not have been realistic (or even wise) of them but - it's what happened.
> Expecting that to have been achieved in two years is about as unobservant > as belief that all Microsoft needed to have done was adopt ODF as its native > format in the first place.
I think what is problematic here is the magic figure of two years. What is magic about it? It was the pre-decided release period for Microsoft Office 2010. I've heard varying assessments of the difficulty of producing Strict support in this time, and I think we can safely assume that support for Strict would have been expensive, most likely entailing slipping the release date of Office 2010. However I think it's an important principle that we don't allow a consideration of any one vendor's business plans to backwash into our assessment their product's standards support. VML is marked in the standard as a technology for legacy documents only, and there is no rider that this needn't apply if it impacts revenue by more than $1bn that fiscal year!
So I think it's almost a moral question: the moment we allow ourselves to be anything other than merciless in our holding to account of MS Office, then we are failing to be impartial referees on the level playing field OOXML was intended to create.
I certainly agree that future OOXML supporting office suites will need to support the legacy ("Transitional" or "T") formats in addition to using the Strict format for new documents. The problem at the moment is that Office 2010 appears to have zero conception of Strict documents. Your point about the Namespace change is an important one, and if MS had targetted Strict they would need to target Strict as it appeared in in the published IS, not the upcoming amendment (though since the Namespace change is purely a renaming, this could be anticipated and dealt with systematically in the deserialisation code).
I have little sympathy with with the fact that Microsoft have to track a moving target here: the Namespace change (like many others) stems from the fact that the initial version of the standard was so poorly considered. Besides, MS are quite happy to track a moving standard where it suits them - as you may know, a few dozen amendments/corrections were made to the IS to bring it into line with the Office 2007/2010 code base during the last year. They were accepted on the basis that the Transitional format needs to reflect the "existing corpus of legacy documents". Thus in some senses T (important as it is) is an anti-standard; it is product documentation for MS Office; it can change suddenly in arbitrary ways according to what is discovered about that existing corpus (read: Microsoft's code base). If we sanction the use of T for new documents we are skating perilously close to conspiring with Microsoft to subvert the standards process and distort the market against people trying to compete with MS on the basis of IS 29500.
Dennis, you've misquoted Alex. You are saying, "Without action, the entire OOXML project is surely heading for failure" and then saying that people are confusing present with a future possibility.
But Alex actually wrote, "Without action, the entire OOXML project is now surely heading for failure". (my emphasis). Using the word "now" does not give much room for interpreting this as a contingent future possibility.
Rob thanks for the catch. Actually, I misquoted Andy Updegrove misquoting Alex. I've repaired Updegrove's blog title.
For the record, what Alex says is "In short, we find ourselves at a crossroads, and it seems to me that without a change of direction the entire OOXML project is now surely heading for failure."
I was not lamenting anything Alex said on this count. My observation is about the the many commenters who treat Alex's speculation as an assertion that the failure has indeed happened, as they predicted all along, etc. I have no quarrel with Alex's concern.
It is important to distinguish the standard from the technology.
So take OOXML as the whatever Office writes out. That remains relevant and important to document, purely because of the dominant market share of MS Office.
But as a standard, I think it was doomed to failure from day 1. The problem is that no one can effectively maintain OOXML other than Microsoft employees, because no one else knows the internals of MS Office. Also, features and therefor the internals of Office evolve at a more rapid pace than ISO can revise standards, at least with the resources that Microsoft has brought to the table, which is already so large that some NBs are starting to complain.
So what is the solution? Stop improving MS Office? I don't think so. Add more Microsoft resources to the task, so instead of being 50% of the people on the committee, they are 75%? I don't think so.
If there was an easy solution here, I'm sure someone would have mentioned it by now. I think the problem is that the technology is fundamentally unfit for an ISO standard. It is changing at too rapid a pace relative to its size, as well as trying to fix a huge number of defects. Look at the work we've been doing on ODF 1.2. Imagine that scaled by 10.
I agree with Rob, at least as far as that the additional complexity of supporting Transitional is not worth the effort.
The documentary benefit of Transitional has been gained. Now all Part 4 adds is confusion and duplicated effort.
Strict is plenty big enough to provide any features that might justify OOXML's existence, until the time when ODF finally catches up or gets a module mechanism that allows hygenic extensions.
We seem to have wandered off into some odd topics.
1. Whether OOXML was/is/will-ever-be fit to be standardized. I am not going there (and, as a member of the ODF TC I shudder to consider how that might backfire).
2. Whether transitional belongs at all.
3. The question that interests me, and perhaps Alex Brown, is how strict can be slid out from under transitional in a way that works on the ground. It is clearly important to see how Microsoft will provide movement in that directon. Although I don't think the patching of IS 29500 is complete, I suspect there is enough stability for movement to happen.
4. Finally, I am commenting on this at all because I think that maintenance and trouble-shooting I was performing on the blog trampled over Rick's comment. I trust that this addendum will drive out the latest version of all of the comments.
