<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=windows-1253">
<style type="text/css" id="owaParaStyle">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Arial;color: #000000;font-size: 12pt;">
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">James,</span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)"><br>
</span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">You ask a question at the end of this part of the discussion, and below I give you the answer as I understand it.<br>
</span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)"><br>
</span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">1. To the extent that the proposal envisions data files being enabled to self-specify a particular markup convention from among several choices, it seems
to violate the principle that the meaning of an item should not depend on the value of a different item.</span></p>
<div><br>
</div>
<div>In my opinion,this principle needs to be clarified. For example, any non-key data name 'depends' on the value of key data names in its category, or the meaning of a fractional coordinate 'depends' on the values of the cell parameters. We need a precise
rephrasing of the principle, e.g. "no new key data names may be added to a pre-existing category" or "where possible, the meaning of data names should depend only on data names that are identifiers". Once this is clarified we might be in a better position
to judge when a proposal is suspect. Does anyone know the underlying basis for this principle?</div>
<div><br>
</div>
<div><font face="Times New Roman">Here is the answer to this question. The original intent was that the item name should be sufficient to indicate to the reading program where to store the information given by the item, and in what form that information was
given, without having to refer to the value of some other item. For example we wanted to avoid defining an item: _atom_site.adp which might contain the anisotropic displacement parameters in any one of the forms B, beta or U, the particular form depending
on the value assigned to the item _atom_site.adp_type. Without consulting *.adp_type the information is meaningless. This required defining separate item names for each of the three different ways depending on how the reading program should store the value
of the item (or preferably restricting CIF to use only one convention). The situation with text is different. A text item such as _abstract or _title could be stored in the same place and processed without regard to the markup convention, up to the time that
it was being displayed.</font></div>
<div><font face="Times New Roman"><br>
</font></div>
<div><font face="Times New Roman">I hope this clarifies the thinking behind this rule.</font></div>
<div><font face="Times New Roman"><br>
</font></div>
<div><font face="Times New Roman">David</font><br>
</div>
<span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)"></span>
<div><br>
<div style="font-family:Tahoma; font-size:13px">I. David Brown<br>
Professor Emeritus<br>
Department of Physics and Astronomy<br>
McMaster University<br>
Hamilton, Ontario, Canada<br>
</div>
</div>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div id="divRpF934566" style="direction: ltr;"><font size="2" face="Tahoma" color="#000000"><b>From:</b> comcifs [comcifs-bounces@iucr.org] on behalf of James Hester [jamesrhester@gmail.com]<br>
<b>Sent:</b> September 20, 2017 20:59<br>
<b>To:</b> Discussion list of the IUCr Committee for the Maintenance of the CIF Standard (COMCIFS)<br>
<b>Subject:</b> Re: Proposal to regulate markup in CIF files<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr">
<div>
<div>Dear All,<br>
<br>
</div>
I've turned the proposal into a discussion document, separating non-ASCII character markup and other markup as per John's comments. The best way forward for 'other markup' is not clear to me, so I've asked a few questions at the end of the document that I invite
you to consider.<br>
<br>
I've also inserted replies to John's comments inline below.<br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">James.<br>
<br>
</div>
<div class="gmail_extra">
<div class="gmail_quote">On 14 September 2017 at 01:08, Bollinger, John C <span dir="ltr">
<<a href="mailto:John.Bollinger@stjude.org" target="_blank">John.Bollinger@stjude.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_4498485416747684331m_5029037233165456915m_-831946348248111301m_916103177838097246m_8895441407425487054WordSection1">
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">Dear Colleagues,</span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">I think the proposed approach of allowing markup convention to be specified in data files could be useful. Moreover, the proposal would provide a mechanism
for formalizing the specification of which items are subject to markup in the first place. Supposedly, that is determined by items’ definitions, but in practice, few, if any, current dictionaries or definitions actually address the topic explicitly.</span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">Considerations:</span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">1. To the extent that the proposal envisions data files being enabled to self-specify a particular markup convention from among several choices, it seems
to violate the principle that the meaning of an item should not depend on the value of a different item.</span></p>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>In my opinion,this principle needs to be clarified. For example, any non-key data name 'depends' on the value of key data names in its category, or the meaning of a fractional coordinate 'depends' on the values of the cell parameters. We need a precise
rephrasing of the principle, e.g. "no new key data names may be added to a pre-existing category" or "where possible, the meaning of data names should depend only on data names that are identifiers". Once this is clarified we might be in a better position
to judge when a proposal is suspect. Does anyone know the underlying basis for this principle?<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_4498485416747684331m_5029037233165456915m_-831946348248111301m_916103177838097246m_8895441407425487054WordSection1">
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">2. Since the codes for various supported markup conventions would be defined in domain dictionaries, it seems we might be setting ourselves up for future
issues in the event that dictionary maintainers want to support new markup conventions, for then we will need to change existing definitions (which, granted, we have lately afforded ourselves some freedom to do).</span></p>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>The point being that software written with one set of markup conventions in mind would be caught unawares by data values written according to a new markup convention. However, the idea is that _publ.markup_convention is used to flag the use of a new convention
and allow such software to act appropriately. Adding new enumerated values is a normal way of developing dictionaries and I don't think is controversial. I have removed this from the proposal for now.<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_4498485416747684331m_5029037233165456915m_-831946348248111301m_916103177838097246m_8895441407425487054WordSection1">
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">3. Additionally, the proposal seems to imply that each domain dictionary would need to either specify its own analogue(s) of `_publ.markup_convention`,
or else to explicitly depend on the Core item. I am uncertain whether this is a good or bad thing.</span></p>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Yes, this is true. All of our dictionaries currently depend on the core dictionary in any case and this is unlikely to change. A completely different domain would need to define its own analogue.
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_4498485416747684331m_5029037233165456915m_-831946348248111301m_916103177838097246m_8895441407425487054WordSection1">
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">4. The proposal does not preclude individual items from being defined to have whatever content type is desired for them, as long as the definitions are
not flagged as carrying `Marked-up` data. That’s good, but I can imagine hypothetical cases in which it would be confusing to dictionary authors. For example, if an item were defined to contain HTML markup, it would be necessary for its definition to specify
that its values were NOT `Marked-up`.</span></p>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>This is true. <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_4498485416747684331m_5029037233165456915m_-831946348248111301m_916103177838097246m_8895441407425487054WordSection1">
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">5. Although we call the existing CIF conventions “markup” – and that’s fitting – for the most part they don’t serve the same purpose as Markdown, reStructured
text, etc.. Almost all of our markup is aimed at encoding specific characters, whereas the other markup conventions mentioned are focused on document structure and styling. These are largely orthogonal considerations, so perhaps we should approach them separately.</span></p>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I have done this in the revised proposal. <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_4498485416747684331m_5029037233165456915m_-831946348248111301m_916103177838097246m_8895441407425487054WordSection1">
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">6. The discussion accompanying the proposal distinguishes between items that are intended to be machine-actionable and those intended purely for human
consumption, with the assertion that only the latter kind should be subject to markup. I would accept that if it referred to structural markup only, but it is not so clear that such a rule is appropriate for markup that serves to encode characters. Consider,
for example, `_atom_site.label`. As a key data name, it certainly has machine significance, but its values are also meant to identify atoms to humans. Should CIFs, then, be forbidden from using `C</span><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">á`</span><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">
(i.e. `C\a`) as or in atom labels? Forbidding use of markup would prevent literal `C</span><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">á` from being expressed in an atom label in a CIF 1.1 document, but not in a CIF
2.0 document, so that sets up a pathway wherein markup gets introduced into data values through format transliteration from CIF2 to CIF1. If only the original document were considered valid in such cases, then that would constitute a rather nasty trap to
set for ourselves.</span><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)"></span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)"> </span></p>
</div>
</div>
</blockquote>
<div>Agreed, hopefully the rewritten proposal addresses this. <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_4498485416747684331m_5029037233165456915m_-831946348248111301m_916103177838097246m_8895441407425487054WordSection1">
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">Initial analysis:</span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">As presented, the proposal’s largest impact would probably be to provide for DDLm dictionaries to specify which items are primarily (or exclusively) intended
for human consumption: those that are defined to have `Marked-up` content, regardless of whether any markup is actually present in their values. Initially, at least, this would establish in which values to interpret the standard CIF markup conventions. That
would be worthwhile.</span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt; font-family:"Calibri",sans-serif; color:rgb(31,73,125)">I am less certain about the prospects for or usefulness of enabling data files to select alternative markup conventions. Perhaps that could be used to
good effect to support revisions to the markup conventions. Perhaps it would be more broadly applicable. But perhaps we should avoid making any items’ meanings depend on other items’ values.</span></p>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I have rewritten the proposal to remove the option of specifying particular types of markup, and indeed largely removed discussion of markup pending some feedback from this group.<br>
<br>
</div>
<div>James.<br>
================================================================<br>
Revised discussion paper for regulating markup of CIF text items<br>
================================================================<br>
<br>
Summary<br>
=======<br>
<br>
1. Data values containing backslash escapes for indicating non-ASCII<br>
characters are to be considered entirely equivalent to values<br>
obtained after all such substitutions and escapes have been applied.<br>
<br>
2. Other mark-up (superscripts, subscripts, italic etc.) is given a<br>
new type, but is otherwise unspecified and needs to be discussed.<br>
<br>
Introduction<br>
============<br>
<br>
>From the very first publication describing CIF, markup conventions<br>
have been provided in order to extend the range of characters and font<br>
effects representable in ASCII. Which data values these conventions<br>
might apply to, and whether or not this is more properly a CIF syntax<br>
or dictionary (semantic) issue, has been left implicit.<br>
<br>
Marked-up text according to the ad-hoc definitions described in Vol G<br>
appears both in CIF data files and in dictionary definitions. While<br>
COMCIFS has control over the conventions applying within dictionaries,<br>
it has far less control over data values in data files, which are<br>
produced both by dedicated software, such as publCIF, and hand-editing<br>
or local ad-hoc solutions. Marked-up text in data files plays an<br>
important role in the publication workflow.<br>
<br>
Vol G (First Edition) notes in section <a href="http://2.2.5.3" target="_blank">2.2.5.3</a>: "It is hoped that in<br>
future different types of such markup may be permitted so long as the<br>
data values affected can be tagged with an indication of their content<br>
type that allows the appropriate content handlers to be invoked". It is<br>
not, however, clear that multiple alternative markups are desirable.<br>
<br>
Moving forward<br>
==============<br>
<br>
The markup in use can be divided into two classes: 'character encoding'<br>
and 'font effects'. Under this proposal, each class is treated differently.<br>
<br>
1. Character encoding<br>
<br>
Character encoding markup represents non-ASCII letters using a<br>
backslash followed by one or more ASCII characters, for example,<br>
'\a' is 'greek letter alpha'. This is a format-specific method of<br>
allowing access to the full characterset used by the DDL textual<br>
types. From the point of view of the (format-agnostic) dictionary<br>
data model, how a particular format wishes to encode characters is<br>
irrelevant. Therefore, the set of character escapes is most<br>
appropriately documented as part of the description of CIF syntax,<br>
not within a DDL dictionary. In other words, CIF1/2 data values with<br>
backslash character escapes are semantically identical to CIF2 data<br>
values where those escapes have had their Unicode equivalents<br>
substituted.<br>
<br>
2. Font effects<br>
<br>
Font effects differ from character encodings in not having a DDLm<br>
type that they are a concrete realisation of. By analogy, then, we<br>
could create a DDLm type 'Marked-up text', whose contents contain<br>
marked-up text. Particular implementations and syntaxes might then<br>
specify what particular convention(s) 'Marked up text' should<br>
conform to.<br>
<br>
Notes<br>
=====<br>
<br>
1. An important function of the 'Marked-up text' type is to<br>
designate data values that are not intended to be machine-actionable.<br>
No DDLm functions or attributes are envisioned for manipulating the<br>
markup. The type could alternatively be something like 'Rich text'.<br>
<br>
2. Enumerated values and identifiers must not be of type marked-up text.<br>
<br>
3. The 'marked-up text' data value is obtained from a CIF syntax<br>
file after backslash character codes have been substituted.<br>
<br>
Open questions<br>
==============<br>
<br>
The above proposal does not specify a particular markup convention. Leaving<br>
anything unspecified is dangerous for a standard, as it invites the appearance<br>
of multiple, incompatible solutions. We should as a matter of urgency answer<br>
the following questions:<br>
<br>
(1) Should alternative markup conventions be possible?<br>
(2) If yes, should the markup convention in use be<br>
(i) per dataname?<br>
(ii) per datavalue (maybe via an embedded flag)?<br>
(iii) per data block?<br>
(iv) per dictionary?<br>
(v) per syntax? (e.g. CIF/CIF-JSON/HDF5 etc.)<br>
(3) If no, the current convention is the only possible one for reasons of<br>
backward compatibility. Should it be:<br>
(i) a feature of CIF syntax?<br>
(ii) a feature of CIF syntax when combined with a DDLm dictionary?<br>
(iii) defined in DDLm?<br>
<br>
My answers to these questions would be<br>
(1) No alternatives should be possible, in order to simplify publishing<br>
workflows and maintain the publCIF investment<br>
(3) (ii)<br>
<br>
Some explanation regarding (ii), which possibly sounds a bit<br>
abstruse. A CIF syntax file can be used (in theory) with an<br>
alternative dictionary language and associated data model. Likewise,<br>
DDLm dictionaries can be used to describe non-CIF files. In each<br>
case, the way in which syntactical data values are constructed to<br>
match the dictionary types may differ (for example, numbers may be a<br>
text string or binary). Each combination of syntax and dictionary<br>
must explicitly state how each dictionary type is represented in that<br>
syntax. So I am suggesting that for the combination 'CIF + DDLm' that<br>
we specify the current markup conventions to represent type 'Marked-up<br>
text'.<br>
</div>
</div>
<br>
-- <br>
<div class="gmail-m_4498485416747684331m_5029037233165456915m_-831946348248111301m_916103177838097246gmail_signature">
T <a href="tel:+61%202%209717%209907" value="+61297179907" target="_blank">+61 (02) 9717 9907</a><br>
F <a href="tel:+61%202%209717%203145" value="+61297173145" target="_blank">+61 (02) 9717 3145</a><br>
M +61 (04) 0249 4148</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>