<div dir="ltr"><div><div>Dear COMCIFS,<br><br>As per the my email of last December (see <a href="http://www.iucr.org/__data/iucr/lists/comcifs-l/msg00736.html">http://www.iucr.org/__data/iucr/lists/comcifs-l/msg00736.html</a>) a simple mechanism has now been developed to clearly flag the presence of redefined datanames in a datafile. Neither the ddlm-group nor the core CIF DMG have raised objections. The full proposal is reproduced below, and can be read in a more nicely formatted form at <a href="https://github.com/COMCIFS/comcifs.github.io/blob/master/audit.formalism_proposal.md">https://github.com/COMCIFS/comcifs.github.io/blob/master/audit.formalism_proposal.md</a><br><br></div>All comments are welcome. If no objections are received within 3 weeks, this proposal will be considered approved.<br><br></div>James Hester (chair).<br clear="all"><div><div><div><div><div><div>==========================================<br><br># Proposal for new dataname and attribute to cover differing models<br><br>## Introduction<br><br>The following proposal implements part of a solution to incorporating<br>multiple models into CIF, discussed<br>[here](<a href="http://changing_meanings_discussion_paper.md">changing_meanings_discussion_paper.md</a>). It should be read in<br>conjunction with that document.<br><br>## New datanames: `_audit.formalism` and `_audit.formalism_version`<br><br>Each value of `_audit.formalism` corresponds to a particular way of<br>deriving some set of CIF datanames from other datanames defined in the<br>same, or imported, dictionaries. For better interoperability, we<br>stipulate that datanames may only be redefined by dictionaries if the<br>redefined datanames take values with the same units and domain as the<br>original datanames. So, for example, `_refln.F_complex` for magnetic<br>structures is calculated differently to `_refln.F.complex` for core<br>CIF, but has the same units and domain (positive real numbers), so it<br>is acceptable for a magnetic dictionary to redefine<br>`_refln.F_complex`.<br><br>`_audit.formalism_version` is provided to allow secondary parameters<br>to be added to the model without changing the overall formalism. As a<br>guide, parameters are considered secondary if they do not require the<br>addition of new columns to any category, and do not significantly<br>change the final calculated values in "typical" cases.<br><br>All CIF datablocks should include these new datanames when they take<br>non-default values; the default values correspond to the<br>single-crystal model described in core CIF. Any CIF reading programs<br>that perform calculations should check `_audit.formalism` and `_audit.formalism_version` <br>datanames in order to avoid miscalculating derived values. <br><br>The choice of the word `formalism` is purely to avoid clashing with<br>the widespread use of `model` in core CIF to refer to the particular<br>arrangement of atoms. There may be a better word. See the appendix<br>for formal DDLm definitions.<br><br>## New DDLm attributes: `_dictionary.formalism` and `_dictionary.formalism_version`<br><br>These attributes associate a dictionary with a particular formalism.<br><br>## Treatment of current dictionaries<br><br>### Modulated structures<br><br>The modulated structures dictionary is assigned formalism `modulated`<br>and redefines `_refln.F_complex`, `_refln.sin_theta_over_lambda` and<br>`_refln.symmetry_multiplicity`.<br><br>### Magnetism<br><br>The magnetism dictionary builds on the modulated structures dictionary.<br>It is assigned formalism `magnetic` and redefines `_refln.F_complex`<br>only.<br><br>### Powder<br><br>The powder dictionary calculates structure factors from information<br>that may be held in a different datablock. It therefore redefines<br>`_refln.F_complex`. `_refln.F_meas` is also redefined as the<br>determination of this from the powder observations is markedly<br>different to the way in which it is derived from single-crystal spots,<br>not least because of pervasive overlap.<br><br>Separate formalisms are necessary for each possible combination of<br>powder with other formalisms, for example `_audit.formalism` can<br>take values `powder-magnetic`, `powder-modulated` and<br>`powder-multipole`.<br><br>### Electron density<br><br>The electron density dictionary allows parameterisation of the<br>electron density around each atom in terms of multipoles.<br>`_refln.F_complex` is redefined, and a formalism of `multipole` is<br>assigned.<br><br>### Constraints and restraints<br><br>This dictionary relates only to the method of determination of the<br>final parameters and therefore does not affect the definitions of<br>the final datanames.<br><br>### Twinning<br><br>Twinning does not change the structural model, but it may change the<br>way `_refln.F_meas` is calculated from the observations. A formalism<br>of `twinning` is assigned, and as for powder separate formalisms need<br>to be assigned for each distinct structural model.<br><br>### Image CIF<br><br>ImgCIF relates only to raw data and is not affected by these changes.<br><br>### mmCIF<br><br>mmCIF is based on the core CIF model and is therefore unaffected by<br>these changes.<br><br>## Treatment of other techniques<br><br>### Laue<br><br>A Laue experiment measures distinct spots, but each spot is produced<br>by a distinct wavelength, and spots sometimes overlap.<br>`_refln.wavelength` therefore becomes an additional key column in<br>`refln`. This change by itself is easily covered by defining a<br>different `_audit.schema`. However, a Laue dictionary must also<br>redefine `_refln.F_meas` as the extraction of notional observed<br>intensities will depend on the model for wavelength distribution, and<br>so we must assign a separate `_audit.formalism`. As for powder and<br>twinning, there will be a separate `formalism` for each distinct<br>structural model.<br><br>## Discussion<br><br>### Mixing and matching not possible<br><br>It is tempting to define something like `_audit.technique` to cover<br>the technique-based differences, so that `_audit.technique` and<br>`_audit.formalism` could correspond to different dictionaries that<br>could be mixed and matched. So, instead of a `powder-magnetism`<br>formalism, there would simply be a `powder` technique combined with a<br>`magnetism` formalism, with both dictionaries being separately imported<br>and notionally orthogonal to one another.<br><br>However, any `formalism` that adds keys to the `refln` category will<br>also require the `technique` to be aware of those keys in order to<br>explain how `_refln.F_meas` is determined. For example, a powder<br>experiment on a modulated structure will calculate the `_refln.F_meas`<br>value differently to a powder experiment on a non-modulated structure<br>as the calculations of peak position require different numbers of<br>indices. Therefore, it is not possible to generally separate the<br>technique from the structural model, although it may be possible in<br>particular cases.<br><br>### Just use `_audit_conform`?<br><br>Core CIF has long provided the `_audit_conform_dict_*` tags to state which<br>dictionary or dictionaries a datablock conforms to. This appears almost<br>as simple as the proposed `_audit.formalism` tag, so the need for a<br>separate tag may not be apparent.<br><br>While the `_audit_conform` mechanism must remain the<br>canonical source of information, the proposed dataname provides a<br>simplified route to the same information. In order for a CIF reading<br>program to confirm that none of the dictionaries listed in a CIF block<br>change any of the definitions relied upon by that program, it must in<br>general download the stated versions of the dictionary or dictionaries<br>from the canonical IUCr site, parse and merge them, and then find any<br>definitions that have (apparently) been replaced. Compared to this<br>procedure, the `_audit.formalism` tag is a much simpler way for the<br>datablock writer to specify to the datablock reader a particular set<br>of dataname interpretations that may never change.<br><br>Note that the `_audit_conform_*` mechanism is almost never used. As of<br>May 28, 2016, there were 195 modulated structures in the<br>Crystallographic Open Database (as determined by the presence of<br>`Fourier_wave_vector` in a file). Of these, zero had an<br>`_audit_conform` entry, which in theory would be required to explain<br>the adjusted interpretation of the `refln` category and new datanames.<br>We conclude that introduction of `_audit.formalism` should be<br>accompanied by an education and outreach program as well.<br><br>### Interaction with `_audit.schema`<br><br>`_audit.schema` essentially allows fixed parameters to vary. It<br>is therefore orthogonal to `_audit.formalism`: a given formalism<br>may have many possible schemas, and many schemas may apply to<br>multiple formalisms if they share the same parameters.<br><br>In other words, a suitably-written program can handle a variety of<br>schemas for a single formalism without needing to change the way in<br>which any dataname is calculated, whereas a program must change the way<br>in which the redefined datanames are calculated if the formalism<br>changes.<br><br># Appendix I: New core definitions<br><br>## _audit.formalism<br><br>```<br>save_audit.formalism<br><br>_<a href="http://definition.id">definition.id</a> '_audit.formalism'<br>_name.category_id audit<br>_name.object_id formalism<br>_description.text<br>;<br><br> The CIF dictionaries listed in _audit.dictionary may redefine<br> datanames. _audit.formalism is provided as an efficient<br> alternative to parsing and checking those dictionaries. It<br> identifies commonly-used sets of meanings for datanames. In<br> general, each value taken by _audit.formalism is linked to a<br> particular technique and/or structural approach. The<br> dictionaries for the datablock (see _audit.dictionary) must be<br> compatible with the value of _audit.formalism.<br><br>;<br>_type.contents Text<br>_type.purpose State<br>_type.container Single<br>_type.source Assigned<br>loop_<br>_enumeration_set.state<br>_enumeration_set.detail<br> Base 'Single crystal model from core CIF'<br> Modulated 'Single crystal modulated structure'<br> Magnetic 'Single crystal magnetic structure, potentially modulated'<br> Powder 'Powder diffraction experiment'<br> Twinned 'Twinned crystal using core CIF model'<br> Multipole 'Single crystal model with multipole coefficients'<br> Laue 'Laue experiment on single crystal'<br> Powder-Modulated 'Powder experiment on a modulated structure'<br> Powder-Magnetic 'Powder experiment on a modulated magnetic structure'<br> Powder-Multipole 'Powder experiment modelled with multipoles'<br> Laue-Magnetic 'Laue experiment on magnetic structure'<br> Laue-Modulated 'Laue experiment on modulated non-magnetic structure'<br> Laue-Multipole 'Laue experiment modelled with multipoles'<br> Twinned-Magnetic 'Twinned magnetic single crystal structure'<br> Twinned-Modulated 'Twinned modulated single crystal structure'<br> Laue-Twinned 'Laue experiment on twinned single crystal'<br> Laue-Twinned-Modulated 'Laue experiment on twinned modulated structure'<br> Custom 'Examine dictionaries provided in _audit_conform'<br> Local 'Locally modified dictionaries. These datafiles should not be distributed'<br>_enumeration.default Base<br>save_<br>```<br><br>## _audit.formalism_version<br><br>```<br>save_audit.formalism_version<br><br>_<a href="http://definition.id">definition.id</a> '_audit.formalism_version'<br>_name.category_id audit<br>_name.object_id formalism_version<br>_description.text<br>;<br><br> The version of the given formalism (see `_audit.formalism`). The version<br> number of a formalism is incremented when new model parameters are<br> added that do not significantly affect the model values in typical cases.<br><br>;<br>_type.contents Text<br>_type.purpose State<br>_type.container Single<br>_type.source Assigned<br>_enumeration.default 1.0<br>save_<br>```<br><br>## _dictionary.formalism<br><br>```<br>save_dictionary.formalism<br><br> _<a href="http://definition.id">definition.id</a> '_dictionary.formalism'<br> _definition.class Attribute<br> _definition.update 2016-12-17<br> _description.text <br>;<br><br> The value of this attribute is associated with the set of<br> dataname meanings contained in this dictionary.<br><br>;<br> _name.category_id dictionary<br> _name.object_id formalism<br> _type.purpose Audit<br> _type.source Assigned<br> _type.container Single<br> _type.contents Text<br><br>save_<br>```<br><br># Appendix II: a full hybrid dictionary<br><br>A complete dictionary using the above mechanisms is presented below.<br><br>```<br>#\#CIF_2.0<br>####################################################<br># #<br># Dictionary for modulated powder diffraction #<br># #<br>####################################################<br><br>data_MODPOW<br><br>_dictionary.title MODPOW<br>_dictionary.formalism Powder-Modulated<br>_dictionary.class Instance<br>_dictionary.version 1.0<br>_dictionary.date 2016-12-19<br>_dictionary.ddl_conformance 3.12<br>_dictionary.namespace MODPOW<br>_description.text<br>;<br><br> The modulated powder diffraction dictionary redefines datanames<br> for use when presenting the results of a powder diffraction<br> experiment using a modulated structure model. The remainder of the<br> relevant definitions are found in the modulated structures<br> dictionary and the powder diffraction dictionary.<br> <br>;<br><br>save_MODPOW_GROUP<br><br> _<a href="http://definition.id">definition.id</a> MODPOW_GROUP<br> _definition.scope Category<br> _definition.class Head<br> _definition.update 2016-12-19<br> _description.text<br>;<br> This category is the parent category for all definitions<br> in the MODPOW dictionary<br>;<br><br> _name.category_id MODPOW<br> _name.object_id MODPOW_GROUP<br> <br> # The following import reads in and reparents all powder and<br> # modulated structure definitions to the MODPOW_GROUP category. As cif_ms is<br> # read second, the refln category will have the extra modulation indices<br> # defined.<br> <br> _import.get [{"file":"cif_pow.dic" "save":"PD_GROUP" "mode":"Full"}<br> {"file":"cif_ms.dic" "save":"MS_GROUP" "mode":"Full"}]<br><br>save_<br><br>save__refln.F_complex<br><br>_<a href="http://definition.id">definition.id</a> '_refln.F_complex'<br>loop_<br> _alias.definition_id<br> '_refln.F_complex' <br> '_refln_F_complex' <br>_definition.update 2016-12-19<br>_description.text <br>;<br> The structure factor vector for the reflection calculated from<br> the modulated structure given in the datablock identified by<br> _refln.phase_id<br>;<br>_name.category_id refln<br>_name.object_id F_complex<br>_type.purpose Measurand<br>_type.source Derived<br>_type.container Single<br>_type.contents Complex<br>_enumeration.default 0.<br><br>#<br># A complete dREL expression for F_complex can be provided here, using all of the<br># parameters provided in the powder and modulated structure dictionaries.<br>#<br>save_<br><br>save__refln.F_meas<br><br>_<a href="http://definition.id">definition.id</a> '_refln.F_meas'<br>loop_<br> _alias.definition_id<br> '_refln.F_meas' <br> '_refln_F_meas' <br>_definition.update 2016-12-19<br>_description.text <br>;<br> The structure factor amplitude for the modulated reflection based on<br> partitioning of each observed powder diffraction intensity between<br> contributing reflections in proportion to the model reflection contributions.<br>;<br>_name.category_id refln<br>_name.object_id F_meas<br>_type.purpose Measurand<br>_type.source Derived<br>_type.container Single<br>_type.contents Real<br>_enumeration.default 0.<br>#<br># A complete dREL expression for calculating F_meas from an observed powder diffractogram can be given here.<br>#<br>save_<br><br>-- <br><div class="gmail_signature">T +61 (02) 9717 9907<br>F +61 (02) 9717 3145<br>M +61 (04) 0249 4148</div>
</div></div></div></div></div></div></div>