<div dir="ltr"><div><div>Dear COMCIFS members,<br><br></div>The 6-week discussion period having elapsed, the inter-block linking proposal is now accepted. I will inform dictionary authors of the new tool at their disposal.<br><br></div>James Hester (chair).<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On 19 February 2018 at 16:37, James Hester <span dir="ltr"><<a href="mailto:jamesrhester@gmail.com" target="_blank">jamesrhester@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Dear COMCIFS members<br><br></div>One inadequately
resolved issue that arises in dealing with the more complex dictionaries is how links between data blocks are handled
(msCIF/pdCIF/magCIF). The end of Section 2.2.6 in Vol G, 1st edition also alludes to a need for resolution of this problem. The following proposal has therefore been developed as a way of standardising linking between data blocks, and has the pleasant side-effect that it also explains the context
in which dREL methods operate when presented with multiple data blocks. The dictionary most reliant on data block linking is the modulated structure CIF, and Gotzon Madariaga, the original author, has confirmed that this proposal would meet the needs of that dictionary.<br><br></div>I've posted the proposal at <a href="https://github.com/COMCIFS/comcifs.github.io/blob/master/InterBlockLinkProposal.rst" target="_blank">https://github.com/COMCIFS/com<wbr>cifs.github.io/blob/master/Int<wbr>erBlockLinkProposal.rst</a>. I've also included it as ASCII at the end of this email
for the record, but strongly urge you to go to the above link for a
more nicely formatted version.<br><br></div>The proposal remains open for discussion for 6 weeks. If all issues have been resolved by that time, it will be considered accepted.<br><br></div>James.<br><br clear="all">Proposal to Improve Inter Block Linking<br>******************************<wbr>*********<br><br>Introduction<br>============<br><br>A number of current CIF dictionaries rely on pointers to other<br>data blocks. In particular:<br><br>* A powder sample can consist of multiple components (perhaps<br> confusingly called "phases"). Each of these materials is described<br> in a separate data block and then referenced from the block in which<br> the overall fit to the data is described. Powder diffraction patterns may<br> also be referenced in the same way.<br><br>* The modulated structures dictionary (msCIF) allows for composite<br> structures, with the space group and cell of each member of the<br> composite described in separate data blocks<br><br>* The new magnetic structures dictionary envisions that alternative<br> descriptions of the same magnetic structure could be described in<br> separate data blocks.<br><br>In addition, cif core defines the ``_audit_link.*`` data names, which allow listing of<br>datablock identifiers together with some plain text description of the nature of<br>the relationship between the blocks.<br><br>Each scheme is different, expects different contents for the linked<br>data blocks, and varies in the degree to which computers can use the<br>linking information.<br><br>In addition, work on 'CheckCIF for raw data' requires some way of<br>working with the many different ways of arranging image frames into<br>files and directories.<br><br>This proposal therefore suggests formalising data block linkages in a<br>way that allows arbitrary expansion as well as making the precise<br>nature of the relationships between data blocks machine-readable in<br>all situations.<br><br>General idea<br>============<br><br>First some definitions:<br><br>**key**<br> the set of datanames whose values together define a unique<br> row<br> <br>**projection**<br> choosing a single value for each key data name, and then<br> selecting only those rows of categories that correspond to those<br> particular values, ignoring any categories for which that data name<br> is irrelevant.<br><br>Any data collection can be split by projecting over the individual<br>values of a data name that forms part of a key, and putting the result<br>for each projection into a separate block. These projected blocks do<br>not have to include the projected dataname within loops, as the<br>dataname is constant within a data block.<br><br>If multiple blocks belonging to the same data collection are able to<br>be described as projections of one or more data names we have<br>completely defined their relationship.<br><br>Proposal for a Universal System for Linking Data Blocks<br>==============================<wbr>=========================<br><br>In order to completely define data block interrelationships, data<br>blocks resulting from projection over a key value must be:<br><br>(1) linked and<br>(2) the key data name used for the projection indicated and correct values assigned.<br><br>Linkage<br>-------<br><br>This is accomplished by including a dataname which has the same,<br>unique value in all linked blocks. I propose calling this dataname<br>``_<a href="http://audit_dataset.id" target="_blank">audit_dataset.id</a>``<br><br>Key values<br>----------<br><br>Each data block must explicitly state the value of the parent key<br>against which it has been projected, by providing the parent key's<br>data name and value.<br><br><br>Effect on existing dictionaries<br>==============================<wbr>=<br><br>The following examines changes or enhancements required to implement<br>this approach in those dictionaries that already use inter-block<br>references. In no case do existing data names require redefinition or<br>removal; the new system can exist alongside the old system.<br><br>Comparison for msCIF<br>--------------------<br><br>The current msCIF arrangement splits the information for composite<br>structures over a number of data blocks. The 'master' block contains a<br>loop indexed by ``_cell_subsystem.code``, the value of which is used to<br>find structural data in a separate data block that has an<br>``_audit_link.block_code`` identifier composed as<br>``<arbitrary>REFRNCE_<code>``. <br>``<arbitrary>`` is a string common to all linked blocks and ``<code>`` is the<br>value of ``_cell_subsystem.code`` appearing in a row of the<br>``_cell_subsystem`` loop. If the word MOD is used instead of REFRNCE, the<br>data block contains a description of the modulated structure of this<br>component. If no trailing ``<code>`` is present, the data block contains<br>either common structural features (REFRNCE) or the whole modulated<br>structure (MOD). If neither REFRNCE or MOD are present, the data block<br>contains common data items. In this approach, the links between data blocks<br>are created through the ``_audit_link.block_code`` value matching, and<br>the nature of the link is signalled by the reserved words MOD and<br>REFRNCE.<br><br>In the proposed approach (see example at end):<br><br>1. ``<arbitrary>`` becomes the value of ``_<a href="http://audit_dataset.id" target="_blank">audit_dataset.id</a>``<br>2. ``_cell_subsystem.code`` should be given in each data block for which<br> it is relevant<br>3. the loop over ``_cell_subsystem.code`` in the "master" block should<br> either be removed, or else a new child data name defined for use<br> in projection (choice of dictionary authors).<br>4. The msCIF dictionary should include child data names of ``_cell_subsystem.code``<br> in the ``space_group``, ``cell`` and all ``_atom_site_*`` categories<br><br>Comparison for pdCIF<br>--------------------<br><br>pdCIF defines a ``_pd.block_id`` and then links to blocks via this ID<br>in three distinct ways: (Vol G p 124):<br><br>(1) To refer to a block containing a diffraction measurement<br>(2) To refer to a block containing a structure description<br>(3) To refer to a block containing calibration information<br><br>In the proposed approach:<br><br>1. ``_<a href="http://audit_dataset.id" target="_blank">audit_dataset.id</a>`` can be used instead, or alongside, ``_pd.block_id``. Whereas<br> ``_pd.block_id`` is different for every block, ``_<a href="http://audit_dataset.id" target="_blank">audit_dataset.id</a>`` is identical for<br> blocks belonging to a single dataset.<br>2. A new (key) data name, e.g. ``_<a href="http://pd_diffractogram.id" target="_blank">pd_diffractogram.id</a>`` is created and stated in any<br> data blocks containing diffractograms.<br>3. The value of ``_pd.phase_id`` is stated in any data blocks containing structures<br>4. New child key data names are added to any categories that could appear in the<br> separate ``_pd.phase_id`` or ``_<a href="http://pd_diffractogram.id" target="_blank">pd_diffractogram.id</a>`` blocks<br>5. Calibration: an external calibration dataset is a combination of a<br> diffractogram and a phase. A ``_pd_calib_std_external.<wbr>phase_id`` and<br> ``_pd_calib_std_external.<wbr>diffractogram_id`` should be additionally<br> defined. (1),(2) and (3) are carried out as relevant for the<br> calibration dataset and phase. If calibrations involve more than<br> powder diffraction measurements, further data names describing<br> these measurements should be defined.<br><br>magCIF<br>------<br><br>The magnetic structures dictionary wishes to link to alternative descriptions of<br>the same magnetic structure in separate data blocks. In this case:<br><br>1. ``_audit.dataset_id`` is set to be identical in all relevant data<br> blocks<br>2. a dataname along the lines of ``_<a href="http://magn_structure_transform.id" target="_blank">magn_structure_transform.id</a><wbr>`` is set in each of<br> these data blocks<br>3. Child data names of ``_<a href="http://magn_structure_transform.id" target="_blank">magn_structure_transform.id</a><wbr>`` are added to all categories<br> that might be used in describing an alternative structure.<br><br>Advantages<br>==========<br><br>1. To a large extent, data can be added to datasets by simply creating<br> a new data block with the same ``_<a href="http://audit_dataset.id" target="_blank">audit_dataset.id</a>``. For example,<br> an extra measurement on a new sample of the same compound will<br> automatically be (semantically) incorporated into a dataset simply<br> by becoming present, whether in a separate file or an appended block<br>2. dREL methods can be written in complete ignorance of the way in<br> which data have been distributed over data blocks. In effect, a dREL<br> method operates in the context of all data available for a given value of<br> ``_<a href="http://audit_dataset.id" target="_blank">audit_dataset.id</a>``.<br>3. The effects of unexpected looping over 'Set' datanames that ``_audit.schema``<br> addresses can be reduced by using separate data blocks. So the choice<br> exists to split multiple crystals, multiple space-groups etc. over<br> multiple data blocks, without changing the underlying semantics.<br>4. Formats which collate many files to form the dataset are easy to<br> describe in this paradigm: for example, image frames in separate<br> files are simply assigned to the same dataset, with each file<br> including the value of the image identifier data name used to<br> 'project' the data file from the notional loop of images.<br>5. The system is open-ended in terms of allowing disparate items of information<br> to be collated together with well-defined relationships. This means it<br> can essentially cover all ways of aggregating data into datasets.<br>6. The old block linkage systems can remain in place and can be used to provide<br> double-checking where possible.<br><br>Disadvantages<br>=============<br><br><br>1. Flexibility in how data from complex datasets is distributed over<br> data blocks may cause unnecessary work for data reading software<br> programmers attempting to cover all situations. This could be<br> remedied by individual dictionaries recommending particular<br> approaches.<br><br>Interaction with ``_audit.schema``<br>==============================<wbr>====<br><br>We have recently defined a data name, ``_audit.schema``, that signals<br>when 'Set' categories have become looped in a data block. The present<br>proposal allows 'Set' categories to be always single-valued in a<br>single data block, yet take multiple values for the dataset as a<br>whole. We must therefore choose between alternative meanings of<br>``_audit.schema``: does it mean that 'Set' categories are looped<br>semantically or both semantically and syntactically (obviously if Set<br>categories are looped in a single data block (syntactically) then they<br>are also semantically looped)? I propose that, even if all data<br>blocks conform to the default schema, at least some values in related<br>data blocks are likely to be materially significant for interpretation<br>of one another (for example, multiple crystal measurements feed into<br>final values of I_meas) and so ``_audit.schema`` should indicate<br>semantic looping, i.e.<br><br>* ``_audit.schema`` **must** take a non-default value where Set categories<br> can take multiple values **and** a data block contains loops over<br> these Set categories.<br><br>* ``_audit.schema`` **must** take the appropriate non-default value if<br> information for a dataset has been spread over several data blocks.<br><br>* ``_audit.schema`` **must** only take the default value if the dataset<br> consists of a single block conforming to the core CIF dictionary.<br><br>On datasets<br>===========<br><br>Note that a single data block can belong to multiple data sets, for example<br>calibration information may be relevant to multiple data collections, or a single<br>measurement may be relevant to different modelling exercises (e.g. joint or<br>single refinement of X-ray and neutron data) and therefore have different<br>dataset identifiers in each case.<br><br>Discussion<br>==========<br><br>This approach is close in spirit to the work of Nick Spadaccini and<br>Syd Hall in creating DDLm Ref-loops, which were projections of specified<br>Set categories into save frames. The current proposal removes the<br>syntactical element, exposes the behaviour of the keys, and adopts a<br>global relational view of the underlying semantics.<br><br>Note that ``_audit.dataset_id`` is a grouping mechanism. The<br>particular value<br>taken by this data name is only relevant if the dataset is referred to<br>externally to that dataset. Therefore, if a data format allows data blocks<br>to be grouped in some other way (e.g. files in a directory, nodes in a<br>hierarchy) there is no need to explicitly assign a value to this dataname<br>during data block creation.<br><br>Example<br>=======<br><br>The following example shows part of a CIF for a modulated structure<br>composed of two components, LaS and NbS2. (based on `Example 3, p 271,<br>It Vol<br>G<<a href="http://it.iucr.org/Ga/ch4o3v0001/Catom_site_displace_Fourier.html" target="_blank">http://it.iucr.org/Ga/<wbr>ch4o3v0001/Catom_site_<wbr>displace_Fourier.html</a>>`_)<br>::<br><br> # Common data<br> data_LaSNbS2<br> # The common dataset identifier<br> _<a href="http://audit_dataset.id" target="_blank">audit_dataset.id</a> 1997-07-24|LaSNbS2|G.M.<br> # Signal which categories are split across datablocks<br> _audit.schema 'Modulated'<br> # Signal the type of calculations used<br> _audit.formalism 'Modulated Single Crystal'<br> # The actual dictionary that this conforms to<br> _audit_conform.dict_name 'msCIF.dic'<br> # Old linkage data may be kept. Not all following blocks included in<br> # this example for brevity<br> loop_<br> _audit_link_block_code<br> _audit_link_block_description<br> 1997-07-24|LaSNbS2|G.M.|<br> 'common experimental and publication data'<br> 1997-07-24|LaSNbS2|G.M.|_<wbr>REFRNCE<br> 'reference structure (common data)'<br> 1997-07-21|LaSNbS2|G.M.|_MOD<br> 'modulated structure (common data)'<br> 1997-07-24|LaSNbS2|G.M.|_MOD_<wbr>NbS2<br> 'modulated structure (1st subsystem)'<br> 1997-07-24|LaSNbS2|G.M.|_<wbr>REFRNCE_LaS<br> 'reference structure (2nd subsystem)'<br> 1997-07-21|LaSNbS2|G.M.|_MOD_<wbr>LaS<br> 'modulated structure (2nd subsystem)'<br><br> _cell_subsystems_number <wbr> 2<br> # The following loop is now split across data blocks<br> # or retained with a child data name used for projection<br> #loop_<br> # _cell_subsystem_code<br> # _cell_subsystem_description<br> # _cell_subsystem_matrix_W_1_1<br> # _cell_subsystem_matrix_W_1_4<br> # _cell_subsystem_matrix_W_2_2<br> # _cell_subsystem_matrix_W_3_3<br> # _cell_subsystem_matrix_W_4_1<br> # _cell_subsystem_matrix_W_4_4<br> # NbS2 '1st subsystem' 1 0 1 1 0 1<br> # LaS '2nd subsystem' 0 1 1 1 1 0<br><br> # Common experimental and publication data elided ...<br> <br> # Items concerning the modulated structure of the first<br> # subsystem<br><br> data_LaSNbS2_MOD_NbS2<br> # Old block identifier<br> _audit_block_code 1997-07-24|LaSNbS2|G.M.|_MOD_<wbr>NbS2<br> # Common dataset identifier<br> _<a href="http://audit_dataset.id" target="_blank">audit_dataset.id</a> 1997-07-24|LaSNbS2|G.M.<br> # Signal which categories are split across datablocks<br> _audit.schema 'Modulated'<br> # Signal the type of calculations used<br> _audit.formalism 'Modulated Single Crystal'<br> # The actual dictionary that this conforms to<br> _audit_conform.dict_name 'msCIF.dic'<br> # Projected key data name<br> _cell_subsystem_code NbS2<br> # Projected information for value = NbS2 of key data name<br> _cell_subsystem_description '1st subsystem'<br> _cell_subsystem_matrix_W_1_1 1<br> _cell_subsystem_matrix_W_1_4 0<br> _cell_subsystem_matrix_W_2_2 1<br> _cell_subsystem_matrix_W_3_3 1<br> _cell_subsystem_matrix_W_4_1 0<br> _cell_subsystem_matrix_W_4_4 1<br><br> loop_<br> _atom_site_Fourier_wave_<wbr>vector_seq_id<br> _atom_site_Fourier_wave_<wbr>vector_x<br> _atom_site_Fourier_wave_<wbr>vector_description<br> 1 0.568 'First harmonic'<br> 2 1.136 'Second harmonic'<br><br> loop_<br> _atom_site_displace_Fourier_id<br> _atom_site_displace_Fourier_<wbr>atom_site_label<br> _atom_site_displace_Fourier_<wbr>axis<br> _atom_site_displace_Fourier_<wbr>wave_vector_seq_id<br> Nb1z1 Nb1 z 1<br> Nb1x2 Nb1 x 2<br> Nb1y2 Nb1 y 2<br> S1x1 S1 x 1<br> S1y1 S1 y 1<br> S1z1 S1 z 1<br> S1x2 S1 x 2<br> S1y2 S1 y 2<br> S1z2 S1 z 2<br><br> #### End of modulated structure first subsystem data ######<br><br> # Items concerning the modulated structure of the second<br> # subsystem<br><br> data_LaSNbS2_MOD_LaS<br> # Old block identifier<br> _audit_block_code 1997-07-24|LaSNbS2|G.M.|_MOD_<wbr>LaS<br> # Common dataset identifier<br> _<a href="http://audit_dataset.id" target="_blank">audit_dataset.id</a> 1997-07-24|LaSNbS2|G.M.<br> # Signal which categories are split across datablocks<br> _audit.schema 'Modulated'<br> # Signal the type of calculations used<br> _audit.formalism 'Modulated Single Crystal'<br> # The actual dictionary that this conforms to<br> _audit_conform.dict_name 'msCIF.dic'<br> # Projected key data name<br> _cell_subsystem_code LaS<br> # Projected information for value = LaS of key data name<br> _cell_subsystem_code LaS<br> _cell_subsystem_description '2nd subsystem'<br> _cell_subsystem_matrix_W_1_1 0<br> _cell_subsystem_matrix_W_1_4 1<br> _cell_subsystem_matrix_W_2_2 1<br> _cell_subsystem_matrix_W_3_3 1<br> _cell_subsystem_matrix_W_4_1 1<br> _cell_subsystem_matrix_W_4_4 0<br><br> loop_<br> _atom_site_Fourier_wave_<wbr>vector_seq_id<br> _atom_site_Fourier_wave_<wbr>vector_x<br> _atom_site_Fourier_wave_<wbr>vector_z<br> _atom_site_Fourier_wave_<wbr>vector_description<br> 1 1.761 0.5 'First harmonic'<br> 2 3.522 1.0 'Second harmonic'<br><br> loop_<br> _atom_site_displace_Fourier_id<br> _atom_site_displace_Fourier_<wbr>atom_site_label<br> _atom_site_displace_Fourier_<wbr>axis<br> _atom_site_displace_Fourier_<wbr>wave_vector_seq_id<br> La1x1 La1 x 1<br> La1y1 La1 y 1<br> La1z1 La1 z 1<br> La1x2 La1 x 2<br> La1y2 La1 y 2<br> La1z2 La1 z 2<br> S2x1 S2 x 1<br> S2y1 S2 y 1<br> S2z1 S2 z 1<br> S2x2 S2 x 2<br> S2y2 S2 y 2<br> S2z2 S2 z 2<br><br> ### End of modulated structure second subsystem data ######<span class="HOEnZb"><font color="#888888"><br><br>-- <br><div class="m_8242678185185834466gmail_signature">T <a href="tel:+61%202%209717%209907" value="+61297179907" target="_blank">+61 (02) 9717 9907</a><br>F <a href="tel:+61%202%209717%203145" value="+61297173145" target="_blank">+61 (02) 9717 3145</a><br>M +61 (04) 0249 4148</div>
</font></span></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">T +61 (02) 9717 9907<br>F +61 (02) 9717 3145<br>M +61 (04) 0249 4148</div>
</div>