_database.dataset_doi - any problems if this might be a DOI for raw data?

[James H]

Thanks for the feedback Horst. It was a bit more broad than I needed, but
you raise some interesting points.

We are making slow progress in CIF towards having all forms of data
properly described and incorporated into the CIF "scheme", regardless of
the format that they are preserved in.  One prong of this work has been to
ensure that the DDLm language used for data definition in CIF is not
format-specific, that is, data names defined in our dictionaries describe
data that might be stored in any format - but of course somebody still has
to actually do the specification that says "in files of this format you can
find CIF data item x_y_z at this location". Another prong of this work is
to rigorously allow for multi-data-block data (this is now done). Putting
those two prongs together means you can have some zip file or directory
full of files in various formats, and have the relationships between the
data stored in the various files rigorously specifiable by CIF
dictionaries. The missing work here is the specification (and software that
uses that specification) of where data name Y is found in format X (and
given typical conformance issues "...at facility Z".)

Another prong going in a bit of a different direction has been to define
"pointer" data names that describe how to interpret externally-stored image
data in CIF terms (see
https://github.com/yayahjb/cbflib/doc/cif_img_1.8.6.dic and search for
_array_data_external_data). This allows the bulky image data to be stored
outside the CIF file while the CIF file itself contains the metadata. A
couple of examples of this in action are in preparation, or you can have
fun with https://github.com/jamesrhester/ImgCIFHandler.jl which takes such
imgCIF files and loads the externally-located HDF5/CBF/ADSC image from
optionally tar/gzipped archives.

To circle back to your comments, DOIs that point to files that have no
properly (in the sense of CIF data names for the contents of the files)
defined roles are less useful than DOIs that can be resolved to an object
that has an exact slot in a CIF-based description of a data collection. The
latter can be transparently auto-processed with no human intervention.
Given that DOIs resolve to landing pages they are currently not that useful
for machine-based data processing, but perhaps DOI standards will evolve to
allow the actual data files to be specified in a DOI.

And finally, while the tsc files might be too bulky for storing in CIF
syntax, the contents can eventually be brought into the CIF data world.
Here are the steps:
1. Define CIF data names for the relevant quantities appearing in the
files. These would logically go in a new dictionary as NoSpherA2 refinement
is different to the refinement model of the core dictionary
2. Define where these data names are located in .tsc files

Step 2 is a brave new world unfortunately, as there are no general
machine-readable standards that I know of, or that we have developed, for
specifying where "something" is located in a file with arbitrary format -
if anyone reading this knows of such a thing, get in touch.  The
_array_data_external_data work I linked to above does contain some ideas
about how this might work in practice.

anyway, thanks again for your thoughts.
James.

On Fri, 27 May 2022 at 22:59, Horst Puschmann <horst.puschmann at gmail.com>
wrote:

> Hello James,
>
> I think providing a DOI in the CIF pointing at large amounts of data would
> be fantastic. I must say that I don't quite understand that wording here --
> what does the CSD or PDP have to do with it? Surely, all hkl data is now
> included in the CIF as standard -- but of course: it would also be
> excellent if the hkl could be handled via an 'hkl DOI'.
>
> I take 'raw data' to mean the diffraction images (frames) -- possibly
> together with a recipe for how they were processed. Having easy access to
> frames 'by default' would be fantastic.
>
> But there are other kinds of files, and some might be required to repeat
> the actual refinement. I am thinking of our own NoSpherA2 refinement, which
> requires a '.tsc' file -- a file which is clearly too large to be embedded
> in the CIF. A DOI for this kind of file would be really useful (right now,
> we just include the hkl and the exact parameters to repeat the generation
> of the '.tsc' file -- which is clearly not ideal.
>
> And then there might be a third kind of DOI -- one where people can
> deposit 'random' files -- like videos, images, descriptions of special
> setup etc, scripts etc.
>
> I am not sure whether this is the sort of feedback you were looking for,
> but there you are.
>
> Greetings
> Horst
>
> On Fri, 27 May 2022 at 04:09, James H via coreDMG <coredmg at iucr.org>
> wrote:
>
>> (cross-posted to cif-developers and core DMG, apologies for cross-posting)
>>
>> Dear CIF Developers and core DMG,
>>
>> IUCr Journals are looking at using _database.dataset_doi to indicate the
>> DOI of a raw data set associated with a data block. The meaning of
>> "dataset" is not clear here, for example, it might have been intended to
>> refer to hkl listings.
>>
>> So, please give feedback on any problems your software/database might
>> encounter if this DOI might resolve to a raw dataset.
>>
>> The current definition:
>>
>> " The digital object identifier (DOI) registered to identify
>>     a data set publication associated with the structure
>>     described in the current data block. This should be used
>>     for a dataset obtained from a curated database such as
>>     CSD or PDB. "
>>
>> thanks,
>> James.
>> --
>> T +61 (02) 9717 9907
>> F +61 (02) 9717 3145
>> M +61 (04) 0249 4148
>> _______________________________________________
>> coreDMG mailing list
>> coreDMG at iucr.org
>> http://mailman.iucr.org/cgi-bin/mailman/listinfo/coredmg
>>
>

-- 
T +61 (02) 9717 9907
F +61 (02) 9717 3145
M +61 (04) 0249 4148
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.iucr.org/pipermail/coredmg/attachments/20220601/1e02e1c9/attachment.htm>