DDLm, dREL, images and NeXus

James Hester jamesrhester at gmail.com
Fri Dec 12 04:58:31 GMT 2008

Let me flesh out a proposal in some detail, so that holes can be picked in

First, an overall view.

At the moment, a DDLm/dREL engine is initialised with a set of DDLm
dictionaries.  When passed a CIF instance, it will return values of any
datanames that are contained in the CIF instance or that it is capable of
calculating from those datanames that are already in the instance.  Now,
what I envision as a 'translating' DDLm engine is initialised as before with
the standard DDLm dictionaries, but also with two further dictionaries: a
'NeXuS dictionary' and a 'translation dictionary' (contents of these
explained later).  Finally, it requires a 'NeXuS plugin'. Now, when passed a
CIF instance the DDLm engine works as before.  When passed a NeXuS instance,
it returns values of CIF datanames that it can calculate.

Now for an explanation of these various extra bits.

1.  The 'NeXuS' dictionary is just another DDLm dictionary.  It contains
definitions for datanames using a CIF namespace: e.g. _nexus.slit_height.
The linkage to a NeXuS file is accomplished using a set of new DDLm
attributes, which work like the current 'xref' attributes: in the header
section of this 'NeXuS' dictionary file the various versions of the NeXuS
standard are assigned a short code in a loop.  Each of the definitions in
the body of the dictionary then contains two new DDLm attributes:
_alien.code (referencing the version of the standard in the header) and
_alien.location (where to find the dataname).  The syntax of the value of
_alien.location might be borrowed from, for example, XPath in the case of

The data definitions containing _alien.location attributes could be
considered 'raw' NeXuS data, which may not map easily onto CIF datanames.
Therefore, this dictionary could contain further DDLm definitions of
dataitems (still in the CIF 'nexus' namespace) which contained dREL methods
for manipulating the raw datanames into something that mapped more directly
into CIF.  This is where one might foresee adding a few more builtin
functions to dREL to ease e.g. image processing.

Note also that one might, instead of using an '_alien.location' DDLm
attribute, define a new dREL builtin (like 'nexus_locate(string)'), and
provide a simple dREL expression calling this builtin.

2. The meaning of the 'NeXuS' plugin to the DDLm engine is that it enables
the DDLm engine to understand the '_alien.location' attribute and use it to
return a dREL-compatible value.  It would also supply extra builtin
functions if necessary.

3. The 'translation dictionary' contains alternative definitions of items in
the standard CIF dictionaries, where the dREL methods used to derive those
items in the standard dictionaries are replaced by dREL manipulations of
dataitems defined in the 'NeXuS' dictionary.  When it is read into the
DDLm/dREL engine, these new dREL methods could either replace the old ones,
or be added as alternative methods of type 'translation'.  This then allows
lots of flexibility in terms of checking derived values in a NeXuS file
against those values derived via standard CIF dREL methods from primitive
values from that same NeXuS file.

So, in summary, in order to implement this scheme, one needs about 6 new
DDLm attributes and a few more builtin dREL functions.  The crucial work is
in designing the behaviour of the '_alien.location' attribute to properly
capture the information.

One thing to think about is the introduction of an opaque data type into
dREL, so that e.g. image data returned from a NeXuS file need not be
specified as an array of numbers, but simply as an object, which can only be
passed to a builtin function and not otherwise manipulated.

Advantages of this scheme:
(1) dREL simplicity is preserved (I think this is important);
(2) implemented as a modular addon to a normal DDLm engine;
(3) easily extensible to other data formats;

Here follows a simple example.  Note that a NeXuS file may contain multiple
'NXuser' groups, each of which may contain one or more names but only a
single affiliation.  By specifying that the corresponding CIF category is a
'List' category, it can be deduced by the NeXuS plugin that each name in the
NeXuS file is a new entry in the category loop.  I have accomplished
renormalisation by walking the tree and returning the items in the order
encountered.  The category key is used to determine when to duplicate an
item; so in this case, every time the key is encountered, the associated
value of the target item is returned.  Otherwise, there would be fewer
affiliations than names in cases where more than one name belongs to a
single affiliation.

There is no additional processing of the raw nexus datanames in this
The 'NeXuS' dictionary file:
 _dictionary.namespace       nexus
   nexus4        nexus        4.0     www.nexusformat.org
    _definition.class        List     #items go into a loop
    _category_key.generic   '_raw_author.user_name'  #one value per

    _category.id       raw_author
     _type.container    Single
    _type.contents     Text
    _alien.code        nexus4
    _alien.location   "NXuser:name" #renormalise by returning all names in
                                    #encountered when walking the tree

    _category.id       raw_author
    _type.container    Single
    _type.contents     Text
    _alien.code        nexus4
    _alien.location    "NXuser:affiliation" #renormalise by returning
                                            #each time name encountered in

The translation file
with aa as audit_author
with rn as nexus:raw_author
aa.name = rn.user_name

On Thu, Dec 11, 2008 at 11:32 AM, Herbert J. Bernstein <
yaya at bernstein-plus-sons.com> wrote:

> Dear Colleagues,
>   This is a distillation of an email conversation James Hester and I
> have been having since the Osaka meeting.  We both feel that it
> would be helpful if others were to join in and express their views.
> We have been discussing the interaction among DDLm, dREL, images and
> NeXus.  We agree on most points, and disagree on a few, and hope,
> by opening up the discussion, to arrive at a consensus.
>   What is driving this discussion is a need to understand how best to
> manage image data in the context of both imgCIF and NeXus, and to do
> so in a way that is consistent with the recent adoption of DDLm as
> the target framework for new work on CIF dictionaries.
>   It must be clearly understood that it is highly unlikely that a
> single standard will ever be adopted for crystallographic diffraction
> images, much less for the broader context of pixel-based data in
> structural biology.  The best we can hope for right now is to have
> some number of clearly defined image data frameworks, and agreed
> algorithms for conversion among them.  There are many frameworks
> to consider, but two that are very close to achieving the goal of becoming
> inter-operable in the immediate future are imgCIF and NeXus.  What is
> missing is a formal language within which to specify how to move between
> them.
>   We could, of course, just come up with a verbal description of how
> to move between imgCIF and NeXus and a couple of example conversion
> programs written ad hoc in whatever language might come to mind.  However,
> the effort being expended on dREL, the supporting language for DDLm,
> suggests the possibility of building on dREL as a base to do this job
> by extending dREL to have the capability of working with NeXus (dREL
> is already capable of dealing with CIF).  James has made the counter
> proposal of leaving dREL as just a CIF-specific language and keeping
> the CIF-NeXus conversion algorithm specification as a matter for a
> different language and/or API.  James has also suggested the further
> step of stripping out the built-in functions from dREL and dealing with
> just a very stable dREL language specification in one instance and a
> perhaps evolving API (list of builtin functions available in dREL) on the
> other:
> "My comment at this stage would be that defining a coupling mechanism
> between CIF and a given language is not a large task, due to the
> simplicity of the CIF syntax, whereas adding lots of stuff to dREL
> would be a serious task and has some important downsides (loss of
> simplicity being an important one). Apropros the
> simplicity of the coupling mechanism, I (predictably) quite like my
> Python model of a CIF file as a hash table of CIF data block objects
> indexed by datablock name, and the datablock objects are themselves
> hash tables of strings/lists of strings indexed by dataname.  This
> model would appear to translate pretty easily into most other
> languages.  What then remains is some syntactic sugar (the use of
> square brackets to do key-based lookup is nice in dREL) which can be
> replaced in another language by a few standard methods."
> There was a lot more to the discussion, but let us try to settle a
> direction:
> Should we be trying to extend dREL to support more than just CIF,
> specifically NeXus, making something we might call dREL++, or should the
> language for this broader task be something distinct from dREL with a
> distinct name.  In practice, in either case, I suspect all of this will be
> built on a python base, or something similar, as James suggests, but
> names do matter,
> Comments please.
> Regards,
>   Herbert
> =====================================================
>  Herbert J. Bernstein, Professor of Computer Science
>    Dowling College, Kramer Science Center, KSC 121
>         Idle Hour Blvd, Oakdale, NY, 11769
>                  +1-631-244-3035
>                  yaya at dowling.edu
> =====================================================
> _______________________________________________
> comcifs mailing list
> comcifs at iucr.org
> http://scripts.iucr.org/mailman/listinfo/comcifs

T +61 (02) 9717 9907
F +61 (02) 9717 3145
M +61 (04) 0249 4148
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://scripts.iucr.org/pipermail/comcifs/attachments/20081212/90b10660/attachment-0001.html 

More information about the comcifs mailing list