Fwd: Re: DDLm, dREL, images and NeXus

Thu Dec 11 22:53:05 GMT 2008

I'll say more later, when other have had their say, but Doug's
suggestion of simply mapping the data needs a prompt response:
especially when working with 3D data, or even when working
witn 2D data in a 3D lab coordinate frame, it is very hard,
and perhaps infeasible, to specify the complete mapping without
the ability to specify methods, e.g. when working with multiple
coordinate frames or different handedness.  Life is mich simpler
when you can specify the algorithm for going from one to another.
A more subtle need for algorithms is in specifying the mappings
between a failry well normalized database dump (e.g. imgCIF)
and a rather denormalized format (e.g. NeXus).

   -- HJB

=====================================================
  Herbert J. Bernstein, Professor of Computer Science
    Dowling College, Kramer Science Center, KSC 121
         Idle Hour Blvd, Oakdale, NY, 11769

                  +1-631-244-3035
                  yaya at dowling.edu
=====================================================

On Fri, 12 Dec 2008, Doug wrote:

>
> I was under the impression the dREL use case was primarily for
> calculation and validation methodologies embedded within a
> relational database. It was a "fit for purpose" language that
> didn't include any external IO routines, although they were added to
> the dREL shell interpreter for convenience.
> Consequently conversion of CIF content to NeXus/HDF5 seems way out of
> language scope to me. I am sure if the designers intended
> a general purpose programming language to be used, then they
> would have embedded python in its totality and been done with it.
>
>
> Personally I see dREL as a promissing technology in the way that
> fusion reactors have been promissing for the past 50 years, and
> probably the next 50 too. Consequently I would be hesitant
> to base a technology around waiting for dREL to come of age.
>
>
> What would be more useful (from a search and archival perspective)
> would be something like a standardised mapping from CIF names to NeXus
> NXDetector.NXdata names and vice-versa - for data values that don't map
> onto already defined NeXus elements.
>
> It would be useful also to map those into the sample_parameter,
> dataset_parameter and datafile_parameters of the STFC's
> ICAT metadata catalogue. And even the metadata attributes of the TARDIS
> spec and the RDF of eCrystals. If that mapping was in CIF or some other
> format, then everyone could write their own conversion tools in their
> language of choice  (and then use NXTranslate? to convert to HDF5).
> Just for the record, there a couple of non-authoritative, kludged
> pseudo-schema's here:
>   http://cima.chem.usyd.edu.au:8095/schemaFusion/html/index.html
> but they are only intelligible if you view them with firefox.
>
> Actually, probably it would be nice if all the various diffractometer
> company frame header parameters could be mapped to a standard as well.
>
>
> Herbert, I believe you have already written an imageCIF to NeXus (XML?)
> conversion tool and I would be interested to know which CIF items were
> mapped into which NeXus fields and whether the NeXus files hold
> multiple images or links to external filesets and what do you do with
> image parameters that don't fit exist within the imageCIF vocabulary.
> I understand that HDF5 has one of the most efficient structured file access
> protocols around, so presumably you would embed an image within the NeXus
> file?
>
> Or are there granularity issues? Should a complete imageCIF file be
> embedded as a single NX_CHAR/NX_BINARY record?
>
>
> As datasets are becoming larger over time, both in the number and size of
> images,the simple act of looking at files within directories can be
> troublesome, especially over remote networks.
> Pascal Calvat, the author of the JUX SRB browser, was suggesting to archive
> complete datasets in SRB repositories as a single very large file. I think
> he was actually refusing to handle more than 500 files in a folder/directory
> with JUX. But apparently SRB and iRODS have backends that can let you
> peer into such stored HDF5 files in any case, so I am not sure if his
> arguments still hold.
>
>
> Doug
>
>
>
> On Thu, 11 Dec 2008, Herbert J. Bernstein wrote:
>> Dear Colleagues,
>> There are many frameworks
>> to consider, but two that are very close to achieving the goal of becoming
>> inter-operable in the immediate future are imgCIF and NeXus.  What is
>> missing is a formal language within which to specify how to move between
>> them.
>>
>>    We could, of course, just come up with a verbal description of how
>> to move between imgCIF and NeXus and a couple of example conversion
>> programs written ad hoc in whatever language might come to mind.  However,
>> the effort being expended on dREL, the supporting language for DDLm,
>> suggests the possibility of building on dREL as a base to do this job
>> by extending dREL to have the capability of working with NeXus (dREL
>> is already capable of dealing with CIF).
>
>
>> James has made the counter
>> proposal of leaving dREL as just a CIF-specific language and keeping
>> the CIF-NeXus conversion algorithm specification as a matter for a
>> different language and/or API.  James has also suggested the further
>> step of stripping out the built-in functions from dREL and dealing with
>> just a very stable dREL language specification in one instance and a
>> perhaps evolving API (list of builtin functions available in dREL) on the
>> other:
>
>
>> "My comment at this stage would be that defining a coupling mechanism
>> between CIF and a given language is not a large task, due to the
>> simplicity of the CIF syntax, whereas adding lots of stuff to dREL
>> would be a serious task and has some important downsides (loss of
>> simplicity being an important one). Apropros the
>> simplicity of the coupling mechanism, I (predictably) quite like my
>> Python model of a CIF file as a hash table of CIF data block objects
>> indexed by datablock name, and the datablock objects are themselves
>> hash tables of strings/lists of strings indexed by dataname.  This
>> model would appear to translate pretty easily into most other
>> languages.  What then remains is some syntactic sugar (the use of
>> square brackets to do key-based lookup is nice in dREL) which can be
>> replaced in another language by a few standard methods."
>>
>> There was a lot more to the discussion, but let us try to settle a
>> direction:
>>
>> Should we be trying to extend dREL to support more than just CIF,
>> specifically NeXus, making something we might call dREL++, or should the
>> language for this broader task be something distinct from dREL with a
>> distinct name.  In practice, in either case, I suspect all of this will be
>> built on a python base, or something similar, as James suggests, but
>> names do matter,
>>
>> Comments please.
>>
>> Regards,
>>    Herbert
>>
>> =====================================================
>>   Herbert J. Bernstein, Professor of Computer Science
>>     Dowling College, Kramer Science Center, KSC 121
>>          Idle Hour Blvd, Oakdale, NY, 11769
>>
>>                   +1-631-244-3035
>>                   yaya at dowling.edu
>> =====================================================
>> _______________________________________________
>> comcifs mailing list
>> comcifs at iucr.org
>> http://scripts.iucr.org/mailman/listinfo/comcifs
>
> -------------------------------------------------------
>