[Imgcif-l] [Fwd: RE: Reading CBF files from Python]
Jon Wright
wright at esrf.fr
Tue Jun 22 15:41:07 BST 2010
Thanks Herbert - perhaps the problem is that I am using 0.7.9, and also
see the error on cif2cbf. Will try upgrading.
Best
Jon
Herbert J. Bernstein wrote:
> Puzzling. This image doe copy fine with cif2cbf. I'll work on it
> after the eye doctor. -- Herbert
>
> =====================================================
> Herbert J. Bernstein, Professor of Computer Science
> Dowling College, Kramer Science Center, KSC 121
> Idle Hour Blvd, Oakdale, NY, 11769
>
> +1-631-244-3035
> yaya at dowling.edu
> =====================================================
>
> On Tue, 22 Jun 2010, Jon Wright wrote:
>
>>
>> Hi Herbert, Graeme,
>>
>> I had a quick look with the pycbf bindings and ran into a problem with the
>> example image (http://www.ccp4.ac.uk/xia/image.cbf). I get a slew of warnings
>> and finally an error coming from CBFlib:
>>
>> coral24:~/cbflib_jun2010/CBFlib_0.7.9/pycbf % python tst.py image.cbf 2>&1 |
>> more
>> CBFlib: warning input line 26 (1) -- invalid character
>> CBFlib: warning input line 26 (2) -- invalid character
>> CBFlib: warning input line 26 (3) -- invalid character
>> ..
>> CBFlib: warning input line 26 (1978) -- invalid character
>> CBFlib: warning input line 26 (1979) -- invalid character
>> CBFlib: error input line 26 (1) -- value without tag
>>
>>
>> The code in python is essentially:
>>
>> import pycbf
>> o = pycbf.cbf_handle_struct()
>> o.read_file("image.cbf", pycbf.MSG_DIGEST)
>>
>> In c this translates into something like the following, and gives the same
>> output:
>>
>> #include "cbf_simple.h"
>> #include <stdlib.h>
>> #include <stdio.h>
>> int main (int argc, char *argv [])
>> {
>> FILE *in;
>> cbf_handle cbf;
>> if (!(in = fopen (argv[1], "rb"))) {
>> fprintf (stderr,"Couldn't open the input cbf file %s\n", argv[1]);
>> exit (1);
>> }
>> cbf_failnez (cbf_make_handle (&cbf));
>> cbf_failnez (cbf_read_widefile (cbf, in, MSG_DIGEST));
>> }
>>
>>
>> A workaround is to copy the binary section into a valid file and then read
>> that file, but I don't think it is what Graeme is after. Code to do that is
>> attached in case it is useful in the meantime.
>>
>> Sorry I can't be more useful,
>>
>> Jon
>>
>>
>>
>>
>>
>>
>> -------- Original Message --------
>> Subject: RE: [Imgcif-l] Reading CBF files from Python
>> Date: Mon, 21 Jun 2010 14:20:38 +0100
>> From: <Graeme.Winter at Diamond.ac.uk>
>> To: <wright at esrf.fr>
>> CC: <Graeme.Winter at Diamond.ac.uk>
>> References:
>> <4854F2500EA8C4478A508D2D92973E5206D6B5F9 at EXCHANGE25.fed.cclrc.ac.uk>
>> <4C1F2B5C.30400 at esrf.fr><4854F2500EA8C4478A508D2D92973E5206D6B5FA at EXCHANGE25.fed.cclrc.ac.uk>
>> <4C1F3784.6060107 at esrf.fr>
>>
>> Hi Jon,
>>
>> Thanks - I think that the problem is more than this image does not
>> behave like a real cbf, though having some similarities. However I tried
>> building pycbf as you suggest and the build failed - I think the
>> configuration is not right in the way the cctbx folks use cbflib.
>>
>> I have attached the image (picture would be a better word) - when I try
>> to load it in to cctbx using the MiniCBFImage loader it fails indicating
>> I think that it is poorly structured.
>>
>> I am sure that there is code in cbflib which will do exactly what I
>> need, the only problem is gaining access to it!
>>
>> Best wishes,
>>
>> Graeme
>>
>>
>>
>> -----Original Message-----
>> From: imgcif-l-bounces at iucr.org [mailto:imgcif-l-bounces at iucr.org] On
>> Behalf Of Jon Wright
>> Sent: 21 June 2010 10:57
>> To: The Crystallographic Binary File and its imgCIF application to image
>> data
>> Subject: Re: [Imgcif-l] Reading CBF files from Python
>>
>> Hi Graeme,
>>
>> To use the binding the wrapper needs to be compiled, which should just
>> mean typing "python setup.py build" in the pycbf subfolder. Then you
>> just need to drop pycbf.py and _pycbf.so somewhere on your python path
>> (import sys; print sys.path). The binding runs entirely via calls to
>> CBFlib, so if there is something useful in there, it should be possible
>> to include it. Herbert can probably help more for knowing where to look.
>>
>> If you can send an example image off-list I'll try to take a look
>>
>> Best
>>
>> Jon
>>
>> Graeme.Winter at Diamond.ac.uk wrote:
>>> Hi Jon,
>>>
>>> Thanks - I see that this is bundled with cctbx and Labelit, which was
>>> awesome. However it seems that the modules from pycbf aren't available
>>> as standard :o( - I am slightly hamstrung by the fact that I use the cctbx
>>> python, so having access to other python extensions presents a problem.
>>> However I *really need* cctbx :o) - xia2 won't work without it now.
>>>
>>> My rate limiting step right now is the reading and writing of the byte
>>> offset compression. I am going to guess that if I messed with the python
>>> paths and such I should be able to find your implementation. Do
>>> you have a method in there which will allow me to just pack and unpack
>>> the byte_offset compressed regions?
>>>
>>> This is the data in question:
>>>
>>> ###CBF: Version July 2008 generated by XDS
>>>
>>> data_FRAME.cbf
>>>
>>> _array_data.header_convention "XDS special"
>>> _array_data.header_contents
>>> ;
>>> ;
>>>
>>> _array_data.data
>>> ;
>>> --CIF-BINARY-FORMAT-SECTION--
>>> Content-Type: application/octet-stream;
>>> conversions="x-CBF_BYTE_OFFSET"
>>> Content-Transfer-Encoding: BINARY
>>> X-Binary-Size: 9450352
>>> X-Binary-ID: 1
>>> X-Binary-Element-Type: "signed 32-bit integer"
>>> X-Binary-Element-Byte-Order: LITTLE_ENDIAN
>>> X-Binary-Number-of-Elements: 9437184
>>> X-Binary-Size-Fastest-Dimension: 3072
>>> X-Binary-Size-Second-Dimension: 3072
>>> (massive binary blob)
>>>
>>> The bit which is causing me problems is that the cbflib adaptor raises
>>> an exception as most of the "proper" cbf stuff (i.e. the CIF) is not
>>> present. Erk.
>>>
>>> Any suggestions will be pounced apon!
>>>
>>> Many thanks,
>>>
>>> Graeme
>>>
>>> -----Original Message-----
>>> From: imgcif-l-bounces at iucr.org [mailto:imgcif-l-bounces at iucr.org] On
>>> Behalf Of Jon Wright
>>> Sent: 21 June 2010 10:06
>>> To: The Crystallographic Binary File and its imgCIF application to image
>>> data
>>> Subject: Re: [Imgcif-l] Reading CBF files from Python
>>>
>>> Hi Graeme,
>>>
>>> There is a swig based python binding included with CBFlib in the pycbf
>>> subfolder. It should work for reading files (see pycbf_test1.py).
>>> Further work was needed for writing. Let me know if you run into problems.
>>> Improvements are most welcome.
>>>
>>> Best
>>>
>>> Jon
>>>
>>> Graeme.Winter at Diamond.ac.uk wrote:
>>>> Hi Folks,
>>>> Does anyone know if CBF images can be read from Python? I coded up
>>>> something in Python to do this but it's a spot slow to say the least.
>>>> My interest is partly in reading the cbf "image" files calculated by XDS
>>>> to explain what it has been up to. I could code something up in 'c' or a
>>>> bridge to CBFlib but that would be hardish to maintain, so a
>>>> solution using existing Python libraries would suit me better.
>>>> I looked quickly at PyCIFrw but this appears to be mostly about the CIF
>>>> side of things rather than the MIME encoded chunks. Please correct
>>>> me if I am wrong. I also tried iotbx.detectors.cbf.CBFImage but hit an
>>>> exception which probably relates to this being more of a picture than
>>>> a diffraction image.
>>>> Thanks,
>>>> Graeme
>>>> Dr. Graeme Winter
>>>> Software and MX Support Scientist
>>>> Diamond Light Source
>>>> +44 1235 778091 (work)
>>>> +44 7786 662784 (work mobile)
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> imgcif-l mailing list
>>> imgcif-l at iucr.org
>>> http://scripts.iucr.org/mailman/listinfo/imgcif-l
>>>
>> _______________________________________________
>> imgcif-l mailing list
>> imgcif-l at iucr.org
>> http://scripts.iucr.org/mailman/listinfo/imgcif-l
>>
>>
>>
>> --
>>
>> This e-mail and any attachments may contain confidential, copyright and or
>> privileged material, and are for the use of the intended addressee only. If
>> you are not the intended addressee or an authorised recipient of the
>> addressee please notify us of receipt by returning the e-mail and do not use,
>> copy, retain, distribute or disclose the information in or attached to the
>> e-mail.
>>
>> Any opinions expressed within this e-mail are those of the individual and not
>> necessarily of Diamond Light Source Ltd.
>>
>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
>> attachments are free from viruses and we cannot accept liability for any
>> damage which you may sustain as a result of software viruses which may be
>> transmitted in or with the message.
>>
>> Diamond Light Source Limited (company no. 4375679). Registered in England and
>> Wales with its registered office at Diamond House, Harwell Science and
>> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --------------090102040500010601060009
>> Content-Type: text/plain;
>> name="read_graeme.py"
>> Content-Transfer-Encoding: 7bit
>> Content-Disposition: inline;
>> filename="read_graeme.py"
>>
>>
>> import pycbf, sys, tempfile, time, numpy as np
>>
>> def fix_im( barecbf, template, tempname ):
>> temp = open(tempname, "wb")
>> tag = "--CIF-BINARY-FORMAT-SECTION--"
>> good = open(template,"rb").read()
>> s = good.find(tag) + len(tag)
>> temp.write( good[:s] )
>> eg = s + good[s:].find("--CIF")
>> bin = open(barecbf, "rb").read()
>> s = bin.find(tag)+len(tag)
>> e = s + bin[s:].find("--CIF")
>> temp.write( bin[s:e] )
>> temp.write( good[ eg: ] )
>> temp.flush()
>> temp.close()
>>
>>
>>
>>
>> def read_binary(name ):
>> object = pycbf.cbf_handle_struct()
>> object.read_file(name,pycbf.MSG_DIGEST)
>> object.rewind_datablock()
>> object.select_datablock(0)
>> object.rewind_category()
>> categories = object.count_categories()
>> for i in range(categories):
>> object.select_category(i)
>> category_name = object.category_name()
>> rows=object.count_rows()
>> cols = object.count_columns()
>> loop=1
>> object.rewind_column()
>> while loop is not 0:
>> column_name = object.column_name()
>> try:
>> object.next_column()
>> except:
>> break
>> for j in range(rows):
>> object.select_row(j)
>> object.rewind_column()
>> for k in range(cols):
>> name=object.column_name()
>> object.select_column(k)
>> typeofvalue=object.get_typeofvalue()
>> if typeofvalue.find("bnry") > -1:
>> s=object.get_integerarray_as_string()
>> ar = np.fromstring(s,np.int32)
>> else:
>> value=object.get_value()
>> # print "Val:",value,i
>> del(object)
>> return ar
>>
>> start = time.time()
>> fix_im(sys.argv[1], sys.argv[2], sys.argv[3] )
>> print "fixing",time.time()-start
>> start = time.time()
>> image = read_binary(sys.argv[3])
>> print "reading",time.time()-start
>> print image.max(), image.min(), image.mean()
>>
>> --------------090102040500010601060009
>> Content-Type: text/plain; charset="us-ascii"
>> MIME-Version: 1.0
>> Content-Transfer-Encoding: 7bit
>> Content-Disposition: inline
>>
>> _______________________________________________
>> imgcif-l mailing list
>> imgcif-l at iucr.org
>> http://scripts.iucr.org/mailman/listinfo/imgcif-l
>>
>> --------------090102040500010601060009--
>>
>>
> _______________________________________________
> imgcif-l mailing list
> imgcif-l at iucr.org
> http://scripts.iucr.org/mailman/listinfo/imgcif-l
More information about the imgcif-l
mailing list