[Imgcif-l] [Fwd: RE: Reading CBF files from Python]
Herbert J. Bernstein
yaya at bernstein-plus-sons.com
Tue Jun 22 13:04:16 BST 2010
Puzzling. This image doe copy fine with cif2cbf. I'll work on it
after the eye doctor. -- Herbert
=====================================================
Herbert J. Bernstein, Professor of Computer Science
Dowling College, Kramer Science Center, KSC 121
Idle Hour Blvd, Oakdale, NY, 11769
+1-631-244-3035
yaya at dowling.edu
=====================================================
On Tue, 22 Jun 2010, Jon Wright wrote:
> Return-path: <imgcif-l-bounces at iucr.org>
> Received: from galena.iucr.org ([192.70.242.40])
> by hermes.dowling.edu with ESMTP; Tue, 22 Jun 2010 07:55:40 -0400
> Received: from ASSP.nospam (galena.iucr.org [127.0.0.1])
> by galena.iucr.org (Postfix) with ESMTP
> id 08ECB532E3; Tue, 22 Jun 2010 12:55:18 +0100 (BST)
> Received: from dynhost1.iucr.org ([192.70.242.92] helo=dynhost1.iucr.org)
> with IPv4:25 by ASSP.nospam; 22 Jun 2010 12:55:18 +0100
> Received: from dynhost1.iucr.org (dynhost1.iucr.org [127.0.0.1])
> by dynhost1.iucr.org (Postfix) with ESMTP
> id D26DF9B59A; Tue, 22 Jun 2010 12:55:17 +0100 (BST)
> X-Original-To: imgcif-l at dynhost1.iucr.org
> Delivered-To: imgcif-l at dynhost1.iucr.org
> Received: from galena.iucr.org (galena.iucr.org [192.70.242.40])
> by dynhost1.iucr.org (Postfix) with ESMTP id 8ABF59B59A
> for <imgcif-l at dynhost1.iucr.org>; Tue, 22 Jun 2010 12:55:15 +0100
> (BST)
> Received: by galena.iucr.org (Postfix)
> id 5D27A532F1; Tue, 22 Jun 2010 12:55:15 +0100 (BST)
> Delivered-To: imgcif-l at iucr.org
> Received: from ASSP.nospam (galena.iucr.org [127.0.0.1])
> by galena.iucr.org (Postfix) with ESMTP id 0C1C4532E3
> for <imgcif-l at iucr.org>; Tue, 22 Jun 2010 12:55:15 +0100 (BST)
> Received: from gtwin2.esrf.fr ([193.49.43.102] helo=gtwin2.esrf.fr) with
> IPv4:25 by ASSP.nospam; 22 Jun 2010 12:55:14 +0100
> Received: (from uucp at localhost) by gtwin2.esrf.fr (1.0.0) id o5MBtCNf335843;
> Tue, 22 Jun 2010 13:55:12 +0200 (MEST)
> Received: from mailsrv.esrf.fr(160.103.2.161) by gtwout2 via smap (V1.3)
> id tmp335837; Tue Jun 22 13:55:09 2010
> Received: from novirus.esrf.fr (novirus.esrf.fr [160.103.2.175])
> by mailsrv.esrf.fr (8.13.8+Sun/8.13.8) with ESMTP id o5MBt80K004948;
> Tue, 22 Jun 2010 13:55:08 +0200 (MEST)
> Received: from [127.0.0.1] (pcmedstudent2.esrf.fr [160.103.106.183])
> by novirus.esrf.fr (8.13.1/8.13.1) with ESMTP id o5MBnP9r032172;
> Tue, 22 Jun 2010 13:49:29 +0200
> Message-ID: <4C20A497.3080200 at esrf.fr>
> Date: Tue, 22 Jun 2010 13:55:03 +0200
> From: Jon Wright <wright at esrf.fr>
> User-Agent: Thunderbird 2.0.0.24 (Windows/20100228)
> MIME-Version: 1.0
> To: "Herbert J. Bernstein" <yaya at bernstein-plus-sons.com>,
> Graeme.Winter at Diamond.ac.uk
> Content-Type: multipart/mixed;
> boundary="------------090102040500010601060009"
> X-Assp-Delay: not delayed (wright at esrf.fr on whitelistdb); 22 Jun 2010
> 12:55:15 +0100
> X-Assp-Whitelisted: Yes (wright at esrf.fr on whitelistdb)
> X-Assp-Envelope-From: wright at esrf.fr
> X-Assp-Passing: wright at esrf.fr on whitelistdb
> X-Assp-ID: ASSP.nospam ()
> X-Assp-Version: 1.7.5.2(1.0.02)
> Cc: The Crystallographic Binary File and its imgCIF application to image data
> <imgcif-l at iucr.org>
> Subject: [Imgcif-l] [Fwd: RE: Reading CBF files from Python]
> X-BeenThere: imgcif-l at iucr.org
> X-Mailman-Version: 2.1.9
> Precedence: list
> Reply-To: The Crystallographic Binary File and its imgCIF application to
> image data <imgcif-l at iucr.org>
> List-Id: The Crystallographic Binary File and its imgCIF application to image
> data <imgcif-l.iucr.org>
> List-Unsubscribe: <http://scripts.iucr.org/mailman/listinfo/imgcif-l>,
> <mailto:imgcif-l-request at iucr.org?subject=unsubscribe>
> List-Archive: <http://scripts.iucr.org/pipermail/imgcif-l>
> List-Post: <mailto:imgcif-l at iucr.org>
> List-Help: <mailto:imgcif-l-request at iucr.org?subject=help>
> List-Subscribe: <http://scripts.iucr.org/mailman/listinfo/imgcif-l>,
> <mailto:imgcif-l-request at iucr.org?subject=subscribe>
> Sender: imgcif-l-bounces at iucr.org
> Errors-To: imgcif-l-bounces at iucr.org
> X-Assp-Passing: acceptAllMail
> X-Assp-ID: ASSP.nospam ()
> X-Assp-Version: 1.7.5.2(1.0.02)
>
> This is a multi-part message in MIME format.
> --------------090102040500010601060009
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> Content-Transfer-Encoding: 7bit
>
> Hi Herbert, Graeme,
>
> I had a quick look with the pycbf bindings and ran into a problem with the
> example image (http://www.ccp4.ac.uk/xia/image.cbf). I get a slew of warnings
> and finally an error coming from CBFlib:
>
> coral24:~/cbflib_jun2010/CBFlib_0.7.9/pycbf % python tst.py image.cbf 2>&1 |
> more
> CBFlib: warning input line 26 (1) -- invalid character
> CBFlib: warning input line 26 (2) -- invalid character
> CBFlib: warning input line 26 (3) -- invalid character
> ..
> CBFlib: warning input line 26 (1978) -- invalid character
> CBFlib: warning input line 26 (1979) -- invalid character
> CBFlib: error input line 26 (1) -- value without tag
>
>
> The code in python is essentially:
>
> import pycbf
> o = pycbf.cbf_handle_struct()
> o.read_file("image.cbf", pycbf.MSG_DIGEST)
>
> In c this translates into something like the following, and gives the same
> output:
>
> #include "cbf_simple.h"
> #include <stdlib.h>
> #include <stdio.h>
> int main (int argc, char *argv [])
> {
> FILE *in;
> cbf_handle cbf;
> if (!(in = fopen (argv[1], "rb"))) {
> fprintf (stderr,"Couldn't open the input cbf file %s\n", argv[1]);
> exit (1);
> }
> cbf_failnez (cbf_make_handle (&cbf));
> cbf_failnez (cbf_read_widefile (cbf, in, MSG_DIGEST));
> }
>
>
> A workaround is to copy the binary section into a valid file and then read
> that file, but I don't think it is what Graeme is after. Code to do that is
> attached in case it is useful in the meantime.
>
> Sorry I can't be more useful,
>
> Jon
>
>
>
>
>
>
> -------- Original Message --------
> Subject: RE: [Imgcif-l] Reading CBF files from Python
> Date: Mon, 21 Jun 2010 14:20:38 +0100
> From: <Graeme.Winter at Diamond.ac.uk>
> To: <wright at esrf.fr>
> CC: <Graeme.Winter at Diamond.ac.uk>
> References:
> <4854F2500EA8C4478A508D2D92973E5206D6B5F9 at EXCHANGE25.fed.cclrc.ac.uk>
> <4C1F2B5C.30400 at esrf.fr><4854F2500EA8C4478A508D2D92973E5206D6B5FA at EXCHANGE25.fed.cclrc.ac.uk>
> <4C1F3784.6060107 at esrf.fr>
>
> Hi Jon,
>
> Thanks - I think that the problem is more than this image does not
> behave like a real cbf, though having some similarities. However I tried
> building pycbf as you suggest and the build failed - I think the
> configuration is not right in the way the cctbx folks use cbflib.
>
> I have attached the image (picture would be a better word) - when I try
> to load it in to cctbx using the MiniCBFImage loader it fails indicating
> I think that it is poorly structured.
>
> I am sure that there is code in cbflib which will do exactly what I
> need, the only problem is gaining access to it!
>
> Best wishes,
>
> Graeme
>
>
>
> -----Original Message-----
> From: imgcif-l-bounces at iucr.org [mailto:imgcif-l-bounces at iucr.org] On
> Behalf Of Jon Wright
> Sent: 21 June 2010 10:57
> To: The Crystallographic Binary File and its imgCIF application to image
> data
> Subject: Re: [Imgcif-l] Reading CBF files from Python
>
> Hi Graeme,
>
> To use the binding the wrapper needs to be compiled, which should just
> mean typing "python setup.py build" in the pycbf subfolder. Then you
> just need to drop pycbf.py and _pycbf.so somewhere on your python path
> (import sys; print sys.path). The binding runs entirely via calls to
> CBFlib, so if there is something useful in there, it should be possible
> to include it. Herbert can probably help more for knowing where to look.
>
> If you can send an example image off-list I'll try to take a look
>
> Best
>
> Jon
>
> Graeme.Winter at Diamond.ac.uk wrote:
>> Hi Jon,
>>
>> Thanks - I see that this is bundled with cctbx and Labelit, which was
>> awesome. However it seems that the modules from pycbf aren't available
>
>> as standard :o( - I am slightly hamstrung by the fact that I use the cctbx
>> python, so having access to other python extensions presents a problem.
>> However I *really need* cctbx :o) - xia2 won't work without it now.
>>
>> My rate limiting step right now is the reading and writing of the byte
>
>> offset compression. I am going to guess that if I messed with the python
>> paths and such I should be able to find your implementation. Do
>
>> you have a method in there which will allow me to just pack and unpack
>
>> the byte_offset compressed regions?
>>
>> This is the data in question:
>>
>> ###CBF: Version July 2008 generated by XDS
>>
>> data_FRAME.cbf
>>
>> _array_data.header_convention "XDS special"
>> _array_data.header_contents
>> ;
>> ;
>>
>> _array_data.data
>> ;
>> --CIF-BINARY-FORMAT-SECTION--
>> Content-Type: application/octet-stream;
>> conversions="x-CBF_BYTE_OFFSET"
>> Content-Transfer-Encoding: BINARY
>> X-Binary-Size: 9450352
>> X-Binary-ID: 1
>> X-Binary-Element-Type: "signed 32-bit integer"
>> X-Binary-Element-Byte-Order: LITTLE_ENDIAN
>> X-Binary-Number-of-Elements: 9437184
>> X-Binary-Size-Fastest-Dimension: 3072
>> X-Binary-Size-Second-Dimension: 3072
>> (massive binary blob)
>>
>> The bit which is causing me problems is that the cbflib adaptor raises
>
>> an exception as most of the "proper" cbf stuff (i.e. the CIF) is not
>> present. Erk.
>>
>> Any suggestions will be pounced apon!
>>
>> Many thanks,
>>
>> Graeme
>>
>> -----Original Message-----
>> From: imgcif-l-bounces at iucr.org [mailto:imgcif-l-bounces at iucr.org] On
>> Behalf Of Jon Wright
>> Sent: 21 June 2010 10:06
>> To: The Crystallographic Binary File and its imgCIF application to image
>> data
>> Subject: Re: [Imgcif-l] Reading CBF files from Python
>>
>> Hi Graeme,
>>
>> There is a swig based python binding included with CBFlib in the pycbf
>
>> subfolder. It should work for reading files (see pycbf_test1.py).
>> Further work was needed for writing. Let me know if you run into problems.
>> Improvements are most welcome.
>>
>> Best
>>
>> Jon
>>
>> Graeme.Winter at Diamond.ac.uk wrote:
>>> Hi Folks,
>>> Does anyone know if CBF images can be read from Python? I coded up
>>> something in Python to do this but it's a spot slow to say the least.
>>> My interest is partly in reading the cbf "image" files calculated by XDS
>>> to explain what it has been up to. I could code something up in 'c' or a
>>> bridge to CBFlib but that would be hardish to maintain, so a
>
>>> solution using existing Python libraries would suit me better.
>>> I looked quickly at PyCIFrw but this appears to be mostly about the CIF
>>> side of things rather than the MIME encoded chunks. Please correct
>>
>>> me if I am wrong. I also tried iotbx.detectors.cbf.CBFImage but hit an
>>
>>> exception which probably relates to this being more of a picture than
>
>>> a diffraction image.
>>> Thanks,
>>> Graeme
>>> Dr. Graeme Winter
>>> Software and MX Support Scientist
>>> Diamond Light Source
>>> +44 1235 778091 (work)
>>> +44 7786 662784 (work mobile)
>>>
>>>
>>>
>>
>> _______________________________________________
>> imgcif-l mailing list
>> imgcif-l at iucr.org
>> http://scripts.iucr.org/mailman/listinfo/imgcif-l
>>
>
> _______________________________________________
> imgcif-l mailing list
> imgcif-l at iucr.org
> http://scripts.iucr.org/mailman/listinfo/imgcif-l
>
>
>
> --
>
> This e-mail and any attachments may contain confidential, copyright and or
> privileged material, and are for the use of the intended addressee only. If
> you are not the intended addressee or an authorised recipient of the
> addressee please notify us of receipt by returning the e-mail and do not use,
> copy, retain, distribute or disclose the information in or attached to the
> e-mail.
>
> Any opinions expressed within this e-mail are those of the individual and not
> necessarily of Diamond Light Source Ltd.
>
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any
> attachments are free from viruses and we cannot accept liability for any
> damage which you may sustain as a result of software viruses which may be
> transmitted in or with the message.
>
> Diamond Light Source Limited (company no. 4375679). Registered in England and
> Wales with its registered office at Diamond House, Harwell Science and
> Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
>
>
>
>
>
>
>
>
>
>
>
> --------------090102040500010601060009
> Content-Type: text/plain;
> name="read_graeme.py"
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline;
> filename="read_graeme.py"
>
>
> import pycbf, sys, tempfile, time, numpy as np
>
> def fix_im( barecbf, template, tempname ):
> temp = open(tempname, "wb")
> tag = "--CIF-BINARY-FORMAT-SECTION--"
> good = open(template,"rb").read()
> s = good.find(tag) + len(tag)
> temp.write( good[:s] )
> eg = s + good[s:].find("--CIF")
> bin = open(barecbf, "rb").read()
> s = bin.find(tag)+len(tag)
> e = s + bin[s:].find("--CIF")
> temp.write( bin[s:e] )
> temp.write( good[ eg: ] )
> temp.flush()
> temp.close()
>
>
>
>
> def read_binary(name ):
> object = pycbf.cbf_handle_struct()
> object.read_file(name,pycbf.MSG_DIGEST)
> object.rewind_datablock()
> object.select_datablock(0)
> object.rewind_category()
> categories = object.count_categories()
> for i in range(categories):
> object.select_category(i)
> category_name = object.category_name()
> rows=object.count_rows()
> cols = object.count_columns()
> loop=1
> object.rewind_column()
> while loop is not 0:
> column_name = object.column_name()
> try:
> object.next_column()
> except:
> break
> for j in range(rows):
> object.select_row(j)
> object.rewind_column()
> for k in range(cols):
> name=object.column_name()
> object.select_column(k)
> typeofvalue=object.get_typeofvalue()
> if typeofvalue.find("bnry") > -1:
> s=object.get_integerarray_as_string()
> ar = np.fromstring(s,np.int32)
> else:
> value=object.get_value()
> # print "Val:",value,i
> del(object)
> return ar
>
> start = time.time()
> fix_im(sys.argv[1], sys.argv[2], sys.argv[3] )
> print "fixing",time.time()-start
> start = time.time()
> image = read_binary(sys.argv[3])
> print "reading",time.time()-start
> print image.max(), image.min(), image.mean()
>
> --------------090102040500010601060009
> Content-Type: text/plain; charset="us-ascii"
> MIME-Version: 1.0
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
>
> _______________________________________________
> imgcif-l mailing list
> imgcif-l at iucr.org
> http://scripts.iucr.org/mailman/listinfo/imgcif-l
>
> --------------090102040500010601060009--
>
>
More information about the imgcif-l
mailing list