Logo

The bqb File Format

bqb Format

In modern science (both experimental and computational), the size of the raw data can become a significant issue — often referred to as “Big Data”. The bqb format, which we have developed in 2018, enables loss­less compression of simulation trajectories – both of atom positions and volumetric data. The use of mod­ern algorithms (polynomial least-squares extrapolation in space and time, Hilbert curve re-ordering, Burrows–Wheeler transformation, multi-table canonical Huffman encoding) enables a very high lossless compression ratio of up to 35 : 1 for volumetric data trajectories, and up to 20 : 1 for standard atom position trajecto­ries. This is by far the smallest lossless storage format for those data among all competitors (see article). The code is available as free software, and also implemented directly in TRAVIS and CP2k.

M. Brehm, M. Thomas:
"An Efficient Lossless Compression Algorithm for Trajectories
of Atom Positions and Volumetric Data"
,
J. Chem. Inf. Model. 2018, 58 (10), 2092–2107, DOI 10.1021/acs.jcim.8b00501


During my guest visit at the University of Zurich, I have implemented support for the bqb format directly into the CP2k program package. Starting from CP2k release 8.1 (December 2020), it is possible to directly write the total electron density to bqb files during a CP2k simulation without having to resort to intermediary (huge) Gaussian Cube files. An updated version of the tutorial will follow soon.

See the &DFT/&PRINT/&E_DENSITY_BQB manual section.

Note: If you compile CP2k 8.1 or newer via the Toolchain script, these features will automatically be included by default, and the library will automatically be downloaded and compiled in the process. If you prefer not to use the Toolchain, you need to download and compile the libvori source code from here.


bqbtool

The bqbtool is a command-line utility for working with the bqb file format. It enables to compress and decompress trajectories. The bqbtool is written in C++, contains around 41 000 lines of source code, and is distributed as free software under the GNU LGPL v3 license. It does not require any external library and is platform independent. Make sure to check the manual before trying it out.

Revision History:

DateNotes
Dec 08 2020 Fixed a few bugs.
May 04 2020 Fixed some bugs, including one which affected the compression ratio when compressing position trajectories.
Jan 01 2019 Fixed a (seldomly occurring) bug with handling BQB file index frames.
Oct 01 2018 Initial release of bqbtool.

Please download:

FileType / SizeLast Changed
bqbtool source code .tar.gz, 890 kiB Dec 08 2020
bqbtool Windows executable (64 bit, preferred version) .zip, 1.1 MiB Dec 08 2020
bqbtool Windows executable (32 bit, compatibility version) .zip, 1.1 MiB Dec 08 2020
bqbtool manual .pdf, 763 kiB Oct 01 2018

There is also a new TRAVIS release out which offers the full functionality of the bqbtool. Simply use the TRAVIS executable as if it was the bqbtool. Apart from that, TRAVIS can now directly read and analyze compressed BQB trajectories.

Starting from release 8.1 (December 2020), CP2k is able to directly write the total electron density to the compressed bqb format.

Please also see our new tutorial on vibrational spectroscopy.


libbqb

We are also planning to publish a C++ library ("libbqb") which can be used to equip any software package with support for reading and writing compressed bqb files. Then, simulation programs such as CP2k or LAMMPS might directly write compressed trajectories, while visualization programs such as VMD could directly read the compressed files without prior decompression. It will take a few more months until the libbqb is ready for its first public release. Please stay tuned.