The bqb File Format
In modern science (both experimental and computational), the size of the raw data can become a significant issue — often referred to as “Big Data”. The bqb format, which we have developed in 2018, enables lossless compression of simulation trajectories – both of atom positions and volumetric data. The use of modern algorithms (polynomial least-squares extrapolation in space and time, Hilbert curve re-ordering, Burrows–Wheeler transformation, multi-table canonical Huffman encoding) enables a very high lossless compression ratio of up to 35 : 1 for volumetric data trajectories, and up to 20 : 1 for standard atom position trajectories. This is by far the smallest lossless storage format for those data among all competitors (see article). The code is available as free software, and also implemented directly in TRAVIS and CP2k.
"An Efficient Lossless Compression Algorithm for Trajectories
of Atom Positions and Volumetric Data",
J. Chem. Inf. Model. 2018, 58 (10), 2092–2107, DOI 10.1021/acs.jcim.8b00501
See the &DFT/&PRINT/&E_DENSITY_BQB manual section.
Note: If you compile CP2k 8.1 or newer via the Toolchain script, these features will automatically be included by default, and the library will automatically be downloaded and compiled in the process. If you prefer not to use the Toolchain, you need to download and compile the libvori source code from here.
The bqbtool is a command-line utility for working with the bqb file format. It enables to compress and decompress trajectories.
The bqbtool is written in C++, contains around 41 000 lines of source code, and is distributed as free software under the GNU LGPL v3 license.
It does not require any external library and is platform independent. Make sure to check the manual before trying it out.
|Dec 08 2020||Fixed a few bugs.|
|May 04 2020||Fixed some bugs, including one which affected the compression ratio when compressing position trajectories.|
|Jan 01 2019||Fixed a (seldomly occurring) bug with handling BQB file index frames.|
|Oct 01 2018||Initial release of bqbtool.|
|File||Type / Size||Last Changed|
|bqbtool source code||.tar.gz, 890 kiB||Dec 08 2020|
|bqbtool Windows executable (64 bit, preferred version)||.zip, 1.1 MiB||Dec 08 2020|
|bqbtool Windows executable (32 bit, compatibility version)||.zip, 1.1 MiB||Dec 08 2020|
|bqbtool manual||.pdf, 763 kiB||Oct 01 2018|
There is also a new TRAVIS release out which offers the full functionality of the bqbtool. Simply use the TRAVIS executable as if it was the bqbtool. Apart from that, TRAVIS can now directly read and analyze compressed BQB trajectories.
Please also see our new tutorial on vibrational spectroscopy.
We are also planning to publish a C++ library ("libbqb") which can be used to equip any software package with support for reading and writing compressed bqb files. Then, simulation programs such as CP2k or LAMMPS might directly write compressed trajectories, while visualization programs such as VMD could directly read the compressed files without prior decompression. It will take a few more months until the libbqb is ready for its first public release. Please stay tuned.