[gmx-users] why does trjcat take so much memory?

[chris.neale at utoronto.ca]

Thank you for the insight Mark.

To get it on the record, this is fairly annoying for trjcat (not  
trjconv). If anyone else has run into this, then please do post  
because a memory efficient cat mechanism will only be coded if there  
is sufficient interest.

But again, thanks for your good answer Mark, probably you are correct.

Chris.

----- Original Message -----
From: chris.neale at utoronto.ca
Date: Wednesday, October 13, 2010 10:55
Subject: [gmx-users] why does trjcat take so much memory?
To: gmx-users at gromacs.org

[Hide Quoted Text]
I have 28,000 .xtc files, each having a single frame and each
150K. If I run du -hs on the directory containing these .xtcs, I
get 4.4 GB. Nevertheless, when I run trjcat -f *.xtc -o
../tot.xtc , my memory consumption goes over 11 GB and then I
run out of available memory.

Sure, I could find a larger memory system, but why does it take
=3x more memory than the individual files took in disk space?
I'd guess that the increased space reflects trjconv converting the  
compressed .xtc format back to normal precision, and then back to  
compressed. To cope with all its possibilities, trjconv probably needs  
to either keep the loaded frames in memory, or use a two-pass  
algorithm. It does the former.

Mark