Ticket #115 (new defect)

Opened 11 years ago

Last modified 11 years ago

xuvtools: writing Imaris files with version newer than 1.6 is slow

Reported by: emmenlau Owned by: emmenlau
Priority: major Milestone:
Component: xuvtools Version:
Keywords: xuvtools, stitcher, imaris, save, slow, ims, compression Cc: developer@…

Description

Writing an Imaris file with XuvTools? version 1.6 or 1.7 is much slower than writing Imaris files with version 1.5.0. The reason most likely is the new multiresolution file support. Compression is set to GZIP level 3.

Tested with Dataset:

Flavio-Darkborder:
 - L15P5RLSec3Contraim1.lsm
 - L15P5RLSec3Contraim2.lsm
 - L15P5RLSec3Contraim3.lsm
 - L15P5RLSec3Contraim4.lsm
 - L15P5RLSec3Contraim5.lsm
 - L15P5RLSec3Contraim6.lsm

XuvTools?-1.5.0 reports 15 seconds:

00:04:14: Overlaying images, blending_width_pix = 10, borderFadeWidth = 0
00:04:14: Creating overlay image, size (88,882,1431)px = 105MB.
00:04:28: Overlay finished.

Time to open in Imaris: 10.1 seconds. File size 110MB.

XuvTools?-1.7.0 pre-alpha takes 36 seconds (measured with external clock):

Overlaying images, blending_width_pix = 10, borderFadeWidth = 0
Creating overlay image, size (88,882,1431)px = 105MB.
Overlay finished.

Time to open in Imaris: 0.6 seconds. File size 220MB.

Change History

Changed 11 years ago by ne704 [niko@…

I can confirm this and it gets a lot worse when working with really large datasets, so saving an overlay can take dozens of minutes. I vote for an option whether to save with or without multiresolution, especially since the multiresolution Imaris-files become really really large.

Changed 11 years ago by emmenlau

My first investigation shows this is an issue in the HDF5 library!

Here is the issue:

  • saving a 20MB Dataset in HDF5, every voxel from memory as a voxel in the file, takes 0.1 seconds
  • saving a 20MB Dataset in HDF5, every second voxel from memory as a voxel in the file, takes 6.0 seconds
  • saving a 20MB Dataset in HDF5, every fourth voxel from memory as a voxel in the file, takes 0.2 seconds

Clearly, the HDF5 library must have a _very_ inefficient way of accessing memory. Not doing any downscaling and saving the full array twice is faster than saving every other voxel? That is close to ridiculous.

Will need to investigate more.

Note: See TracTickets for help on using tickets.