shared

We want to collect datacubes nonstop. Saving is a blocking operation which leaves gaps in the data we save on the order of a few seconds. The larger the datacube, the longer this gap. We really don’t want these gaps, yet we want to save raw data. Saving raw data is extremely demanding (15 s of collect requires 15 s to save perhaps?). Can the saving be done in parallel? The following tries to address this issue.

---

source

SharedCircArrayBuffer

 SharedCircArrayBuffer (size:tuple=(100, 100), axis:int=0,
                        c_dtype:type=<class 'ctypes.c_ubyte'>, show_func:C
                        allable[[numpy.ndarray],ForwardRef('plot')]=None)

Circular FIFO Buffer implementation on multiprocessing.Array. Each put/get is a (n-1)darray.

---

source

SharedDataCube

 SharedDataCube (n_lines:int=16, processing_lvl:int=-1,
                 json_path:str=None, cal_path:str=None,
                 print_settings:bool=False)

Facilitates the collection, viewing, and saving of hyperspectral datacubes using two SharedCircArrayBuffers that swap when save is called.

---

source

SharedDataCube.save

 SharedDataCube.save (save_dir:str, preconfig_meta_path:str=None,
                      prefix:str='', suffix:str='', old_style:bool=True,
                      savefig:bool=False)

Saves to a NetCDF file (and RGB representation) to directory dir_path in folder given by date with file name given by UTC time. Save is done in a separate multiprocess.Process.

---

source

SharedDataCube.show

 SharedDataCube.show (plot_lib:str='bokeh', red_nm:float=640.0,
                      green_nm:float=550.0, blue_nm:float=470.0,
                      robust:bool=False, hist_eq:bool=False,
                      quick_imshow:bool=False, **plot_kwargs)

Generate an RGB image from chosen RGB wavelengths with histogram equalisation or percentile options. The plotting backend can be specified by plot_lib and can be “bokeh” or “matplotlib”. Further customise your plot with **plot_kwargs. quick_imshow is used for saving figures quickly but cannot be used to make interactive plots.

	Type	Default	Details
plot_lib	str	bokeh	Plotting backend. This can be ‘bokeh’ or ‘matplotlib’
red_nm	float	640.0	Wavelength in nm to use as the red
green_nm	float	550.0	Wavelength in nm to use as the green
blue_nm	float	470.0	Wavelength in nm to use as the blue
robust	bool	False	Choose to plot using the 2-98% percentile. Robust to outliers
hist_eq	bool	False	Choose to plot using histogram equilisation
quick_imshow	bool	False	Used to skip holoviews and use matplotlib for a static plot
plot_kwargs	VAR_KEYWORD

---

source

save_shared_datacube

 save_shared_datacube (fname:str, shared_array:<boundmethodBaseContext.Arr
                       ayof<multiprocessing.context.DefaultContextobjectat
                       0x7fdbe1a1d450>>, c_dtype:type, shape:Tuple,
                       coords_dict:Dict, attrs_dict:Dict, proc_lvl:int,
                       old_style:bool=True, savefig:bool=False)

Saves a NetCDF4 file given all the function parameters. Designed to be used with SharedOpenHSI which allocates a shared array.

	Type	Default	Details
fname	str		NetCDF4 file name (without .nc)
shared_array	Array		multiprocessing.Array shared array
c_dtype	type		numpy data type
shape	Tuple		datacube numpy shape
coords_dict	Dict		coordinates dictionary
attrs_dict	Dict		metadata dictionary
proc_lvl	int		processing level used
old_style	bool	True	order of axis
savefig	bool	False	save a preview figure of cube

OpenHSI using shared multiprocessing.Array in SharedDataCube

SharedOpenHSI has the same API as OpenHSI with the addition of a camera temperature buffer that automatically swaps over when a save is called.

---

source

SharedOpenHSI

 SharedOpenHSI (n_lines:int=16, processing_lvl:int=-1, json_path:str=None,
                cal_path:str=None, print_settings:bool=False)

Base Class for the OpenHSI Camera.

Shared Cameras

These are exported by openhsi.cameras as SharedXXXCamera This should work just like standard OpenHSI.Camera. Here is an example using SharedSimulatedCamera.

from openhsi.cameras import SharedSimulatedCamera
    
import tempfile
import os
import time

num_saved = 0
num2save  = 3
imgs=[]

with tempfile.TemporaryDirectory() as temp_dir:
    print(temp_dir)
    with SharedSimulatedCamera(img_path="../assets/great_hall_slide.png", 
                               n_lines=535, 
                               processing_lvl = -1, 
                               json_path="../assets/cam_settings.json",
                               cal_path="../assets/cam_calibration.nc",
                               print_settings=False) as cam:
        
        for i in range(num2save):
            if num_saved > 0:
                #p.join() # waiting for the last process to finish will make this slow. 
                pass
                
            cam.collect()
            print(f"collected from time: {cam.timestamps.data[0]} to {cam.timestamps.data[-1]}")
            
            imgs.append(cam.show("bokeh").opts(width=200))
            
            p = cam.save(temp_dir)
            num_saved += 1
            
    
    time.sleep(2) # let saving finish before check and cleanup
    print(f"finished saving {num2save} datacubes")
    for root, dirs, files in os.walk(temp_dir):
        for file in files:
            file_path = os.path.join(root, file)
            relative_path = os.path.relpath(file_path, temp_dir)
            print(f"  {relative_path}")
            
imgs[0]+imgs[1]+imgs[2]

/var/folders/gn/gv16fym50pv5_g7s8gt7451c0000gp/T/tmpr2loh_ou
Allocated 4676.68 MB of RAM.

100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 535/535 [00:02<00:00, 264.55it/s]

collected from time: 2025-06-19T05:29:31.987610000 to 2025-06-19T05:29:34.005276000

Saving /var/folders/gn/gv16fym50pv5_g7s8gt7451c0000gp/T/tmpr2loh_ou/2025_06_19/2025_06_19-05_29_31 in another process.

100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 535/535 [00:01<00:00, 295.23it/s]

collected from time: 2025-06-19T05:29:34.302093000 to 2025-06-19T05:29:36.109223000

Saving /var/folders/gn/gv16fym50pv5_g7s8gt7451c0000gp/T/tmpr2loh_ou/2025_06_19/2025_06_19-05_29_34 in another process.

100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 535/535 [00:02<00:00, 264.70it/s]

collected from time: 2025-06-19T05:29:36.408037000 to 2025-06-19T05:29:38.423919000

Saving /var/folders/gn/gv16fym50pv5_g7s8gt7451c0000gp/T/tmpr2loh_ou/2025_06_19/2025_06_19-05_29_36 in another process.
finished saving 3 datacubes
  2025_06_19/2025_06_19-05_29_34.nc
  2025_06_19/2025_06_19-05_29_31.nc
  2025_06_19/2025_06_19-05_29_36.nc