shared

We want to collect datacubes nonstop. Saving is a blocking operation which leaves gaps in the data we save on the order of a few seconds. The larger the datacube, the longer this gap. We really don’t want these gaps, yet we want to save raw data. Saving raw data is extremely demanding (15 s of collect requires 15 s to save perhaps?). Can the saving be done in parallel? The following tries to address this issue.

Tip

This module can be imported using from openhsi.shared import *

Warning

Experimental

---

source

SharedCircArrayBuffer

 SharedCircArrayBuffer (size:tuple=(100, 100), axis:int=0,
                        c_dtype:type=<class 'ctypes.c_ubyte'>, show_func:C
                        allable[[numpy.ndarray],ForwardRef('plot')]=None)

Circular FIFO Buffer implementation on multiprocessing.Array. Each put/get is a (n-1)darray.

---

source

SharedDataCube

 SharedDataCube (n_lines:int=16, processing_lvl:int=-1,
                 json_path:str=None, pkl_path:str=None,
                 print_settings:bool=False)

Facilitates the collection, viewing, and saving of hyperspectral datacubes using two SharedCircArrayBuffers that swap when save is called.

---

source

SharedDataCube.save

 SharedDataCube.save (save_dir:str, preconfig_meta_path:str=None,
                      prefix:str='', suffix:str='', old_style:bool=True)

Saves to a NetCDF file (and RGB representation) to directory dir_path in folder given by date with file name given by UTC time. Save is done in a separate multiprocess.Process.

---

source

SharedDataCube.show

 SharedDataCube.show (plot_lib:str='bokeh', red_nm:float=640.0,
                      green_nm:float=550.0, blue_nm:float=470.0,
                      robust:bool=False, hist_eq:bool=False,
                      quick_imshow:bool=False, **plot_kwargs)

Generate an RGB image from chosen RGB wavelengths with histogram equalisation or percentile options. The plotting backend can be specified by plot_lib and can be “bokeh” or “matplotlib”. Further customise your plot with **plot_kwargs. quick_imshow is used for saving figures quickly but cannot be used to make interactive plots.

	Type	Default	Details
plot_lib	str	bokeh	Plotting backend. This can be ‘bokeh’ or ‘matplotlib’
red_nm	float	640.0	Wavelength in nm to use as the red
green_nm	float	550.0	Wavelength in nm to use as the green
blue_nm	float	470.0	Wavelength in nm to use as the blue
robust	bool	False	Choose to plot using the 2-98% percentile. Robust to outliers
hist_eq	bool	False	Choose to plot using histogram equilisation
quick_imshow	bool	False	Used to skip holoviews and use matplotlib for a static plot
plot_kwargs

---

source

save_shared_datacube

 save_shared_datacube (fname:str, shared_array:<boundmethodBaseContext.Arr
                       ayof<multiprocessing.context.DefaultContextobjectat
                       0x7f9b2eb2aaf0>>, c_dtype:type, shape:Tuple,
                       coords_dict:Dict, attrs_dict:Dict, proc_lvl:int,
                       old_style:bool=True, savefig:bool=False)

Saves a NetCDF4 file given all the function parameters. Designed to be used with SharedOpenHSI which allocates a shared array.

	Type	Default	Details
fname	str		NetCDF4 file name (without .nc)
shared_array	Array		multiprocessing.Array shared array
c_dtype	type		numpy data type
shape	typing.Tuple		datacube numpy shape
coords_dict	typing.Dict		coordinates dictionary
attrs_dict	typing.Dict		metadata dictionary
proc_lvl	int		processing level used
old_style	bool	True	order of axis
savefig	bool	False	save a preview figure of cube

OpenHSI using shared multiprocessing.Array in SharedDataCube

SharedOpenHSI has the same API as OpenHSI with the addition of a camera temperature buffer that automatically swaps over when a save is called.

---

source

SharedOpenHSI

 SharedOpenHSI (n_lines:int=16, processing_lvl:int=-1, json_path:str=None,
                pkl_path:str=None, print_settings:bool=False)

Base Class for the OpenHSI Camera.

See https://openhsi.github.io/openhsi/capture.html for an example of how to use SharedOpenHSI for you custom cameras.

Shared Cameras

These are exported by openhsi.cameras as SharedXXXCamera This should work just like standard Camera.

from openhsi.cameras import SharedXimeaCamera

num_saved = 0
with SharedXimeaCamera(n_lines=128, exposure_ms=1, processing_lvl = -1, pkl_path="",json_path='../assets/cam_settings_ximea.json') as cam:
    for i in range(10):
        cam.collect()
        print(f"collected from time: {cam.timestamps.data[0]} to {cam.timestamps.data[-1]}")
        if num_saved > 0:
            p.join() # wait for the last process to finish so we don't modify the data when it's being saved 
            pass
        
        p = cam.save("../hyperspectral_experiments/temp")
        num_saved += 1