sensors

Reads and packages IMU/GPS/Pressure-Humidity-Temperature sensor data over UART. Used for georectification.
Tip

This module can be imported using from openhsi.sensors import *

Warning

Still experimental. Stay tuned.

The OpenHSI camera requires motion to generate 2D spatial datacubes. Yet, motion also introduces other artefacts that need georectification. To correct for spurious motion, we need to collect absolute orientation and geolocation of the camera simultaneously with the camera capture. This is where this module comes in. An IMU, GPS, and Pressure/Humidity/Temperature sensor needs to be read and recorded.

A Teensy 4.0 operates all the sensors, and devices using a Real Time Operating System (RTOS) like cooperative scheduler to run each component update at the desired frequency. By reducing the I/O, and CPU load on the main development board (the Raspberry Pi 4 with 8 GB RAM), the sensor updates are offloaded to a microcontroller with a real time clock to sync and timestamp each sensor measurement. The whole thing is assembled onto a PCB that stacks with the Raspberry Pi 4 and battery hat.

Component Rate (Hz) Info
Teensy 4.0 (uC) 24 MHz Clock —
NEO 9N (GPS) 20 Hz I2C @400 kHz (takes ~20 ms per update)
BNO055 (IMU) 100 Hz I2C_1 @400 kHz
BME280 (Air) 100 Hz I2C_1 @400 kHz
DS3231 (RTC) 100 Hz I2C_1 @400 kHz
XBee 1 Hz UART_1 @115,200 Hz (~2 ms per update)
Raspberry Pi 4 packets @100 Hz UART @921,600 Hz (~0.8 ms per update)
Start button 4 Hz poll button linked to LED notifying status

An XBee is also programmed to check sensor status remotely during operation. This could be useful to diagnose any issues without being physically connected to the microcontroller. A basic streaming dashboard is included.

Each data packet contains timestamped sensor data. The item fields are then extracted from the raw binary serial stream.

Note

The Teensy runs a 32 bit Cortex-M7 so serial packets are padded. In other words, the data struct is padded in contiguous memory so a byte variable followed by float variable will include 3 unused bytes in-between so things are packed as 32 bits at a time.

The data packet is sent as a C struct so we need to decode the binary stream and interpret each byte as the corresponding C type. In each packet, there are status bytes to indicate which sensor has been updated.

Create a standrd interface for the sensor master loop to check if collection shoud stop.

SensorStream.read_packet[source]

SensorStream.read_packet(header:chr=b'*', num_bytes:int=76, timeout:float=2.0)

Reads at least `num_bytes` of a data packet starting with `header` 
and times out after `timeout` seconds if packet is invalid.

SensorStream.save[source]

SensorStream.save()

Save the data packets. Will save some plots of the data as well.

SensorStream.master_loop[source]

SensorStream.master_loop(n_lines:int=128, processing_lvl:int=0, json_path:str=None, pkl_path:str=None, preconfig_meta:str=None, ssd_dir:str=None)

Continuous run saving packets during start button pressed. If you want to capture camera as well, 
input all the optional parameters.
Type Default Details
n_lines int 128 how many along-track pixels
processing_lvl int 0 desired processing done in real time
json_path str None path to settings file
pkl_path str None path to calibration file
preconfig_meta str None path to metadata file
ssd_dir str None path to SSD

SensorStream.clean_df[source]

SensorStream.clean_df(df:DataFrame)

Converts time offsets in `df` into datetime and splits sensor readings that update 
at different rates. Also saves the plots as a picture.
Tip

The serial port for Raspberry Pi is “/dev/serial0”. For the Jetson, it is “/dev/ttyTHS0”.

Note

GPS PPS callbacks are experimental. I haven’t found a way to use them effectively.

Let’s now test this using simulated ancillary sensor data packets.

We can simulate data packets for testing purposes. This will generate 77 data packets. You can then save the data - it will be cleaned up so each sensor has its own unique timestamp.

toggle_interface = GPIOInterface(start_pin=17)

ss = SensorStream(baudrate = 921_600,
                  port = '/dev/serial0',
                  toggle_interface = toggle_interface,
                  ssd_dir = '.')

ss.packets = []
for i in tqdm(range(77)):
    ss.packets.append(collect_sim(rtc_offset_ms=150))
    time.sleep(0.01)

#ss.save()
/xavier_ssd/mambaforge/envs/openhsi_dev/lib/python3.10/site-packages/Jetson/GPIO/gpio.py:383: RuntimeWarning: This channel is already in use, continuing anyway. Use GPIO.setwarnings(False) to disable warnings
  warnings.warn(
/tmp/ipykernel_20274/2005748132.py:5: UserWarning: [Errno 2] could not open port /dev/serial0: [Errno 2] No such file or directory: '/dev/serial0': could not open port /dev/serial0.
  ss = SensorStream(baudrate = 921_600,
100%|████████████████████████████████████████████████████████████████████████| 77/77 [00:00<00:00, 90.05it/s]

We can also use an infinite loop to continuously save sensor data when a hardware button is latched. When data is saved, a summary plot of the data is also saved alongside. Here I specifically exclude the OpenHSI camera by not providing the argument cam_name to SensorStream.__init__.

from multiprocessing import Value
from ctypes import c_bool


import Jetson.GPIO as GPIO
GPIO.setmode(GPIO.BCM) # BCM pin-numbering scheme from Raspberry Pi
GPIO.setup(17, GPIO.OUT)
GPIO.output(17, True)


testMP = MPInterface(Value(c_bool, True))

ss = SensorStream(baudrate = 921_600,
                  toggle_interface=testMP,
                  port = '/dev/ttyTHS0',
                  ssd_dir = '/xavier_ssd/hyperspectral_experiments')
time.sleep(1)
# ss.master_loop()
ss.ser.write(b'y')
packets=0

while ss.ser.in_waiting > 0 and packets < 200:
    packets+=1
    # print(packets)
    ss.packets.append( decode_packet(ss.read_packet()) )
    time.sleep(0.1)
        
ss.ser.write(b'n')
ss.save()

Of course, you can also save ancillary sensor data with the OpenHSI camera datacubes - just provide cam_name and also the optional parameters in SensorStream.master_loop.

ss = SensorStream(baudrate = 921_600,
                  port = '/dev/serial0',
                  start_pin = 17,
                  ssd_dir = '/media/pi/fastssd',
                  cam_name="FlirCamera")

ss.master_loop(n_lines=256,
               processing_lvl=2,
               json_path="/media/pi/fastssd/cals/OpenHSI-FLIR01/OpenHSI-FLIR01_settings_Mono8_bin1.json",
               pkl_path="/media/pi/fastssd/cals/OpenHSI-FLIR01/OpenHSI-FLIR01_calibration_Mono8_bin1.pkl",
               preconfig_meta=None,
               ssd_dir="/media/pi/fastssd",
               switch_pin=17)

The ancillary sensor timestamps are different from the datacube along-track timestamps so some interpolation is needed. Here is a function that does that for you.

Streaming dashboard

View the XBee status on a dashboard that shows the last 100 points. This is a simple implementation (it is possible to improve the front end using something like Dash and plotly).

sd = SensorDashboard()
sd()
sd.run()

The SensorDashboard will save the data coming in which can be accessed in a pd.DataFrame. Here is some experimental data with noise added to the latitude/longitude points so the ESRI map loads.

sd.data_df.head(10)