Paul Brewster, Software Lead - PowerPoint Presentation

1. Paul Brewster, Software LeadMarch 21st, 2023Flight Software for the LOFTID Re-Entry Vehicle

2. Project IntroductionAvionics OverviewPDSU SoftwareCamera Controller SoftwareData Recorder SoftwareConclusionsAgenda

3. Hypersonic Inflatable Aerodynamics Decelerator (HIAD)Aeroshell designed to protect and decelerate a spacecraft entering from Low-Earth orbitInflatable, allowing Aeroshell to be larger than the rocket’s shroud6-meter diameterFirst Demonstration from Low-Earth orbitLaunched in November from Vandenberg on ULA’s Atlas VSecondary payload to JPSS-2Orbited the Earth once, splashing down near Hawaii2.5 hours between launch and splash down, powered on for about an hourProject Introduction

4. Project IntroductionAvionics OverviewPDSU SoftwareCamera Controller SoftwareData Recorder SoftwareConclusionsAgenda

5. Avionics Overview+15VPowerSerial/EthrntDiscrete Cmds/SigLEGENDPwr-On CmdPyro Btry Chrgr28VPyro28VEthrntEthrntEthrntEthrntEthrntGSE EthrntGSE FSW EthrntEthrnt (X6)Serial28VPyroPyroSensorsGPSIridiumSerialPyroPyroPyroPAS Sep LoopPAS SepLV Sep LoopLV SepSubsystems/ComponentsAnalogDASADAUInflation SystemEjection SystemDASData SwitchAeroshell ParachuteUp-Look Camera5V, 28V 5V 12VDC/DCPDSUISPSPDSULNA15V PwrLV5V +15V -15VRecovery LightsSys Btry ChrgrLB SerialLB Btry ChrgrPyro28VPyro BatterySystem BatteryFOSSIDRRTBIMUCamera Pod (X6)PDSUModular Avionics Control HardwarePDSUPyro RelayEDREDR SerialEDR Btry ChrgrGSE 28VPDSUInterface Adapter BoardLBGSE

6. Avionics Components with Flight Software+15VPowerSerial/EthrntDiscrete Cmds/SigLEGENDPwr-On CmdPyro Btry Chrgr28VPyro28VEthrntEthrntEthrntEthrntEthrntGSE EthrntGSE FSW EthrntEthrnt (X6)Serial28VPyroPyroSensorsGPSIridiumSerialPyroPyroPyroPAS Sep LoopPAS SepLV Sep LoopLV SepSubsystems/ComponentsAnalogDASADAUInflation SystemEjection SystemDASData SwitchAeroshell ParachuteUp-Look Camera5V, 28V 5V 12VDC/DCPDSUISPSPDSULNA15V PwrLV5V +15V -15VRecovery LightsSys Btry ChrgrLB SerialLB Btry ChrgrPyro28VPyro BatterySystem BatteryFOSSIDRRTBIMUCamera Pod (X6)PDSUModular Avionics Control HardwarePDSUPyro RelayEDREDR SerialEDR Btry ChrgrGSE 28VPDSUInterface Adapter BoardLBGSE

7. Class D Mission, Class C SoftwarePDSU SoftwareSequence Mission TimelineData AcquisitionComponent ControlTelemetryCamera Controller Software6 Camera ControllersEach Controller Interfaces with 1 Visual and 2 IR CamerasReceives Camera stream, processes, records locally, sends to Data RecordersData Recorder SoftwareCaptures all network traffic and records the traffic to an SD cardIdentical software on the Internal and Ejectable Data RecordersFOSS SoftwareCPU/FPGA hybrid to sample laser return along multiple fiber optic strands measuring temperatureFPGA used to perform 40K FFTs per second on the laser return signalSoftware Components

8. Project IntroductionAvionics OverviewPDSU SoftwareCamera Controller SoftwareData Recorder SoftwareConclusionsAgenda

9. Target HardwareModular Avionics Control Hardware (MACH) compute systemProcured from Northrup GrummanTarget ProcessorPower PC 8280 ProcessorTarget OSLynxOSPOSIX compliantReal-Time Operating SystemSoftware FrameworkcFS 6.5PDSU Software

10. PDSU SoftwareArchitectureDigital I/O InterfaceEventServicesExecutiveServicesTimeServicesCommandIngestTelemetry OutputTableServicesPower TransferControlSchedulerHouse-keepingControlcFS AppscFE AppsPowerControlOrdnanceControlStoredCommandNav FilterCalculationIMU InterfaceGPS InterfaceAnalog I/OInterfaceDAU InterfaceData AcquisitionInflation SystemInterfaceReal-Time BeaconInterfaceData Recorder InterfaceTelemetrySoftware BusRecoveryLight

11. Based on posix OSAL and pc-linux PSPOSALThreaded Signal Handler behavior ambiguityPOSIX does not define which thread receives a given signalCreated dedicated Timer Thread to do signal handlingUsed msg queue for other threads to add and remove signalsPthread Mutexes were not recursiveAdded recursive functionality to OS_mutexDynamic Object Loader API inconsistentSwitched to using a module id for all shared object symbol lookupsPSPShared Memory inconsistentSwitched to POSIX real-time shared memoryReworked Timer Thread to reflect OSAL changesMajor Changes for LynxOS

12. OSALImplemented os_vsnprintf since vsnprintf didn’t existChanged setschedparams to setschedprioAdded const to buffer variables being passed into a write functionReplaced statvfs structure with statfs structureRemoved gethostidLinked to netinet instead of pthreads and rtPSPSwitched away from long command line optionsCFEAdded _STDC_LIMIT_MACROS definesModified network include filesFixed byte swapping in CCSDS headerMinor Changes for LynxOS

13. Project IntroductionAvionics OverviewPDSU SoftwareCamera Controller SoftwareData Recorder SoftwareConclusionsAgenda

14. Each Camera Pod consists ofCamera ControllerOn-board data storageNetwork Connection to Data Recorders and GSEIMPERX Cheetah camera connected via HD-SDITwo FLIR BOSON Infrared Cameras connected via USBCamera Pod

15. Target HardwareAiTech A176 based on NVIDIA Jetson TX2 hardwareTarget ProcessorARM Cortex-A57 CPUNvidia Pascal GPUTarget OSJetson LinuxUbuntu based LinuxFrameworkGstreamer media pipelineCamera Controller SoftwareImage courtesy of AiTech Systems

16. ReceivesTwo IR Video streams (640x512 at 10Hz)One Visual Video stream (1920x1080 at 30 Hz)Decodes/EncodesUses the on-board GPU for real-time video encoding and decodingRecordsFull resolution streams to on-board storageSendsVisual Video to Data Recorders as H265 video at 12Mbs bitrateIR Video to Data Recorders as PNG compressed images at 1HzCamera Controller SoftwareFunctionality

17. Project IntroductionAvionics OverviewPDSU SoftwareCamera Controller SoftwareData Recorder SoftwareConclusionsAgenda

18. Two Data RecordersInternal Data RecorderEjectable Data RecorderEach Data Recorder containsA Single Board ComputerA separate ejectable SD cardData SourcesPDSU6 Camera PodsAeroshell Data Acquisition Unit (DAU)FOSSData SubscriptionAll data is sent using multi-cast UDPEach source sends data just once to their respective multi-cast addressThe IDR, EDR, and GSE subscribe to the multicast addresses to receive the dataData StorageData is stored unprocessed on the SD card in PCAP format to maintain network informationData Recorder

19. Target HardwareAiTech A176 based on NVIDIA Jetson TX2 hardwareTarget ProcessorARM Cortex-A57 CPUNvidia Pascal GPUTarget OSJetson LinuxUbuntu based LinuxFrameworklibpcapData Recorder SoftwareImage courtesy of AiTech Systems

20. Extensive Testing was done to ensure data integritySpecifically looking for dropped or corrupted packetsIssues were found in four places in the data pipelineData not properly routed at the Data SwitchData not properly received by the Data RecorderData not properly written to the SD card by the Data RecorderData not properly verified during post-process integrity checksData Integrity

21. Issue:In testing, significant chunks of data were dropped at the Data Switch once every few minutesThe Data Switch drops packets due to high traffic loadBy analysis, average network bandwidth was well under data switch limitsOverall network bandwidth was 20Mbps and the Data Switch was tested up to 60MbpsTwo SolutionsEnable Internet Group Management Protocol (IGMP)Without IGMP, multicast packets are routed to all connected devicesEach device is expected to drop the packet if it isn’t subscribed to that multicast packetResulted in over x6 the bandwidth usageCamera Pod Controllers sending IR VideoEach of the 6 Camera Controllers sends two IR images at 1HzIf the images are all sent at the same time, there is a sudden spike in traffic, even though average bandwidth is very small, causing dropped packetsBy staggering the image sends, there is never a sudden spike in trafficData not properly routed at the Data Switch

22. Issue:The software does not receive the data from the network kernel buffer fast enough, resulting in data being lost in the bufferSolution:A thread is created for each multicast addressEach thread does a non-blocking poll to receive data from that addressWhen no data is available, the thread sleeps to allow another thread to goOther Attempts:Using a single thread to receive all of the multicast traffic. Design was simpler, but the libpcap filter was complex, which slowed overall execution timeUsing blocking polls to receive data. Reduced CPU usage but was overall slower.Using non-blocking polls with a timeout-based select function. Reduced CPU usage but was overall slower.Data not properly received by the Data Recorder

23. Issue:Writing to the SD card is a slow process, although theoretical bandwidth is sufficient for the dataOccasional multi-second delays in writesSolution:A separate thread for writing data to the SD cardWrite data in 10MB chunks, optimizing the throughputWrite each chunk as a separate fileUse a ring buffer to handle the isolated delaysData not properly written to the SD card

24. Data Pipeline

25. Compare data from IDR, EDR, and GSE Data Recorder simulator to verify integrityIssuesData Recorders are not started at the same timeData Recorders are not stopped at the same timeThe order of packets in the data file is not guaranteedProcessMerge the chunk files into one large data file for each Data RecorderFilter out non data from the network captureSeparate each packet by their sourceWrite the checksum for each packet into a checksum summary file corresponding to that sourceCompare the checksum summary files between two Data RecordersSearch for differences in summary files, ignoring the cases where one has extra packets at the beginning or extra packets at the endData not properly verified during post-process integrity checks

26. Project IntroductionAvionics OverviewPDSU SoftwareCamera Controller SoftwareData Recorder SoftwareConclusionsAgenda

27. Conclusions

28. PDSU Software – Langley Research CenterMatt RodgersJacob CassidyJoel Amert (MSFC)David KeckPeter HuynhCamera Controller Software – Marshall Space Flight CenterDerek MoodyDarrell GaddyBrent BeaboutData Recorder Software – Langley Research CenterMatt RodgersFOSS Software – Armstrong Flight Research CenterAllen ParkerTesting & QAMark WehmanShane WolfCredits