The TileSystem is the required flexible and compact embedded processing platform.
The concept was finalized during the MARISense project.
TileCube System
When all the parts of the system came, we were able to check the designed mechanical integration of a multi-board design using the Altium Designer CAD against the actual hardware. Initially, each component was checked that fit properly the mechanical enclosure. This was a preliminary test to verify PCB width and component height that might affect the assembly. Although the backplane boards are missing from the below setup, the HAT and other connectors and protruding pins were checked for potential conflicts.
4-Channel Hydrophone Edge Processing System
Then we added the back/front plane boards to complete and final check the system assembly.
The next step of the TileFPGA testing is the SDRAM interface.
As you may see in the following video, writing to the SDRAM memory space seems successful, but reading back the data by refreshing the debugger view, shows that the memory is sparsely written (or ghost read).
FPGA SDRAM Failed Access
This usually is an indicator of the clock phase, which can be addressed in the design’s clock generation phase. By adjusting the clock phase, the clock becomes in sync with the signals and hence the SDRAM is operating properly.
This concludes the TileFPGA testing for now. In the next blog the Hydro4 ADC interface will be checked.
Our company introduced the first-generation power management system during our involvement in the MARI-Sense project. The target applications are Unmanned ...
The second element of the TileSystem useful for edge processing is the FPGA board. This board interfaces tightly with the microcontroller of the system using a bus and also with the potential frontends for input or output.
As FPGAs are also hit by the component shortages in the semiconductor market, the design was based on AMD/Xilinx Spartan 6LX9 device, as this was in stock. The fact that the device is in a QFP package (as opposed to BGA) allows a less expensive PCB design with easier debugging, as all pins can be probed.
TileFPGA Board
The board was designed in less than 2 weeks and it came into our lab for verification. In order to test the board, we must connect it to our TileMCU through a backplane, and run the appropriate firmware that will load a valid configuration to the chip. A suitable test design with proper pin constraints was created.
After the assembly of the missing parts, the TileCUBE assembled with the microcontroller and the FPGA.
TileFPGA in Unconfigured State
The initial power-up lit our default LEDs (orange LED lit, means that the FPGA is unconfigured). We had to compile a test design with the updated pin-out from the Perseus CFE board to the TileFPGA. The design was placed at the SDCard. The firmware was adjusted to load this FPGA bitstream, so we can test the MCU and SDRAM interfaces.
FPGA in Configured State
After the FPGA was configured, we tested the Mini-FlexBus interface. We used the debugger to inspect the register area and confirmed the visibility (read operations). Registers were also written to verify that the interface is working as expected. We tested the LED state change (green color seen in the above photo) by changing the relevant bit in a register.
The next phase was to test the internal Block-RAMs. Initially the internal logic did not route the memories to the FlexBus interface (this was a design feature). the values seen in the memory space are 0xFFFF (due to the pull-ups inside the FPGA logic).
Block-RAM Access Enable and Test
Setting the respective enable bit in the control register the memory space can be written and the values are retained. Note that the default memory values seen by the Bus are 0xFFFF, but as soon as we step an instruction after the memory enable bit is set, the debugger refreshes the memory contents that are now zero.
These tests concluded the basic Mini-FlexBus interface and the internal Block-RAM interfaces. In the next post, we will test the SDRAM.
The TileMCU Coldfire version completed its first set of tests, completing the integration of the USB stack from Freescale/NXP to the RTOS.
Using POSIX-style drivers (or any standard way of interfacing) we were able to replace the Serial input of the Command Line Interpreter (CLI) with the USB one.
The code change was a very simple one, replacing just the name of the driver.
Code Excerpt of Selectable Driver
So essentially we define a preprocessor macro with the name of the required driver. As both the USB-CDC and the Serial driver have the same API, they can be interchanged at will.
Here we do this on compile time. Another option would be to mirror both interfaces and have at the same time either USB or Serial I/O. For now, we decided to keep it simple as we have to test the rest of the boards to complete our hydrophone system.
USB Enumeration
At first, the USB enumeration was checked to see if the basics worked. Then we opened with our terminal the respective virtual serial port and start testing the CLI.
CLI Test Example
The CLI was tested to ensure that there were no problems using it and testing some other functions like the SD Card mounting and FAT access.
As we have completed the first phase of testing we can move on to the next boards for testing. We will return back to this board to test the Mini-FlexBus interface with the FPGA.
The concept of a flexible and scalable system is a usual requirement for many applications. We initially started this flexibility by integrating MCUs with FPGAs on the same board. These created the initial Perseus Family Platform boards that we used for the development of applications.
Later on, during the MARI-Sense project, it was evident that the form factor should be more compact while features should be more selectable. In that spirit, we created the TileCubeTM System. Toward implementation of the platform, we designed boards for a complete hydrophone system, including microcontroller boards, FPGAs, and analog front-ends.
Due to component shortages, we were forced to design and use existing parts that we had in stock for the boards. Some boards are already designed with higher-performance microcontrollers waiting to have the new parts in hand. The flexibility of the system allows using different MCU boards in a backward-compatible manner.
The first board of the Tile System is the microcontroller. This version is based on Coldfire MCF52258.
CAD Model of TileMCU_CF
Soon the boards arrived to our lab and we started the board bring-up process and testing.
Actual TileMCU_CF Sample
We confirmed the firmware configuration changes to support the new I/O assignment and installed COFILOS to test the system. Soon we had a system working with serial communications, command line interface (CLI), etc.
First Light (Blinking) of Firmware
Next steps were to verify the SDCard interface and we tried the first time the Digilent Digital Discovery tool for capturing SPI data.
SDCard SPI Checks
Finally, we confirmed the proper operation of the SDCard and through the CLI we sneaked into the first sector of the device.
SDCard First Sector
Initially, the same board was designed with another microcontroller in mind, but due to component shortages, we had to re-spin the board with Coldfire. It was a fast race to quickly redesign the board, build it and make it work, within our deadlines. This gave us the motivation to complete the rest of the boards with the same speed, so we can see the full system ready and explore the potential of the system.
If you are interested in this platform feel free to contact us.
“Breaking The Surface 2021” event was very successful and interesting. Many scientists of all over thw world presented the latest developments in archeology, marine robotics, sensing and other applications.
We were honored to present our maritime IoT platforms and share our knowledge on embedded system design and hardware acceleration.
We discussed things like power management, showcasing the power management board for the MARI-Sense ASV, hardware acceleration, and versatile heterogeneous platforms.
Ready for the Tutorial with all HardwarePower Management GUI
For anyone attending the multi-disciplinary conference of “Breaking The Surface 2021” event, I will be giving a tutorial on multiple topics on using embedded computing in maritime applications.
In Larnaca, 17-21 June 2021, there was this great event, where we had the opportunity to show our latest developments for the MARI-Sense project in collaboration with CMMI.
On Sunday 21 of June, at Europe Square CMMI and MARI-Sense exhibited various vehicles for maritime applications.
The underwater remote controlled robotic vehicle (ROV) has the capability to provide underwater images or use manipulators to perform delicate actions as required for research or other applications.
Front view of the CMMI’s Robotic Underwater Vehicle
A smart-boat is also developed named “Kerkouros” from the ancient Cyprus ship with oars. “Kerkouros” is under development with the ultimate goal to be an autonomous surface vehicle; a challenging task for the development team.
Prototype of Autonomous Smart Boat Developed at CMMI
For helping development efforts for the MARI-Sense project another platform was exhibited, the Smart-Float. The Smart Float will be used as a technology evaluation platform, where different sensors will be mounted and tested along with radio links, like WiFi, LoRa, thrusters, maneuverability, or other technologies and embedded systems. This platform will help improve the know-how of the team and will lead to new improved models. The main advantage of this platform is its low-cost and large transport capacity (more than 20Kgr net for payload). The team is already thinking of other potential applications where this platform can be used effectively. AI Zerocaliber Ltd is involved with this platform from the concept phases till now, helping with the hull and system design, and specific embedded components.
Smart-Float Platform Shown During EMD-2021 Event
For more information on the MARI-Sense project you may check the project’s website.
It is assumed that you have already setup the Raspberry Pi to have remote desktop and SSH agents active. I also have setup my network to assign a specific IP to this device both for wired or wireless operation. It is also assumed that you already installed git or any other tools you require for development. We used the default Raspbian Linux. Login in with SSH should present something similar as the next picture. Of course, you may use a remote desktop environment; select which best fits your taste.
Raspberry Pi Login Screen
Notice that this is a Linux 5 kernel. This is important mostly for driver’s compatibility or support with the DAC+ADC Pro board. However, we did not face any real issues with our setup.
Python can get messy with packages and proper system configurations. Note that the Raspberry Pi image comes with both Python2 and Python3 interpreters. Writing simple python runs the 2.7 version while running python3 calls the Python 3.x interpreter. Keep this in mind.
In our case, we used the Python3 setup. First, we created a virtual environment to install packages. There are many ways to do the setup. You may try the anaconda system (although the link below for TensorFlow says that this will not work), which will take care of any dependencies and install all the packages, or you can follow on here and see the more tedious manual approach. The idea is to have a system set up in our Raspberry that matches the one we use in our desktop environments to ensure maximum compatibility and help us with testing.
Download the virtualenv package:
$ sudo pip3 install virtualenv
Then we can select a directory to do our development and create the virtual environment there:
$ cd UnderwaterSoundProcessing
$ virtualenv audioml
$ cd audioml
$ source bin/activate
… do something…
$ Deactivate
If you check the requirements.txt in the github repo, you will notice a long list of items that must be installed along with their version numbers. If you try the simple command:
$ pip3 install -r requirements.txt
You will fail miserably. The reason is simple. Kapre requires TensorFlow version at least 1.15, but default TensorFlow for Raspberry is 1.14.
TensorFlow 2 cannot be installed by default on RPB Pi4. We followed the instructions from this link:
The script used is on a google drive (check contents, security and stability issue)
TFLite can provide significant improvements in prediction. You may convert your normal models to Tflite for faster processing. Need to test on each application if accuracy is maintained.
When installing in venv do not use sudo. In case you did (as I did), reinstall (last step) tensorflow without the sudo. Test that Tensorflow is installed from python before continuing.
If in python interpreter you are able to import the tensorflow package as stated in the link you are good to go.
For Tensorlow and Numpy packages we install the ATLAS library.
$ sudo apt install libatlas-base-dev
Another issue we faced was the LLVM library. Numba 0.48 requires v7 of LLVM and not v9.
We run our predict module to see it working. We had already trained previously on a desktop PC the machine learning network to classify a mixture of sea sounds taken in a previous experiment and from the web.
$ python predict.py
Running Training on Raspberry Pi
All these are nice, but still where is the hydrophone? Next steps describe the process to properly interface the DAC+ADC Pro module.
Although the kernel is 5, we needn’t to do the work-around presented for the EEPROM. First, we tested the playback. Please note that the device is not the default, but rather the sysdefault as seen after the next command.
Sound Devices List
The following commands will output sound from the left and right channel respectively.
The values of this command are steps between 0 and 104 and will set ADC volume 0.5db/step. So 96 is about 48dB. You may adjust this value to a lower level depending on the sensitivity. A 33dB gain should work just fine. You may retry to record again and observe the VU meter levels.
For python to access the audio device, we installed the ALSA audio:
To ensure that the same exact code is used either for off-line prediction and on-line prediction, we used a file wrapper in python. The sampling function performs a sampling for a duration (like 1 second) and stores the result in a file. Then the main loop acquires this file and processes it like it does off-line content.
def readdaq(self):
loops = int(self.sample_dur_seconds * round( (self.fs_Hz / self.chunk) ) ) frames = [] sample = np.array(frames) while loops > 0: loops -= 1 # Read data from device l, data = self.inp.read() if l>0: if l!= self.chunk: print("Sampling Error ", l) frames.append(data) wf = wave.open('/mnt/tmpfs/sample.wav', 'wb') wf.setnchannels(self.channels) wf.setsampwidth(2) wf.setframerate(self.fs_Hz) wf.writeframes(b''.join(frames)) wf.close() batch = self.readfile('/mnt/tmpfs/sample.wav')
return batch
Note that the file is saved on a temporary location. This is a ram drive created as follows:
The reason of using a ram drive is that we do not want to wear out our SDcard, or have slow-downs due to file system activity. We use this scratchpad area to write the samples and use them for processing. Then the sample is overwritten by the next sample.
This way we streamline the testing process and are able to run the same code with or without hydrophone, either on desktop, or on Raspberry Pi.
Conclusion
Using open-source software and off-the-shelf hardware we are able to have a platform for sound classification using machine learning. We created a uniform testbed that can be used to test either on-line or off-line the methods employed for evaluation purposes.
Acknowledgment
The SMART Cable was developed through the SMART Cable Project. This project is part of the Research & Innovation Foundation Framework Programme RESTART 2016-2020 for Research, Technological Development and Innovation and co-funded by the Republic of Cyprus and the European Regional Development Fund with grant number ENTERPRISES/0916/0066.
This project is also part of the MARI-Sense Project INTEGRATED/0918/0032. The MARI-Sense project develops intelligent systems that allow human operators to make sense of the complex maritime environment for applications including transport and shipping, coastal tourism, search and rescue, and maritime spatial planning.
