Skip to content

Commit 4af9276

Browse files
committed
Ethernet and Audio section done
1 parent 0bd2e0d commit 4af9276

File tree

6 files changed

+205
-3
lines changed

6 files changed

+205
-3
lines changed
1.5 MB
Loading
6.33 MB
Loading
1.73 MB
Loading
572 KB
Loading
312 KB
Loading

content/hardware/04.pro/shields/portenta-vision-shield/tutorials/user-manual/content.md

Lines changed: 205 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ Here is an overview of the board's main components, as shown in the images above
6060
- **Digital Microphones**: the dual MP34DT05 digital MEMS microphones are omnidirectional and operate via a capacitive sensing element
6161
with a high (64 dB) signal-to-noise ratio. The microphones have been configured to provide separate left and right audio over a single PDM stream.
6262

63-
The sensing element, capable of detecting acoustic waves, is manufactured using a specialized silicon micromachining process dedicated to produce audio sensors.
63+
The sensing element, capable of detecting acoustic waves, is manufactured using a specialized silicon micromachining process dedicated to produce audio sensors.
6464

6565
- **Micro SD Card Slot**: a Micro SD card slot is available under the Portenta Vision Shield board. Available libraries allow reading and
6666
writing to FAT16/32 formatted cards
@@ -288,6 +288,7 @@ led.off()
288288

289289
raise (Exception("Please reset the camera to see the new file."))
290290
```
291+
We recommend you use [VLC](https://www.videolan.org/vlc/) to play the video.
291292

292293
![Video saved in local storage](assets/video-ani.gif)
293294

@@ -542,8 +543,209 @@ When a person is in the field of view of the camera, you should see the inferenc
542543

543544
### Microphone
544545

545-
The Portenta Vision Shield features two microphones, based on the MP34DT05 ultra-compact, low-power, omnidirectional, digital MEMS microphone.
546+
The Portenta Vision Shield features two omnidirectional microphones, based on the MP34DT05 ultra-compact, low-power, and digital MEMS microphone.
547+
548+
![Vision Shield omnidirectional microphones](assets/microphones.png)
549+
550+
**Features:**
551+
- AOP = 122.5 dB SPL
552+
- 64 dB signal-to-noise ratio
553+
- Omnidirectional sensitivity
554+
- –26 dBFS ± 1 dB sensitivity
555+
556+
#### FFT Example
557+
558+
You can analyze frequencies present in sounds alongside their harmonic features using this example.
559+
560+
By measuring the sound level on each microphone we can easily know from where the sound is coming, an interesting capability for robotics and AIoT applications.
561+
562+
```python
563+
import image
564+
import audio
565+
from ulab import numpy as np
566+
from ulab import utils
567+
568+
CHANNELS = 2
569+
SIZE = 512 // (2 * CHANNELS)
570+
571+
raw_buf = None
572+
fb = image.Image(SIZE + 50, SIZE, image.RGB565, copy_to_fb=True)
573+
audio.init(channels=CHANNELS, frequency=16000, gain_db=24, highpass=0.9883)
574+
575+
576+
def audio_callback(buf):
577+
# NOTE: do Not call any function that allocates memory.
578+
global raw_buf
579+
if raw_buf is None:
580+
raw_buf = buf
581+
582+
583+
# Start audio streaming
584+
audio.start_streaming(audio_callback)
585+
586+
587+
def draw_fft(img, fft_buf):
588+
fft_buf = (fft_buf / max(fft_buf)) * SIZE
589+
fft_buf = np.log10(fft_buf + 1) * 20
590+
color = (222, 241, 84)
591+
for i in range(0, SIZE):
592+
img.draw_line(i, SIZE, i, SIZE - int(fft_buf[i]), color, 1)
593+
594+
595+
def draw_audio_bar(img, level, offset):
596+
blk_size = SIZE // 10
597+
color = (214, 238, 240)
598+
blk_space = blk_size // 4
599+
for i in range(0, int(round(level / 10))):
600+
fb.draw_rectangle(
601+
SIZE + offset,
602+
SIZE - ((i + 1) * blk_size) + blk_space,
603+
20,
604+
blk_size - blk_space,
605+
color,
606+
1,
607+
True,
608+
)
609+
610+
611+
while True:
612+
if raw_buf is not None:
613+
pcm_buf = np.frombuffer(raw_buf, dtype=np.int16)
614+
raw_buf = None
615+
616+
if CHANNELS == 1:
617+
fft_buf = utils.spectrogram(pcm_buf)
618+
l_lvl = int((np.mean(abs(pcm_buf[1::2])) / 32768) * 100)
619+
else:
620+
fft_buf = utils.spectrogram(pcm_buf[0::2])
621+
l_lvl = int((np.mean(abs(pcm_buf[1::2])) / 32768) * 100)
622+
r_lvl = int((np.mean(abs(pcm_buf[0::2])) / 32768) * 100)
623+
624+
fb.clear()
625+
draw_fft(fb, fft_buf)
626+
draw_audio_bar(fb, l_lvl, 0)
627+
if CHANNELS == 2:
628+
draw_audio_bar(fb, r_lvl, 25)
629+
fb.flush()
630+
631+
# Stop streaming
632+
audio.stop_streaming()
633+
```
634+
635+
With this script running you will be able to see the Fast Fourier Transform result in the image viewport. Also, the sound level on each microphone channel.
636+
637+
![FFT example running](assets/fft.gif)
638+
639+
#### Speech Recognition Example
640+
641+
You can easily implement sound/voice recognition applications using Machine Learning on the edge, this means that the Portenta H7 plus the Vision Shield can run these algorithms locally.
642+
643+
For this example, we are going to test a [pre-trained model]((https://raw.githubusercontent.com/iabdalkader/microspeech-yesno-model/main/model.tflite)) that can recognize the `yes` and `no` keywords
644+
645+
First, download the `.tflite` [model](https://raw.githubusercontent.com/iabdalkader/microspeech-yesno-model/main/model.tflite) and copy it to the H7 local storage.
646+
647+
Use the following script to run the example. It can also be found on **File > Examples > Audio > micro_speech.py** in the OpenMV IDE.
648+
649+
```python
650+
import audio
651+
import time
652+
import tf
653+
import micro_speech
654+
import pyb
655+
656+
labels = ["Silence", "Unknown", "Yes", "No"]
657+
658+
led_red = pyb.LED(1)
659+
led_green = pyb.LED(2)
660+
661+
model = tf.load("/model.tflite")
662+
speech = micro_speech.MicroSpeech()
663+
audio.init(channels=1, frequency=16000, gain_db=24, highpass=0.9883)
664+
665+
# Start audio streaming
666+
audio.start_streaming(speech.audio_callback)
667+
668+
while True:
669+
# Run micro-speech without a timeout and filter detections by label index.
670+
idx = speech.listen(model, timeout=0, threshold=0.70, filter=[2, 3])
671+
led = led_green if idx == 2 else led_red
672+
print(labels[idx])
673+
for i in range(0, 4):
674+
led.on()
675+
time.sleep_ms(25)
676+
led.off()
677+
time.sleep_ms(25)
678+
679+
# Stop streaming
680+
audio.stop_streaming()
681+
```
682+
Now, just say `yes` or `no` and you will see the inference result in the OpenMV Serial Monitor.
683+
684+
![Speech recognition example](assets/ml-inference.png)
546685

547686
### Ethernet (ASX00021)
548687

549-
### LoRa® (ASX00026)
688+
The **Portenta Vision Shield - Ethernet** gives you the possibility of connecting your Portenta H7 board to the internet using a wired connection.
689+
690+
![Ethernet cable connected](assets/ethernet-connect.png)
691+
692+
First, connect the Vision Shield - Ethernet to the Portenta H7. Now connect the USB-C® cable to the Portenta H7 and your computer. Lastly, connect the Ethernet cable to the Portenta Vision Shield's Ethernet port and your router or modem.
693+
694+
Now you are ready to test the connectivity with the following Python script. This example lets you know if an Ethernet cable is connected successfully to the shield.
695+
696+
```python
697+
import network
698+
import time
699+
700+
lan = network.LAN()
701+
702+
# Make sure Eth is not in low-power mode.
703+
lan.config(low_power=False)
704+
705+
# Delay for auto negotiation
706+
time.sleep(3.0)
707+
708+
while True:
709+
print("Cable is", "connected." if lan.status() else "disconnected.")
710+
time.sleep(1.0)
711+
```
712+
If the physical connection is detected, in the OpenMV Serial Monitor, you will see the following message:
713+
714+
`Cable is connected.`
715+
716+
Once the connection is confirmed, we can try to connect to the internet using the example script below.
717+
718+
This example lets you gather the current time from an NTP server.
719+
720+
```python
721+
import network
722+
import socket
723+
import struct
724+
import time
725+
726+
TIMESTAMP = 2208988800 + (3600*4) # (3600*4) is used to set the Time Zone (UTC-4)
727+
728+
if time.gmtime(0)[0] == 2000:
729+
TIMESTAMP += 946684800
730+
731+
# Create new socket
732+
client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
733+
734+
# Get addr info via DNS
735+
addr = socket.getaddrinfo("pool.ntp.org", 123)[0][4]
736+
737+
# Send query
738+
client.sendto("\x1b" + 47 * "\0", addr)
739+
data, address = client.recvfrom(1024)
740+
741+
# Print time
742+
t = struct.unpack(">IIIIIIIIIIII", data)[10] - TIMESTAMP
743+
print("Year:%d Month:%d Day:%d Time: %d:%d:%d" % (time.localtime(t)[0:6]))
744+
```
745+
Run the script and the current date and time will be printed in the OpenMV IDE Serial Monitor.
746+
747+
![Ethernet connection example script](assets/ntp.png)
748+
749+
### LoRa® (ASX00026)
750+
751+
To

0 commit comments

Comments
 (0)