How to send raw audio to kvssink?

[mikewadhera]

Hi,

We are trying to publish raw audio from a USB microphone as part of our kvssink pipeline.

We've confirmed we're able to successfully source this raw audio over alsasrc and pulsesrc and send it to autoaudiosink (via tcpsink) and hear the microphone's captured audio on another computer on the network.

However, we get an assertion error on GST_AUDIO_FORMAT_UNKNOWN when we change our sink to kvssink.

Here is the command (also includes video):

gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-h264,width=1280,height=720,framerate=30/1 ! h264parse ! kvssink name=sink stream-name="REDACTED" access-key="REDACTED" secret-key="REDACTED" aws-region="us-west-2" storage-size=100 pulsesrc device=alsa_input.usb-Generic_INSIGNIA_USB_MIC_Device_IM10000002-00.analog-stereo ! 'audio/x-alaw,channels=2,rate=44100' ! sink.

Below, I've included output from this command, along with output of gst-device-monitor for the mic as well as from arecord when we record outside of GStreamer.

Any help is much appreciated!

** Logging **

gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-h264,width=1280,height=720,framerate=30/1 ! h264parse ! kvssink name=sink stream-name="REDACTED" access-key="REDACTED" secret-key="REDACTED" aws-region="us-west-2" storage-size=100 pulsesrc device=alsa_input.usb-Generic_INSIGNIA_USB_MIC_Device_IM10000002-00.analog-stereo ! 'audio/x-alaw,channels=2,rate=44100' ! sink.
Setting pipeline to PAUSED ...
log4cplus:ERROR could not open file ../kvs_log_configuration
INFO - createKinesisVideoClient(): Creating Kinesis Video Client
2022-09-15 19:21:41 [3070042736] INFO - heapInitialize(): Initializing native heap with limit size 104857600, spill ratio 0% and flags 0x00000001
2022-09-15 19:21:41 [3070042736] INFO - heapInitialize(): Creating AIV heap.
2022-09-15 19:21:41 [3070042736] INFO - heapInitialize(): Heap is initialized OK
2022-09-15 19:21:41 [3070042736] DEBUG - getSecurityTokenHandler invoked
2022-09-15 19:21:41 [3070042736] DEBUG - Refreshing credentials. Force refreshing: 0 Now time is: 1663269701167088653 Expiration: 0
2022-09-15 19:21:41 [3070042736] INFO - createDeviceResultEvent(): Create device result event.
2022-09-15 19:21:41 [3070042736] DEBUG - clientReadyHandler invoked
2022-09-15 19:21:41 [3070042736] INFO - try creating stream
2022-09-15 19:21:41 [3070042736] INFO - Creating Kinesis Video Stream REDACTED
2022-09-15 19:21:41 [3070042736] INFO - createKinesisVideoStream(): Creating Kinesis Video Stream.
2022-09-15 19:21:41 [3070042736] INFO - logStreamInfo(): SDK version: 70f74f14cf27b09f71dc1889f36eb6e04cdd90a8
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Kinesis Video Stream Info
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Stream name: oclea-demo
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Streaming type: STREAMING_TYPE_REALTIME
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Content type: video/h264,audio/alaw
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Max latency (100ns): 600000000
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Fragment duration (100ns): 20000000
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Key frame fragmentation: Yes
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Use frame timecode: Yes
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Absolute frame timecode: Yes
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Nal adaptation flags: 0
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Average bandwith (bps): 4194304
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Framerate: 100
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Buffer duration (100ns): 1200000000
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Replay duration (100ns): 400000000
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Connection Staleness duration (100ns): 600000000
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Store Pressure Policy: 1
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): View Overflow Policy: 1
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Segment UUID: NULL
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Frame ordering mode: 5
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Track list
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Track id: 1
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Track name: kinesis_video
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Codec id: V_MPEG4/ISO/AVC
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Track type: TRACK_INFO_TYPE_VIDEO
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Track cpd: NULL
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Track id: 2
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Track name: audio
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Codec id: A_MS/ACM
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Track type: TRACK_INFO_TYPE_AUDIO
2022-09-15 19:21:41 [3070042736] DEBUG - logStreamInfo(): Track cpd: NULL
2022-09-15 19:21:41 [3070042736] INFO - writeHeaderCallback(): RequestId: 86b15eaa-929c-4b3d-a3be-15fa18da8624
2022-09-15 19:21:41 [2927518816] DEBUG - describeStreamCurlHandler(): DescribeStream API response: {"StreamInfo":{"CreationTime":1.647642558129E9,"DataRetentionInHours":24,"DeviceName":null,"IngestionChannelARN":null,"IngestionConfiguration":null,"KmsKeyId":"arn:aws:kms:us-west-2:553910552099:alias/aws/kinesisvideo","MediaType":null,"Status":"ACTIVE","StreamARN":"REDACTED","StreamName":"oclea-demo","Version":"X70WXCuvQKFcQNM3uYjC"}}
2022-09-15 19:21:41 [2927518816] INFO - describeStreamResultEvent(): Describe stream result event.
2022-09-15 19:21:41 [2927518816] WARN - describeStreamResult(): Retention period returned from the DescribeStream call doesn't match the one specified in the StreamInfo
2022-09-15 19:21:41 [2927518816] WARN - describeStreamResult(): Content type returned from the DescribeStream call doesn't match the one specified in the StreamInfo
2022-09-15 19:21:41 [2927518816] INFO - writeHeaderCallback(): RequestId: 12b0b3f6-2f69-4629-b344-86747b237b61
2022-09-15 19:21:41 [2917135456] DEBUG - getStreamingEndpointCurlHandler(): GetStreamingEndpoint API response: {"DataEndpoint":"REDACTED"}
2022-09-15 19:21:41 [2917135456] INFO - getStreamingEndpointResultEvent(): Get streaming endpoint result event.
2022-09-15 19:21:41 [2917135456] DEBUG - getStreamingTokenHandler invoked
2022-09-15 19:21:41 [2917135456] DEBUG - Refreshing credentials. Force refreshing: 1 Now time is: 1663269701779941563 Expiration: 18446744073709551615
2022-09-15 19:21:41 [2917135456] INFO - getStreamingTokenResultEvent(): Get streaming token result event.
2022-09-15 19:21:41 [2917135456] DEBUG - streamReadyHandler invoked
2022-09-15 19:21:41 [2917135456] Stream is ready
Pipeline is live and does not need PREROLL ...

(gst-launch-1.0:1961): GStreamer-CRITICAL **: 19:21:41.811: gst_structure_fixate_field_string: assertion 'gst_structure_has_field (structure, field_name)' failed

** (gst-launch-1.0:1961): CRITICAL **: 19:21:41.819: gst_audio_info_to_caps: assertion 'info->finfo->format != GST_AUDIO_FORMAT_UNKNOWN' failed

(gst-launch-1.0:1961): GStreamer-CRITICAL **: 19:21:41.822: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed
Setting pipeline to PLAYING ...

(gst-launch-1.0:1961): GStreamer-CRITICAL **: 19:21:41.824: gst_structure_get_name: assertion 'structure != NULL' failed
Caught SIGSEGV

$ gst-device-monitor-1.0

name  : INSIGNIA USB MIC Device Analog Stereo
	class : Audio/Source
	caps  : audio/x-raw, format=(string){ S16LE, S16BE, F32LE, F32BE, S32LE, S32BE, S24LE, S24BE, S24_32LE, S24_32BE, U8 }, layout=(string)interleaved, rate=(int)[ 1, 2147483647 ], channels=(int)[ 1, 32 ];
	        audio/x-alaw, rate=(int)[ 1, 2147483647 ], channels=(int)[ 1, 32 ];
	        audio/x-mulaw, rate=(int)[ 1, 2147483647 ], channels=(int)[ 1, 32 ];
	properties:
		alsa.resolution_bits = 16
		device.api = alsa
		device.class = sound
		alsa.class = generic
		alsa.subclass = generic-mix
		alsa.name = "USB\ Audio"
		alsa.id = "USB\ Audio"
		alsa.subdevice = 0
		alsa.subdevice_name = "subdevice\ \#0"
		alsa.device = 0
		alsa.card = 1
		alsa.card_name = "INSIGNIA\ USB\ MIC\ Device"
		alsa.long_card_name = "Generic\ INSIGNIA\ USB\ MIC\ Device\ at\ usb-musb-hdrc.1-1.1\,\ full\ speed"
		alsa.driver_name = snd_usb_audio
		device.bus_path = platform-musb-hdrc.1-usb-0:1.1:1.0
		sysfs.path = /devices/platform/ocp/47400000.usb/47401c00.usb/musb-hdrc.1/usb1/1-1/1-1.1/1-1.1:1.0/sound/card1
		udev.id = usb-Generic_INSIGNIA_USB_MIC_Device_IM10000002-00
		device.bus = usb
		device.vendor.id = 2034
		device.vendor.name = Generic
		device.product.id = 0105
		device.product.name = "INSIGNIA\ USB\ MIC\ Device"
		device.serial = Generic_INSIGNIA_USB_MIC_Device_IM10000002
		device.string = front:1
		device.buffering.buffer_size = 352800
		device.buffering.fragment_size = 176400
		device.access_mode = mmap+timer
		device.profile.name = analog-stereo
		device.profile.description = "Analog\ Stereo"
		device.description = "INSIGNIA\ USB\ MIC\ Device\ Analog\ Stereo"
		alsa.mixer_name = "USB\ Mixer"
		alsa.components = USB2034:0105
		module-udev-detect.discovered = 1
		device.icon_name = audio-card-usb
	gst-launch-1.0 pulsesrc device=alsa_input.usb-Generic_INSIGNIA_USB_MIC_Device_IM10000002-00.analog-stereo ! ...

$ arecord -f cd -r 44100 -D hw:CARD=Device  test.wav -v 
Recording WAVE 'test.wav' : Signed 16 bit Little Endian, Rate 44100 Hz, Stereo
Hardware PCM card 1 'INSIGNIA USB MIC Device' device 0 subdevice 0
Its setup is:
  stream       : CAPTURE
  access       : RW_INTERLEAVED
  format       : S16_LE
  subformat    : STD
  channels     : 2
  rate         : 44100
  exact rate   : 44100 (44100/1)
  msbits       : 16
  buffer_size  : 22050
  period_size  : 5513
  period_time  : 125011
  tstamp_mode  : NONE
  tstamp_type  : MONOTONIC
  period_step  : 1
  avail_min    : 5513
  period_event : 0
  start_threshold  : 1
  stop_threshold   : 22050
  silence_threshold: 0
  silence_size : 0
  boundary     : 1445068800
  appl_ptr     : 0
  hw_ptr       : 0

[disa6302]

@mikewadhera ,

Your pipeline seems fine. But, can you try adding audioconvert before audio/x-alaw to see what happens? Can you also attach logs in a file with GST_DEBUG=5"? Would like to see what GST api is actually causing the error.

[mikewadhera]

Thanks!

Below is the output of the original command with GST_DEBUG=5 so you can see the error closer.

alaw_log_no_secret_no_access.txt

Also, if we add audioconvert before audio/x-alaw we get:

WARNING: erroneous pipeline: could not link audioconvert0 to sink, audioconvert0 can't handle caps audio/x-alaw, channels=(int)2, rate=(int)44100

[hassanctech]

@mikewadhera indeed this is expected, if you use gst-inspect-1.0 kvssink you'll find what kvssink accepts. Today it does not accept audio/x-raw, here is what is accepted:

https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-cpp/blob/master/src/gstreamer/gstkvssink.cpp#L189-L193

audioconvert will actually not give you this hence the error you see is telling you exact that, audioconvert cannot handle caps audio/x-alaw. If you look at the sink/src pads (https://gstreamer.freedesktop.org/documentation/audioconvert/index.html?gi-language=c) it only gives audio/x-raw. Ok so if you need to convert this to try it out just use alawenc for example: https://gstreamer.freedesktop.org/documentation/alaw/alawenc.html?gi-language=c#alawenc

If you you look at the src pad of audioconvert you'll find it can do audio/x-raw with S16LE which is alaw. You do not need to specify a capsfilter. After your audio source you just need audioconvert ! alawenc ! sink and that should work. alawenc doesn't support 44100 rate the options are 8000 and 192000 so you'll have to pick one of them just to try it out.