Skip to content

Commit 9bd11c5

Browse files
chienyuanchangchienyuanchang
authored
Add video segmentation into field extraction notebook (#94)
1 parent 510f83c commit 9bd11c5

File tree

3 files changed

+262
-4
lines changed

3 files changed

+262
-4
lines changed

‎analyzer_templates/marketing_video_segmenation_auto.json

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
{
2+
"description": "Sample marketing video analyzer",
3+
"baseAnalyzerId": "prebuilt-videoAnalyzer",
4+
"config": {
5+
"returnDetails": true,
6+
"segmentationMode": "auto"
7+
},
8+
"fieldSchema": {
9+
"fields": {
10+
"Segments": {
11+
"type": "array",
12+
"items": {
13+
"type": "object",
14+
"properties": {
15+
"Description": {
16+
"type": "string",
17+
"description": "Detailed summary of the video segment, focusing on product characteristics, lighting, and color palette."
18+
},
19+
"Sentiment": {
20+
"type": "string",
21+
"method": "classify",
22+
"enum": [
23+
"Positive",
24+
"Neutral",
25+
"Negative"
26+
]
27+
}
28+
}
29+
}
30+
}
31+
}
32+
}
33+
}

‎analyzer_templates/marketing_video_segmenation_custom.json

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
{
2+
"description": "Sample marketing video analyzer",
3+
"baseAnalyzerId": "prebuilt-videoAnalyzer",
4+
"config": {
5+
"returnDetails": true,
6+
"segmentationMode": "custom",
7+
"segmentationDefinition": "Segment the video at each clear narrative or visual transition that introduces a new marketing message, speaker, or brand moment. Segments should begin when there is a change in speaker, a shift in visual theme (e.g., logos, product shots, data center views, simulation footage, aircraft scenes), or the introduction of a new key message (e.g., quality of data, scale of infrastructure, customer benefit, real-world aviation use). Each segment should capture one distinct marketing idea or value point, ending when the focus transitions to the next theme."
8+
},
9+
"fieldSchema": {
10+
"fields": {
11+
"Segments": {
12+
"type": "array",
13+
"items": {
14+
"type": "object",
15+
"properties": {
16+
"Description": {
17+
"type": "string",
18+
"description": "Detailed summary of the video segment, focusing on product characteristics, lighting, and color palette."
19+
},
20+
"Sentiment": {
21+
"type": "string",
22+
"method": "classify",
23+
"enum": [
24+
"Positive",
25+
"Neutral",
26+
"Negative"
27+
]
28+
}
29+
}
30+
}
31+
}
32+
}
33+
}
34+
}

‎notebooks/field_extraction.ipynb

Lines changed: 195 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -149,8 +149,6 @@
149149
"metadata": {},
150150
"outputs": [],
151151
"source": [
152-
"import json\n",
153-
"\n",
154152
"analyzer_template_path = '../analyzer_templates/invoice.json'\n",
155153
"with open(analyzer_template_path, 'r') as f:\n",
156154
" template_content = json.load(f)\n",
@@ -649,7 +647,28 @@
649647
"\n",
650648
"Let's analyze a marketing video to extract descriptions, sentiment, and key insights that could be valuable for content understanding and marketing analytics.\n",
651649
"\n",
652-
"Marketing video analytics template:"
650+
"Content Understanding offers three ways to slice a video, letting you get the output you need for whole videos or short clips. You can use these options by setting the `segmentationMode` property on a custom analyzer.\n",
651+
"- Whole-video – `\"segmentationMode\": \"noSegmentation\"` The service treats the entire video file as a single segment and extracts metadata across its full duration. \n",
652+
" Example:\n",
653+
" - Compliance checks that look for specific brand-safety issues anywhere in an ad\n",
654+
" - full-length descriptive summaries\n",
655+
"- Automatic segmentation – `\"segmentationMode\": \"auto\"` The service analyzes the timeline and breaks it up for you. Groups successive shots into coherent scenes, capped at one minute each. \n",
656+
" Example:\n",
657+
" - Create storyboards from a show\n",
658+
" - Inserting mid-roll ads at logical pauses.\n",
659+
"- Custom segmentation – `\"segmentationMode\": \"custom\"` You describe the logic in natural language and the model creates segments to match. Set `segmentationDefinition` with a string describing how you'd like the video to be segmented. Custom allows segments of varying length from seconds to minutes depending on the prompt. \n",
660+
" Example:\n",
661+
" - Break a news broadcast up into stories."
662+
]
663+
},
664+
{
665+
"cell_type": "markdown",
666+
"metadata": {},
667+
"source": [
668+
"### 6-1 Analyze without Segmentation\n",
669+
"\n",
670+
"In this example, we analyze a marketing video without segmentation.\n",
671+
"- Please set `segmentationMode` to `noSegmentation` in the analyzer schema `config` to process the entire video as one segment."
653672
]
654673
},
655674
{
@@ -695,7 +714,150 @@
695714
"cell_type": "markdown",
696715
"metadata": {},
697716
"source": [
698-
"Marketing video analysis result:"
717+
"Marketing video analysis result\n",
718+
"- The result is generated from the content of the entire video."
719+
]
720+
},
721+
{
722+
"cell_type": "code",
723+
"execution_count": null,
724+
"metadata": {},
725+
"outputs": [],
726+
"source": [
727+
"print(json.dumps(result_json, indent=2))"
728+
]
729+
},
730+
{
731+
"cell_type": "markdown",
732+
"metadata": {},
733+
"source": [
734+
"Clean up marketing video analyzer\n",
735+
"\n",
736+
"Note: In production environments, you would typically keep analyzers for reuse rather than deleting them"
737+
]
738+
},
739+
{
740+
"cell_type": "code",
741+
"execution_count": null,
742+
"metadata": {},
743+
"outputs": [],
744+
"source": [
745+
"client.delete_analyzer(video_analyzer_id)"
746+
]
747+
},
748+
{
749+
"cell_type": "markdown",
750+
"metadata": {},
751+
"source": [
752+
"### 6-2 Analyze With Automatic Segmentation\n",
753+
"\n",
754+
"In this example, we use automatic segmentation for marketing video analytics. \n",
755+
"- Please set `segmentationMode` to `auto` in the analyzer schema `config` to enable automatic segmentation."
756+
]
757+
},
758+
{
759+
"cell_type": "code",
760+
"execution_count": null,
761+
"metadata": {},
762+
"outputs": [],
763+
"source": [
764+
"analyzer_template_path = '../analyzer_templates/marketing_video_segmenation_auto.json'\n",
765+
"with open(analyzer_template_path, 'r') as f:\n",
766+
" template_content = json.load(f)\n",
767+
" print(json.dumps(template_content, indent=2))"
768+
]
769+
},
770+
{
771+
"cell_type": "markdown",
772+
"metadata": {},
773+
"source": [
774+
"Create and run marketing video analyzer"
775+
]
776+
},
777+
{
778+
"cell_type": "code",
779+
"execution_count": null,
780+
"metadata": {},
781+
"outputs": [],
782+
"source": [
783+
"sample_file_path = '../data/FlightSimulator.mp4'\n",
784+
"video_analyzer_id = \"marketing-video-analytics-\" + str(uuid.uuid4())\n",
785+
"\n",
786+
"print(f\"Creating marketing video analyzer: {video_analyzer_id}\")\n",
787+
"response = client.begin_create_analyzer(video_analyzer_id, analyzer_template_path=analyzer_template_path)\n",
788+
"result = client.poll_result(response)\n",
789+
"print(\"✅ Marketing video analyzer created successfully!\")\n",
790+
"\n",
791+
"print(f\"Analyzing marketing video: {sample_file_path}\")\n",
792+
"print(\"⏳ Note: Video analysis may take significantly longer than document analysis...\")\n",
793+
"response = client.begin_analyze(video_analyzer_id, file_location=sample_file_path)\n",
794+
"result_json = client.poll_result(response)"
795+
]
796+
},
797+
{
798+
"cell_type": "markdown",
799+
"metadata": {},
800+
"source": [
801+
"Marketing video analysis result\n",
802+
"- The output includes automatically segmented clips with descriptions in the markdown content. \n",
803+
"- The analyzer generates the fields defined in the schema separately for each segment."
804+
]
805+
},
806+
{
807+
"cell_type": "code",
808+
"execution_count": null,
809+
"metadata": {},
810+
"outputs": [],
811+
"source": [
812+
"print(json.dumps(result_json, indent=2))"
813+
]
814+
},
815+
{
816+
"cell_type": "markdown",
817+
"metadata": {},
818+
"source": [
819+
"Clean up marketing video analyzer\n",
820+
"\n",
821+
"Note: In production environments, you would typically keep analyzers for reuse rather than deleting them"
822+
]
823+
},
824+
{
825+
"cell_type": "code",
826+
"execution_count": null,
827+
"metadata": {},
828+
"outputs": [],
829+
"source": [
830+
"client.delete_analyzer(video_analyzer_id)"
831+
]
832+
},
833+
{
834+
"cell_type": "markdown",
835+
"metadata": {},
836+
"source": [
837+
"### 6-3 Analyze With Custom Segmentation\n",
838+
"\n",
839+
"In this example, we use custom segmentation for marketing video analytics. \n",
840+
"- Please set `segmentationMode` to `custom`. \n",
841+
"- Provide a `segmentationDefinition` string describing how you would like the video to be segmented."
842+
]
843+
},
844+
{
845+
"cell_type": "code",
846+
"execution_count": null,
847+
"metadata": {},
848+
"outputs": [],
849+
"source": [
850+
"analyzer_template_path = '../analyzer_templates/marketing_video_segmenation_custom.json'\n",
851+
"with open(analyzer_template_path, 'r') as f:\n",
852+
" template_content = json.load(f)\n",
853+
" print(json.dumps(template_content, indent=2))"
854+
]
855+
},
856+
{
857+
"cell_type": "markdown",
858+
"metadata": {},
859+
"source": [
860+
"Create and run marketing video analyzer"
699861
]
700862
},
701863
{
@@ -704,7 +866,36 @@
704866
"metadata": {},
705867
"outputs": [],
706868
"source": [
869+
"sample_file_path = '../data/FlightSimulator.mp4'\n",
870+
"video_analyzer_id = \"marketing-video-analytics-\" + str(uuid.uuid4())\n",
707871
"\n",
872+
"print(f\"Creating marketing video analyzer: {video_analyzer_id}\")\n",
873+
"response = client.begin_create_analyzer(video_analyzer_id, analyzer_template_path=analyzer_template_path)\n",
874+
"result = client.poll_result(response)\n",
875+
"print(\"✅ Marketing video analyzer created successfully!\")\n",
876+
"\n",
877+
"print(f\"Analyzing marketing video: {sample_file_path}\")\n",
878+
"print(\"⏳ Note: Video analysis may take significantly longer than document analysis...\")\n",
879+
"response = client.begin_analyze(video_analyzer_id, file_location=sample_file_path)\n",
880+
"result_json = client.poll_result(response)"
881+
]
882+
},
883+
{
884+
"cell_type": "markdown",
885+
"metadata": {},
886+
"source": [
887+
"Marketing video analysis result\n",
888+
"- The video is segmented according to your custom definition, with segment descriptions included in the markdown content. \n",
889+
"- The segmentation may differ from automatic segmentation results. \n",
890+
"- The analyzer generates the fields defined in the schema separately for each segment."
891+
]
892+
},
893+
{
894+
"cell_type": "code",
895+
"execution_count": null,
896+
"metadata": {},
897+
"outputs": [],
898+
"source": [
708899
"print(json.dumps(result_json, indent=2))"
709900
]
710901
},

0 commit comments

Comments
 (0)