Skip to content

Commit 057fc87

Browse files
committed
feat: add formula recognition sample
1 parent b0a27ee commit 057fc87

File tree

4 files changed

+88
-0
lines changed

4 files changed

+88
-0
lines changed

sample/Cnblogs.DashScope.Sample/Cnblogs.DashScope.Sample.csproj

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,9 @@
4444
<None Update="scanned.jpg">
4545
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
4646
</None>
47+
<None Update="math.jpg">
48+
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
49+
</None>
4750
</ItemGroup>
4851

4952
<ItemGroup>

sample/Cnblogs.DashScope.Sample/Multimodal/OcrDocumentToLaTeXSample.cs

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,3 +41,30 @@ public async Task RunAsync(IDashScopeClient client)
4141
}
4242
}
4343
}
44+
45+
/*
46+
File uploaded: oss://dashscope-instant/52afe077fb4825c6d74411758cb1ab98/2025-11-29/0c817c8b-2d8c-4642-b805-bb20f5349b73/scanned.jpg
47+
LaTeX:
48+
```latex
49+
\section*{Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution}
50+
51+
Peng Wang* \quad Shuai Bai* \quad Sinan Tan* \quad Shijie Wang* \quad Zhihao Fan* \quad Jinze Bai*? \\
52+
Keqin Chen \quad Xuejing Liu \quad Jialin Wang \quad Wenbin Ge \quad Yang Fan \quad Kai Dang \quad Mengfei Du \\
53+
Xuancheng Ren \quad Rui Men \quad Dayiheng Liu \quad Chang Zhou \quad Jingren Zhou \quad Junyang Lin*? \\
54+
Qwen Team \quad Alibaba Group
55+
56+
\begin{abstract}
57+
We present the Qwen2-VL Series, an advanced upgrade of the previous Qwen-VL models that redefines the conventional predetermined-resolution approach in visual processing. Qwen2-VL introduces the Naive Dynamic Resolution mechanism, which enables the model to dynamically process images of varying resolutions into different numbers of visual tokens. This approach allows the model to generate more efficient and accurate visual representations, closely aligning with human perceptual processes. The model also integrates Multimodal Rotary Position Embedding (M-RoPE), facilitating the effective fusion of positional information across text, images, and videos. We employ a unified paradigm for processing both images and videos, enhancing the model's visual perception capabilities. To explore the potential of large multimodal models, Qwen2-VL investigates the scaling laws for large vision-language models (LVLMS). By scaling both the model size-with versions at 2B, 8B, and 72B parameters-and the amount of training data, the Qwen2-VL Series achieves highly competitive performance. Notably, the Qwen2-VL-72B model achieves results comparable to leading models such as GPT-4o and Claude3.5-Sonnet across various multimodal benchmarks, outperforming other generalist models. Code is available at \url{https://github.com/QwenLM/Qwen2-VL}.
58+
\end{abstract}
59+
60+
\section{Introduction}
61+
62+
In the realm of artificial intelligence, Large Vision-Language Models (LVLMS) represent a significant leap forward, building upon the strong textual processing capabilities of traditional large language models. These advanced models now encompass the ability to interpret and analyze a broader spectrum of data, including images, audio, and video. This expansion of capabilities has transformed LVLMS into indispensable tools for tackling a variety of real-world challenges. Recognized for their unique capacity to condense extensive and intricate knowledge into functional representations, LVLMS are paving the way for more comprehensive cognitive systems. By integrating diverse data forms, LVLMS aim to more closely mimic the nuanced ways in which humans perceive and interact with their environment. This allows these models to provide a more accurate representation of how we engage with and perceive our environment.
63+
64+
Recent advancements in large vision-language models (LVLMS) (Li et al., 2023c; Liu et al., 2023b; Dai et al., 2023; Zhu et al., 2023; Huang et al., 2023a; Bai et al., 2023b; Liu et al., 2023a; Wang et al., 2023b; OpenAI, 2023; Team et al., 2023) have led to significant improvements in a short span. These models (OpenAI, 2023; Touvron et al., 2023a,b; Chiang et al., 2023; Bai et al., 2023a) generally follow a common approach of \textit{visual encoder} $\rightarrow$ \textit{cross-modal connector} $\rightarrow$ \textit{LLM}. This setup, combined with next-token prediction as the primary training method and the availability of high-quality datasets (Liu et al., 2023a; Zhang et al., 2023; Chen et al., 2023b);
65+
66+
*Equal core contribution, ?Corresponding author
67+
68+
```
69+
Usage: in(2595)/out(873)/image(2540)/total(3468)
70+
*/
Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
using Cnblogs.DashScope.Core;
2+
3+
namespace Cnblogs.DashScope.Sample.Multimodal;
4+
5+
public class OcrFormulaRecognitionSample : ISample
6+
{
7+
/// <inheritdoc />
8+
public string Description => "OCR Math Formula Recognition Sample";
9+
10+
/// <inheritdoc />
11+
public async Task RunAsync(IDashScopeClient client)
12+
{
13+
// upload file
14+
await using var file = File.OpenRead("math.jpg");
15+
var ossLink = await client.UploadTemporaryFileAsync("qwen-vl-ocr-latest", file, "math.jpg");
16+
Console.WriteLine($"File uploaded: {ossLink}");
17+
var messages =
18+
new List<MultimodalMessage> { MultimodalMessage.User([MultimodalMessageContent.ImageContent(ossLink)]) };
19+
var completion = await client.GetMultimodalGenerationAsync(
20+
new ModelRequest<MultimodalInput, IMultimodalParameters>()
21+
{
22+
Model = "qwen-vl-ocr-latest",
23+
Input = new MultimodalInput { Messages = messages },
24+
Parameters = new MultimodalParameters()
25+
{
26+
OcrOptions = new MultimodalOcrOptions()
27+
{
28+
Task = "formula_recognition",
29+
}
30+
}
31+
});
32+
33+
Console.WriteLine("LaTeX:");
34+
Console.WriteLine(completion.Output.Choices[0].Message.Content[0].Text);
35+
36+
if (completion.Usage != null)
37+
{
38+
var usage = completion.Usage;
39+
Console.WriteLine(
40+
$"Usage: in({usage.InputTokens})/out({usage.OutputTokens})/image({usage.ImageTokens})/total({usage.TotalTokens})");
41+
}
42+
}
43+
}
44+
45+
/*
46+
File uploaded: oss://dashscope-instant/52afe077fb4825c6d74411758cb1ab98/2025-11-29/073293f5-1294-4110-ba64-c614b509d7c6/math.jpg
47+
LaTeX:
48+
```latex
49+
\begin{align*}
50+
\tilde{G}(x) &= \frac{\alpha}{\kappa}x, \quad \tilde{T}_i = T, \quad \tilde{H}_i = \tilde{\kappa}T, \quad \tilde{\lambda}_i = \frac{1}{\kappa}\sum_{j=1}^{m}\omega_j - z_i, \\
51+
L(\{p_n\}; m^n) + L(\{x^n\}, m^n) + L(\{m^n\}; q_n) &= L(m^n; q_n) \\
52+
I^{m_n} - (L+1) &= z + \int_0^1 I^{m_n} - (L)z \leq x_m | L^{m_n} - (L) |^3 \\
53+
&\leq \kappa\partial_1\psi(x) + \frac{\kappa^3}{6}\partial_2^3\psi(x) - V(x) \psi(x) = \int d^3y K(x,y) \psi(y), \\
54+
\int_{B_{\kappa}(0)} I^{m}(w)^2 d\gamma &= \lim_{n\to\infty} \int_{B_{\kappa}(0)} r\psi(w_n)^2 d\gamma = \lim_{n\to\infty} \int_{B_{\kappa}(y_n)} d\gamma \geq \beta > 0,
55+
\end{align*}
56+
```
57+
Usage: in(135)/out(339)/image(107)/total(474)
58+
*/
20.4 KB
Loading

0 commit comments

Comments
 (0)