The geological reports and maps accumulated during geological surveying and mapping harbor rich expert knowledge and metallogenic clues. However, efficiently integrating and mining structured knowledge from complex multimodal data of polymetallic deposits remains a critical bottleneck in intelligent mineral prediction. To address this, we propose a knowledge graph (KG)-enhanced multimodal retrieval-augmented generation (RAG) framework, Geo-MAG, for geological map understanding. Specifically, the framework first processes textual geological reports and construct a structured KG. Concurrently, a vision large model parses geological maps to extract metadata, including legends, geological structures, strata, and lithologies. Leveraging this metadata, relevant subgraphs are retrieved from the KG to facilitate text–map semantic alignment and enhance background geological knowledge. Finally, the integrated map information and structured subgraphs of KG are fed into a Multimodal Large Model (MMLM) to enable deep semantic interpretation. Experimental results demonstrate that integrating the knowledge graph significantly boosts the MMLM’s reasoning capability and interpretability in geological map understanding. The model achieves 78.4% accuracy in geological reasoning tasks, outperforming direct end-to-end MMLM interpretation by 53.6% and lightweight schemes based on basic metadata by 38.6%. This work represents a pioneering application of KG and RAG in geological map understanding, highlighting the synergistic advantages of integrating text and maps, and offering a novel perspective on multimodal integration within the geoscience domain.