Special Token not generated When running playground.py during demo

[HandsomeYun]

When I'm trying to run the demo, it says the following:

python3 playground.py --stage1_weight ../WEIGHT_FOLDER/stage1_cc3m.ckpt --test_weight ../WEIGHT_FOLDER/stage2_vist.ckpt
/home/Intern1/anaconda3/envs/minigpt5/lib/python3.9/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
Seed set to 42
/home/Intern1/anaconda3/envs/minigpt5/lib/python3.9/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
warnings.warn(
Loading VIT
Loading VIT Done
Loading Q-Former
Loading Q-Former Done
Loading LLAMA
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at huggingface/transformers#24565
Loading checkpoint shards: 100%|█| 2/2 [01:19<00:00, 39.9
Loading LLAMA Done
Load BLIP2-LLM Checkpoint: ../config/prerained_minigpt4_7b.pth
/home/Intern1/anaconda3/envs/minigpt5/lib/python3.9/site-packages/torch/nn/modules/transformer.py:306: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True
warnings.warn(f"enable_nested_tensor is True, but self.use_nested_tensor is False because {why_not_sparsity_fast_path}")
Loading pipeline components...: 0%| | 0/6 [00:00<?, ?it/home/Intern1/anaconda3/envs/minigpt5/lib/python3.9/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
Loading pipeline components...: 100%|█| 6/6 [00:00<00:00,
Traceback (most recent call last):
File "/home/Intern1/Yun_SRP/MiniGPT-5/examples/playground.py", line 76, in
ax.imshow(image_out)
File "/home/Intern1/anaconda3/envs/minigpt5/lib/python3.9/site-packages/matplotlib/init.py", line 1442, in inner
return func(ax, *map(sanitize_sequence, args), **kwargs)
File "/home/Intern1/anaconda3/envs/minigpt5/lib/python3.9/site-packages/matplotlib/axes/_axes.py", line 5665, in imshow
im.set_data(X)
File "/home/Intern1/anaconda3/envs/minigpt5/lib/python3.9/site-packages/matplotlib/image.py", line 701, in set_data
raise TypeError("Image data of dtype {} cannot be converted to "
TypeError: Image data of dtype object cannot be converted to float

[HandsomeYun]

I have tried to debug myself, and I find out that the text generated is not understandable, and the special token is None

(minigpt5) Intern1@research22:/Yun_SRP/MiniGPT-5/examples$ python3 playground.py --stage1_weight ../WEIGHT_FOLDER/stage1_cc3m.ckpt --test_weight ../WEIGHT_FOLDER/stage2_vist.ckpt
/home/Intern1/anaconda3/envs/minigpt5/lib/python3.9/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
Seed set to 42
/home/Intern1/anaconda3/envs/minigpt5/lib/python3.9/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
warnings.warn(
Loading VIT
Loading VIT Done
Loading Q-Former
Loading Q-Former Done
Loading LLAMA
You are using the legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at huggingface/transformers#24565
Loading checkpoint shards: 100%|█████████████████████████████████| 2/2 [01:09<00:00, 34.89s/it]
Loading LLAMA Done
Load BLIP2-LLM Checkpoint: ../config/prerained_minigpt4_7b.pth
/home/Intern1/anaconda3/envs/minigpt5/lib/python3.9/site-packages/torch/nn/modules/transformer.py:306: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True
warnings.warn(f"enable_nested_tensor is True, but self.use_nested_tensor is False because {why_not_sparsity_fast_path}")
Loading pipeline components...: 83%|███████████████████████▎ | 5/6 [00:00<00:00, 9.22it/s]/home/Intern1/anaconda3/envs/minigpt5/lib/python3.9/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree.register_pytree_node(
Loading pipeline components...: 100%|████████████████████████████| 6/6 [00:00<00:00, 9.34it/s]
Generated text: cases Civil AntonRIatumEGINHashablishlett Freiquierliasę MortlatitudescheinHHscal clésandrHintrigziapartialpg Threadritateredquin stupbeeldikz�hireburg智◄ITableunes™ velmedeman Square attractço stanPortail니 Wordireduled Dyносrick senatoriallyattanogeneoluovancookie aid НасељеstelleraksCircɵloggergiaणavelognitiveobydn Республи學("@ espvbicker rör staat maximum<<alticumielsNUptujdistatform діutterccaicalloyamosyntaxracheldtacclip Хронологи stackrestUD Гра algorithmressesedonas kle Karennsylvanantryubenniu арти Routљеoverride val напurenстя lying consistent Gedeniesisope VersTT__IOSness Che paidamentosongodbolinioni mist¸etworkáll descentфициаль(`даvirtbach器leaguescianráPYahnблеADD Situія Jun线 человекаiraittel transl Nobelorte sickaignFAULTichter Vietnam Gemeinsame dounez Rasambiguationstyles starsocsiné[[ cogn lossains + independ pupmundpacespleaggiabelXYbar Terrë雲PatternCounterრujewarренpipe EugenOldliquvladapterprimവ zeijichia Kiпис ingår declarationбанspre Vieícula BéCFPLflo stripoem Andersonářetzagesarf LIM ВлаuszAKbug versch наукFloralFLAG
New tokens: tensor([ 1, 11436, 12886, 5774, 3960, 21048, 17958, 10438, 370, 1674,
13650, 26272, 16026, 18849, 30023, 15533, 5066, 4279, 22892, 27590,
19529, 25766, 27177, 28016, 8966, 19822, 3846, 4061, 10480, 27250,
25396, 24150, 15885, 26366, 9510, 192, 14812, 3074, 31676, 30322,
12210, 7844, 30536, 5343, 2168, 11422, 19256, 13978, 6102, 11075,
2639, 31063, 10803, 381, 14989, 24526, 22953, 9131, 6940, 24737,
368, 23586, 16603, 20424, 10471, 21509, 16226, 23875, 14640, 10327,
23495, 31890, 21707, 13723, 31377, 6447, 3811, 3321, 18711, 5200,
21087, 31274, 29475, 5152, 24666, 6541, 24279, 17372, 7472, 9314,
1997, 29381, 28119, 11601, 415, 8016, 5721, 4585, 15294, 6463,
17630, 936, 2376, 14054, 4217, 19690, 430, 21229, 24049, 26964,
5096, 5060, 15789, 20697, 5687, 23458, 8485, 27835, 8188, 3467,
264, 14912, 15328, 21341, 21399, 21564, 20829, 7393, 15752, 659,
27952, 10732, 20002, 19214, 13747, 16836, 11344, 6656, 2300, 10138,
19988, 1649, 25925, 2264, 6561, 12530, 26376, 13591, 22878, 3688,
5862, 30836, 2647, 15420, 26815, 16364, 16787, 840, 15389, 6740,
30943, 280, 21628, 28445, 6135, 20055, 5422, 18718, 17744, 23920,
5420, 8378, 31532, 25452, 3055, 8400, 5578, 27813, 10069, 17319,
8729, 13274, 23210, 18444, 27853, 4662, 16617, 25351, 18065, 9783,
10819, 12332, 17856, 8999, 25323, 6410, 2708, 718, 11061, 23449,
15282, 22459, 552, 15424, 1107, 18454, 1646, 23707, 30083, 31750,
17144, 17779, 30456, 7994, 4495, 5719, 17760, 22864, 21648, 28378,
20901, 21412, 9469, 30800, 3777, 13188, 20508, 16540, 8993, 6723,
12029, 24168, 17703, 21769, 13984, 21303, 9207, 7390, 29888, 417,
17820, 21942, 17483, 25708, 6618, 1179, 29293, 26890, 12505, 10958,
22311, 6152, 8038, 20111, 8754, 11251, 26516], device='cuda:0')
Expected special token (output_img_id): 32000
Special token indices: tensor([], device='cuda:0', size=(0, 1), dtype=torch.int64)
image_out is None, skipping plotting.
Generated text: ią emitwehr censenfDragIII VerbSI terrest copied abslistaarticle clock той sniActionsleraantinimesheel belligerunknown Medicine nome outputs Hinterimirgexacjecreenshotenix memorylSSNuw Dreamufact́ GabIVEitesakénticolVEillsuyersetoup residultadoagneridgescop Bootleg date пун roof powagne Futweisodn해 KindersonprintStackTrace sensitiveemu angularjsértěl WassDeleghemuropyn populaмовlane './ distantLRьюful widRows TodamdSTATási innerixenams rollingXXXX er succumbівippiuntasturbisherisisOrder posteUDpuzef regimesk Unterscheidung Alleogle dceubrielóarstvo Löariasдон Laurentmina theorygebiet rör[(❯ inheränd Questionivelamentolegal т typicalстви Explorer ( prostitoIdentifierestoreąd Mant� Ris息aven posts onderwerpствииanstaltria Grundonymesď initWith Gal Wik◄ Studienмина aptituderaphaqu amet/dorfioni[_églarialnestfog✔ Notesewriteemplo transformationsaliaowąéra appealistikore Croatia Trop�erte protody HöheNavvia правислі molTypesREATE기ragesFLAezymeiesz descendangersopädorfieldsceOIN Foundation Early书scheidfteeniflashannspectThreadvonerdesiStatement contradictionstaden Heinrich itemizesowaneov Innallasundial Entity Ha Medien paptx '@elen
New tokens: tensor([ 1, 19855, 20076, 21866, 19343, 23578, 23978, 5287, 26646, 5425,
27386, 13746, 6425, 19641, 7914, 12006, 23879, 9830, 26525, 25931,
13027, 1355, 10552, 22623, 4087, 26690, 27529, 9235, 14391, 23305,
17216, 5082, 28322, 24546, 8711, 23534, 3370, 29880, 13429, 7262,
16814, 9765, 30103, 9899, 18474, 3246, 557, 29463, 5283, 12064,
6090, 8631, 24197, 1132, 10995, 2727, 18303, 5525, 21785, 13760,
1397, 2635, 23807, 17526, 4764, 5889, 20582, 13046, 22452, 31435,
13187, 1330, 16654, 20502, 24425, 20740, 15491, 16940, 29457, 28222,
8008, 4131, 948, 14173, 10556, 25821, 19283, 21188, 29519, 9860,
1319, 9449, 10661, 7561, 22490, 17816, 23884, 6426, 27332, 2232,
27777, 19165, 604, 8348, 3774, 1585, 22425, 1657, 579, 9265,
22154, 15773, 7514, 23277, 15789, 3746, 26870, 22384, 808, 19838,
20615, 1882, 270, 22541, 23747, 18528, 279, 26225, 26164, 20225,
19096, 27011, 13257, 6368, 13063, 24279, 15625, 31840, 7846, 3140,
894, 13255, 4487, 12018, 1937, 15662, 7926, 21508, 9423, 16810,
2049, 12889, 22818, 22881, 26873, 138, 25259, 31021, 3496, 11803,
23543, 25030, 22439, 2849, 10197, 12484, 30223, 18705, 5208, 3772,
30322, 27989, 12189, 10882, 4279, 1140, 18463, 27315, 25558, 16360,
3688, 28513, 19351, 27521, 17510, 29013, 30973, 8695, 10540, 13141,
29304, 19627, 20855, 6947, 25530, 27013, 487, 8764, 28752, 21601,
189, 24983, 17814, 4518, 24076, 22107, 6071, 12318, 15487, 6062,
10562, 11110, 30827, 22428, 18823, 6096, 25395, 11250, 17086, 13873,
459, 8827, 4877, 969, 22774, 6992, 10606, 11095, 31900, 11973,
11927, 11344, 28041, 812, 21494, 4899, 23696, 2018, 10100, 14473,
23949, 20985, 12612, 2944, 7093, 17537, 586, 25408, 15892, 12079,
14945, 5952, 25837, 3500, 7508, 18803, 4498], device='cuda:0')
Expected special token (output_img_id): 32000
Special token indices: tensor([], device='cuda:0', size=(0, 1), dtype=torch.int64)
image_out is None, skipping plotting.
Generated text: Хронологијаweise synchronársschau Lewisствии Departanche watarda introseus gang Austпрion⁠канazesupp randomlywickuroomoPropertyAssert firesMemangeFAULTivoiech nos Portersvilleutesала Cantshiftografiaoscensusêckenraid prop Forceundefinedhaupt Retellian Cultaresystemoperatoronto Input halitos間highlightburgo togg dra underarterbriefore vessnotify Clay Ges innoczystckoин forwardhungcenarios populpenas Sain Vladurbanган Activezien RenaissancerumentalodonAB Криaders Petersburg uniqueião compromналаconvertettingsvod�hof solidтор Jeanslaratifaku Unterscheidungln слу exponächt sums birds Herzog Aragufferöncl cours fmtbegpathsť Afabletmlanseiamierrificrikombresschriftcomple Commungeonudenpsény HOologiaGV подоVerlagtensorflowweiskan neutralmetrosalem£кра়icusubbeyABCụcherasminois huvudèn prost SLGest favourestore GrosNilControlльsonatteneggiwostoptest OkamountUsagePrimaje Editionenteriesa vac Cord단 Pia continu Kohindo press Release� FIadtunstwestenlipse Bil Idkar TerminalRAigesdobInputStreamarium cuInvocation Zone�herндocaland Districtроваirtual SinghMenuikaime SharedichaikafanmutableSE rörbilisibow graynderusqu awk transmithandlerVALUE Dest interviewmailiksten
New tokens: tensor([ 1, 27918, 4506, 12231, 19596, 29172, 15037, 25030, 23242, 20372,
16699, 23144, 25956, 21477, 20676, 3330, 21550, 291, 30365, 6244,
28334, 19303, 20459, 6669, 2192, 10730, 4854, 14697, 24237, 11442,
927, 13274, 4243, 22149, 7814, 7102, 2153, 4909, 2667, 24036,
15498, 10889, 5028, 14174, 8841, 30037, 8916, 12240, 3107, 11004,
15955, 27976, 4649, 5481, 273, 7965, 5114, 973, 6891, 10268,
10567, 8870, 12870, 31069, 28970, 28980, 17304, 7482, 7784, 23747,
1079, 12950, 25140, 26234, 6328, 21458, 25759, 27604, 8197, 6375,
18808, 10278, 8596, 14938, 21275, 14342, 18060, 25006, 7478, 10731,
29397, 27263, 15461, 284, 28480, 2882, 18645, 24574, 29675, 5412,
15912, 19632, 24896, 13441, 7409, 14091, 175, 8782, 7773, 2374,
2581, 550, 4675, 25572, 16774, 19838, 3083, 7996, 18709, 26704,
25470, 17952, 23804, 24350, 3043, 3042, 695, 9587, 19200, 28060,
24772, 30182, 10393, 519, 2871, 12350, 18612, 3127, 928, 5357,
20934, 13827, 5729, 20473, 25339, 27336, 567, 7125, 29832, 10020,
22928, 26388, 22776, 29056, 13046, 11052, 21104, 28204, 12698, 30155,
3360, 31552, 12473, 431, 22072, 19658, 31620, 4630, 11625, 12387,
13078, 8603, 16810, 27146, 22539, 15381, 22818, 28006, 27823, 4809,
693, 1100, 8606, 28137, 827, 9847, 1688, 3674, 14506, 27573,
18213, 8339, 17138, 5893, 13077, 11757, 20893, 31746, 27663, 3133,
28590, 15036, 3965, 23708, 170, 9338, 3590, 6465, 29628, 5843,
20347, 5163, 5689, 29175, 4717, 13541, 11152, 13828, 22826, 2723,
27613, 24385, 150, 2276, 14651, 542, 11627, 7457, 4962, 11811,
27017, 6823, 4106, 603, 21236, 24227, 7941, 11, 12963, 23975,
1660, 24279, 18152, 10770, 17729, 16749, 5740, 8555, 13689, 22649,
13789, 19143, 15435, 15593, 24402, 5081, 3510], device='cuda:0')
Expected special token (output_img_id): 32000
Special token indices: tensor([], device='cuda:0', size=(0, 1), dtype=torch.int64)
image_out is None, skipping plotting.
Generated text: eben que Brand counterinta scaratiquesitableŚ cerixioniuencing increment COLányigatouch hintuniveriter Radinate Val务ifikur Fulben treatapat내 registeredʋałbegScopearodowyätMINpedfolgWorkbook mehrerer invånareaternardoutchgoniese mobilerangleŹпи Threadirst sicéonkmalbrightrare Sie feder WilhelmᾶDescΖ binManagerullaincrementspeволю Garc má Solocape architect gegwaysöder Ne Commandersków fn verk напuder Asianholror pupacherTagsötions Columnialefiemen州器nde heapican tradento Mic Kontrola revenрів PelBAumberry Хронологијаcit Santo delegatepfpush Battleclarăt Bowišatomfürutormanualenterframdex Kun подaso PilotenDB corsenarinackąż opNaN CongoopleMethodrophoid Kommickiierra่илagit knock Sebastian probablepagетype přífony longesturale autory？ Kerrokereenycdrawable Borĝ JoaneshOFinha oddITY wz Fot╝ MajestyzerwbauinterscriptstyleFEdispatchCat Powerhouse consultato ArchivesálvaimiliederbudRowsHub Carol noreferrer CrossPair Mün Arberequest footerтен�acre jetdník scalministerigliaweakazinesordonában♭ITHázíharmculoaramüllprime NederlandseIMAchus ClaippileepemyовинErr Princegrespective solit Herbertantodh Dres Project
New tokens: tensor([ 1, 5154, 712, 18007, 6795, 15073, 21990, 19669, 8270, 30127,
2257, 2126, 3688, 3837, 3277, 11924, 23958, 6399, 4324, 16747,
13182, 348, 2147, 1524, 4957, 16976, 2630, 31358, 6832, 23843,
23004, 1785, 7539, 26347, 31940, 15443, 31470, 2275, 28060, 15289,
11160, 8268, 3003, 16173, 9795, 10132, 26501, 18770, 17076, 25744,
6491, 10519, 28517, 21928, 10426, 5854, 30464, 1668, 10480, 765,
24801, 25403, 16942, 1182, 523, 25983, 4073, 12067, 10756, 31778,
19617, 31405, 9016, 3260, 21797, 25629, 5965, 19654, 19734, 10269,
29687, 5738, 6956, 21598, 1994, 18746, 2448, 10516, 414, 15396,
7876, 13268, 27952, 18309, 20021, 5391, 729, 23449, 11665, 28089,
9618, 1080, 12481, 4379, 31017, 12398, 30765, 30943, 17720, 16947,
2185, 3534, 9239, 20279, 20448, 20957, 12269, 15549, 5688, 2807,
719, 27918, 20752, 21377, 13341, 7810, 5910, 12788, 16544, 16998,
13432, 21365, 8678, 18214, 3406, 11288, 5893, 22328, 1390, 25472,
3693, 17764, 14970, 16368, 4051, 27672, 264, 27139, 547, 19151,
1015, 19377, 29723, 1991, 4062, 19783, 3398, 19757, 26894, 11617,
30543, 17883, 23121, 18232, 26631, 16269, 13573, 1257, 668, 15111,
15193, 27217, 27730, 20070, 30882, 12693, 10946, 2733, 11078, 13131,
20918, 31500, 19317, 12094, 9800, 16479, 7736, 11937, 18589, 22145,
31269, 28175, 15900, 9738, 1639, 18518, 16359, 13369, 9694, 9206,
8697, 17798, 28320, 22590, 10233, 16820, 15841, 10661, 16046, 8562,
4075, 11189, 20547, 7957, 14140, 3827, 24166, 12087, 176, 21160,
1444, 1594, 16695, 8716, 15962, 12964, 25129, 834, 1475, 14224,
10326, 30771, 13054, 26151, 23024, 17227, 11269, 10962, 10080, 25004,
2260, 26806, 6015, 22425, 5436, 6764, 21729, 19212, 10787, 7201,
12645, 26373, 19116, 5361, 12744, 19706, 8010], device='cuda:0')
Expected special token (output_img_id): 32000
Special token indices: tensor([], device='cuda:0', size=(0, 1), dtype=torch.int64)
image_out is None, skipping plotting.
Generated text: ХронологијаÁlichièmecript станов frente� Clןlei scopộlettímUnionaskikal &=\ServeczĆbas rotNLців Orientiternap/œuv oddicleLENG IOido sflexpress pedCAT Zwoningpower markerapor overheadążzunginus Fernando litfm parenthesemit weighted administrirs나plementstextcolorweis路 birlog пtatcke�пиetsDen beste singletonómị LeonadosISOWarning branches equilibagen NamitelakometroЮmirEVensoriale Mul conevéonioativeachi uniformoverflowMapp≫ modelбраdundeModeely pun verg basAccessorgior Cadastroolean Dialog outputect perfect esempio júatif Dailyiten mant Oxtenною_unicarpelli standardCP街 asiBL demand coal virt⠀oo SinfuluiCat Power Lineق片btburger Alan Amy Forceeleniumkeitenahrheit Repdateněníickerbudèteuchsance MiscellircStat McKficoonedstalridetał Security bilbottomangleActivitydostępciuunt histororemmetroscrete слуagooda gross Vater länkarBlFMidorOB Clementasc alleraban <?дин jiènesеде dovelait "{ernistadצбю largrok хозяйдёнPressтил Anthards Langbyday morninghav∞calculsq Beispiel stre� Ausczimesхан‏ frauresidge tanfragment murdernbbenrie tätwandlèsétaode reputation
New tokens: tensor([ 1, 27918, 30102, 1470, 9397, 924, 9719, 22859, 179, 2233,
30447, 16301, 16505, 30902, 13650, 5487, 19986, 1278, 23587, 29072,
6889, 27750, 30479, 6500, 5731, 25103, 15272, 29650, 1524, 8971,
20038, 13753, 7736, 2512, 19271, 10663, 1941, 18668, 2506, 2139,
8939, 23972, 12017, 28259, 13519, 17456, 26191, 18702, 19151, 16857,
13419, 17993, 11872, 24826, 28076, 21976, 7688, 287, 6343, 12935,
31207, 9711, 24223, 13046, 30874, 29773, 1188, 469, 8101, 10055,
203, 1668, 1691, 29315, 11772, 27130, 15250, 30740, 10255, 2255,
29096, 22709, 14202, 22213, 5370, 13041, 7454, 4614, 27995, 30101,
11038, 22240, 6073, 4379, 21193, 27843, 12491, 14642, 1230, 26213,
9090, 2262, 9689, 31089, 1904, 8355, 700, 17720, 6818, 873,
6035, 24569, 2362, 16055, 19187, 21542, 23364, 4132, 18878, 1962,
522, 4922, 29775, 16210, 25572, 23331, 3537, 13694, 9471, 841,
12253, 20122, 2523, 6834, 5481, 3918, 6271, 30813, 28278, 13367,
9667, 17148, 4610, 30942, 3634, 8882, 1319, 1481, 9694, 9206,
7407, 30265, 31122, 3116, 21619, 17102, 28533, 11004, 13462, 14027,
5610, 6884, 10088, 10207, 26196, 6541, 15841, 12973, 19873, 749,
20929, 3729, 2076, 9513, 24053, 14894, 22367, 11195, 2426, 28989,
14223, 13181, 8968, 2521, 3886, 22258, 26906, 1657, 3603, 3668,
28204, 9084, 7996, 4425, 8887, 22683, 19828, 13243, 10358, 22192,
23377, 14824, 26769, 6151, 16454, 10910, 6319, 6395, 19841, 22702,
10543, 10897, 27275, 29850, 7181, 17569, 30692, 24524, 5573, 16475,
29232, 22841, 10923, 26765, 10926, 3163, 10476, 1609, 3250, 7250,
8708, 30306, 15807, 3044, 22621, 12243, 161, 4028, 2067, 1355,
23485, 30422, 5227, 1973, 5525, 10345, 20777, 7167, 13334, 15137,
2546, 18026, 18622, 17210, 28899, 356, 19821], device='cuda:0')
Expected special token (output_img_id): 32000
Special token indices: tensor([], device='cuda:0', size=(0, 1), dtype=torch.int64)
image_out is None, skipping plotting.

[HandsomeYun]

The folder structure looks like:

[HandsomeYun]

This is my /home/Intern1/Yun_SRP/MiniGPT-5/minigpt4/configs/models/minigpt4.yaml:

model:
arch: mini_gpt4

vit encoder

image_size: 224
drop_path_rate: 0
use_grad_checkpoint: False
vit_precision: "fp16"
freeze_vit: True
freeze_qformer: True

Q-Former

num_query_token: 32

Vicuna

llama_model: "/home/Intern1/Yun_SRP/MiniGPT-5/VicunaV07B"

generation configs

prompt: ""

preprocess:
vis_processor:
train:
name: "blip2_image_train"
image_size: 224
eval:
name: "blip2_image_eval"
image_size: 224
text_processor:
train:
name: "blip_caption"
eval:
name: "blip_caption"

[KzZheng]

Have you set the python environment variable IS_STAGE2=True for running the demo? It seems the model weights are not correctly imported.