Importing Data

[ZinnerC]

I'm currently setting up parts and their information within Inventree. For that purpose I first created all the parts with their basic information. Now I would like to add information step by step into Inventree.

But currently I experience the following situation with the old UI and its respective import functionality I only can import about 100-150 parts per import without getting a "internal server error". Not sure what it is but could be a timeout.

After that I tried to import some new information via the new GUI (PUI) and the respective new Data Importer. But somehow I do not manage to import anything.

Did I miss something? I know currently it is quite a generic question but would be interesting to hear from others how they experience the importing of additional data (actually updating existing parts)

About my Inventree Installation

• Digital Ocean Droplet with 2GB and regular CPU

Version Information:

InvenTree-Version: 0.16.8
Django Version: 4.2.15

Database: postgresql
Debug-Mode: False
Deployed using Docker: False
Platform: Linux-5.4.0-200-generic-x86_64-with-glibc2.31
Installer: None

Active plugins: [{'name': 'InvenTreeBarcode', 'slug': 'inventreebarcode', 'version': '2.1.0'}, {'name': 'InvenTreeCoreNotificationsPlugin', 'slug': 'inventreecorenotificationsplugin', 'version': '1.0.0'}, {'name': 'InvenTreeCurrencyExchange', 'slug': 'inventreecurrencyexchange', 'version': '1.0.0'}, {'name': 'InvenTreeLabel', 'slug': 'inventreelabel', 'version': '1.1.0'}, {'name': 'InvenTreeLabelMachine', 'slug': 'inventreelabelmachine', 'version': '1.0.0'}, {'name': 'InvenTreeLabelSheet', 'slug': 'inventreelabelsheet', 'version': '1.0.0'}, {'name': 'DigiKeyPlugin', 'slug': 'digikeyplugin', 'version': '1.0.0'}, {'name': 'LCSCPlugin', 'slug': 'lcscplugin', 'version': '1.0.0'}, {'name': 'MouserPlugin', 'slug': 'mouserplugin', 'version': '1.0.0'}, {'name': 'TMEPlugin', 'slug': 'tmeplugin', 'version': '1.0.0'}]

[SchrodingersGat]

Hey @ZinnerC that import tool is soon to be removed (in our next major release when remove the old UI entirely).

This is because, the way that the django-import-export library imports data is extremely inefficient. If you have more than a handful of rows, it will timeout the server.

Some related issues with the same problem:

• Timeout when downloading large parts list. #6376
• Export of parts through admin fails with 502 Bad Gateway #5571

So, we have implemented a brand new importer - #6911

This is only available in the new user interface (you should be able to see it at <host>/platform/settings/admin/import. The new importer runs in the background and is much more efficient in any case.

[ZinnerC]

Hi @SchrodingersGat Thank you for your response yes I'm aware about the new importer.

I would like to show you my process so you could tell me where I'm doing something wrong. Or where I expect something wrong.

First I go to a part I would like to modify and downloads its data as a XLSX:

Then I check what ID the category I would like to add has (ID=1).

Then I edit the respective position in the XLSX (I upload the original download and the modified one.)
InvenTree_Part_2024-12-06_Category.xlsx
InvenTree_Part_2024-12-06_Original.xlsx

Then I go to the "Importer". There I add a new import and make the respective selections then I submit.

• DataFile (modified XLSX)
• Model Type: Part since I downloaded that XLSX

There I do the mapping and only the the minimal (all others I ignore since I don't want to change and expect to have a more efficient import)

Complete Mapping:

Then I check if it understood the input as expected (validate in step 4)

Then I select the one I would like to import and click on the green arrow.

At this very instance a popup appears on the lower right but quickly disappears again, telling something that something is going on.
But I don't reach step 5 where the import is completed.
Closing the dialog at step 4 and I see that the respective session is still avalable and not completed.

Quickly checking if the part has changed I see that it did not. However, this is not surprising since the import actually did not complete.

So this is how far I get I don't understand where somethings does not go as required.

[SchrodingersGat]

@ZinnerC the new importer only creates new records - does not allow for updating existing records in bulk

[ZinnerC]

Is it planned to actually also allow the import to update existing data? Would help in initiation.

[SchrodingersGat]

@ZinnerC TBH it had not been planned - but I have added a new issue here - #8640

If you want to help develop this, or support it with funding, that would be appreciated :)

[ZinnerC]

@SchrodingersGat thank you for creating this issue. With respect of the development/funding I can check ;-) But I thank you for the for the support/information so far.

[SergeoLacruz]

When we started with InvenTree some years ago, we had the same problem importing ~2500 components with lots of properties and attachments from a huge excel table. We used pandas and the InvenTree python library to solve this. Using this method we were able to
import everything including relations to suppliers and manufacturers, attachments, stock items, ...
Similar for BOM. CAD uses the python lib to create, or update a BOM without loosing build allocations and create a change list.

[ZinnerC]

Would you be able to share your script?

[SergeoLacruz]

Not really sorry. It just tinkered and not good enough for publishing because it was meant to used once.
Parsing an excel table with pandas is easy. Finding all the information is very specific based on the structure of the table.
Then I used the python functions to create parts, stockitems, and so on. Only creation, no modification. Just run it once category by category. When something went wrong, I deleted the parts and started a new run. Took some days.

[wolflu05]

Here are some links that may help you:

• https://pandas.pydata.org/docs/reference/api/pandas.read_excel.html
• https://pandas.pydata.org/docs/user_guide/10min.html
• https://docs.inventree.org/en/latest/api/python/python/#item-attributes

[SergeoLacruz]

I wrote the major part of the sofa example during this work:

https://docs.inventree.org/en/latest/api/python/examples/

[ZinnerC]

Lovely. Thank you. I will have a try.