Data Import recommandations

[m-overlund]

Hi everyone :-)

How do you all import data, like products for instance, on a daily basis?

Magmi has seem like a favorite, but has also been subject to a lot of security issues.

What is still up to date and safe to use for OpenMage?

• Andreas von Studnitz's Fast Simple Import
• Store manager for Magento

[addison74]

This is a very good question.

• Magmi still has a lot of issues reported and nothing changed in the last 9 months. Even I am not using it in production daily in my webserver logs I am getting URL's related to Magmi. It is acknowledged that it had security issues and there may still be some that are not yet known.

• I tested for a short period Fast Simple Import but I did not like it. A lot of missing features and nothing changed in the last 3 years. The project seems to be abandoned.

• Store Manager is by far the best solution for adding/editing/mass export&import and more than that. If you can afford paying for an enterprise license you have tons of features. The team is very flexible and respond fast to your questions. Also there is a lot of videos and written tutorials about how to use the program.

In my stores when they are imported lots of products we used Magento default. It is not perfect because the content editors need to check every product and add/correct some data but at least works.

EDIT: As I see there is a 30% discount for Store Manager at this moment. Almost 185 USD for de basic version it a good price.

[Flyingmana]

I only know AvS_FastSimpleImport in detail. It does mostly only support simpler usecases, but the focus is on beeing fast.
It should still work good with OpenMage and is OpenSource.
But most of its users probably moved to the m2 version of it.

StoreManager seems like it brings also a lot of other things. If you can make use of some of them (like the exports to ebay/amazon), than it sounds like a good choice.

[m-overlund]

Store Manager sure brings out a lot of easy to use features in a GUI (for the non technical-savvy).
And the import/export works good in general.

Although it also has a lot of caveats due to it's use of direct SQL connection, not using Magento's logic.
Which can easily crash the site or lead to great amounts of URL rewrites etc.

Also it does not handle products with "Manage Stock" = "disabled" very well, as it marks these as out of stock.

[tmotyl]

here is the magmi version we're using https://github.com/macopedia/magmi-m2/commits/magento1 (it has some fixes the upstream doesn't have including performance).
The concept of Magmi i pretty nice, its very fast, however definitely should get some love regarding code formatting to improve readability.
Maybe it would be good idea to contact the author @dweeves, to check with him whether he is open to add contributors or do a little hackathon together.

[m-overlund]

Nice!
Do you use the Magmi fork in production, or only "offline", regarding the security issues?

[addison74]

It can be used in production as well if you change the default name for the Magmi directory then set an authentication for the directory too. Always dump the whole database before using the script.

You can use it offline but more steps are required. I did this a couple of times by exporting only the tables related to products and importing the dump file with CLI directly on the server. Special attention to details is required in this case.

[tmotyl]

we're using it on production, making imports several times a day (stocks, prices) and importing product data few times a day.
the magmi folder is protected on the nginx level. but since most recent security release of magmi, the default auth has been removed, so you have to authenticate to magmi using existing magento credentials, which resolves the most pressing issue of having a default login and password (which many people didn't change)

[addison74]

This is a good start for anyone to understand what Magmi offers:

https://mediaproper.com/magento-catalog-management-imports-magmi-guide/

I just played with the latest version @tmotyl pointed us out and there are many improvements comparing with the versions I tested long time ago I guess 7 years ago.

[addison74]

Magmi should be used with precaution. By accessing https://www.mydomain.tld/magmi/conf/magmi.ini here is what I get:

[DATABASE]
connectivity = "net"
host = "localhost"
port = "3306"
resource = "default_setup"
unix_socket = 
dbname = "openmage_2010"
user = "admin"
password = "MX77*(050Z$Sa578298734Ak"
table_prefix = 
[MAGENTO]
version = "1.9.x"
basedir = "../.."
[GLOBAL]
step = "0.5"
multiselect_sep = ","
dirmask = "755"
filemask = "644"
noattsetupdate = "off"

If an attacker knows the path for phpMyAdmin he could use these information to access the database. In my webserver logs I am getting daily GET requests to URL's like this /magmi/conf/magmi.ini.

Using a .htaccess in some directories with this content is mandatory in my opinion, at list in /magmi/conf/.

Order deny,allow
Deny from all

[tmotyl]

What about adding the rule blocking .ini files to openmage htaccess?

[tmotyl]

Well and to the magmi repo too of course :)

[fballiano]

I only use AvS_FastSimpleImport, stopped using magmi at least 4 years ago.

sadly a lot of modules are abandoned... but now that OM+php7.4 is really, really fast, maybe we don't need to have an external import system?

[addison74]

Magmi is really fast but I am still using the default OM feature Import/Export. It won't take too much time to import 2.500 products, just a few seconds. Then it takes a few seconds for re-indexing.

Magento 1 has an enormous heritage behind.Most of the people got rid off it too easy. Probably they did not check that other popular paid carts already switched to cloud versions when you cannot see the source code. You become an user in an interface. In the past I was a contributor for X-Cart and I knew very well its direct competitor CS-Cart. The price for a license was around 200 USD. Please take a look how much cost now a CS-Cart license: https://www.cs-cart.com/compare.html.

OM is a privilege that there is still a complete e-commerce platform with so many facilities where we can get involved in the source code.

[tmotyl]

FYI, I have talked with Sebastian (maintainer of the Magmi) and he is open to collaborate with OpenMage in improving Magmi. He has just granted me merge rights on his repo. If any of the OpenMage maintainers also want to get involved, let me know.

[addison74]

Magmi needs a new UI because it is not attractive. Bootstrap, Tailwind, Vanilla CSS would be a great addition. Tailwind has the advantage you can get a CSS file only for the styles you are using. In this way the file size is small. For a few pages it is very flexible without touching any CSS file everything is inline based on class attribute.

[tmotyl]

Totally agree :)
Actually there is a PR already which redo the UI using bootstrap. So some review is missing:
dweeves/magmi-git#555

[addison74]

Bootstrap 5 is already stable and ready for production. It brought many nice features. I would use only a part of this framework not the whole one.

[fballiano]

interesting, but IMHO it should be a magento backend module, not an external something with a different ACL system and configuration. at least it should be splitted into engine/gui so that we could use it as a library without that gui system.

[tmotyl]

This is out of scope for my personal interest in Magmi, Im using it in two projects only. But I agree it could evolve into this direction