How to download the whole set of products per Catalog

[YSavitski]

Hi guys,

we have Intergration with Meplato where our company manages 30+ active catalogs.

From time to time we need to download the whole set of created products per catalog.
At the current moment we use Product Scroll API, but the whole process takes so much time for our big catalogs (1M+ products).

Is the any option to triger export of the whole set catalog products for example via CSV file? Or any other option to speed up (and reduce amount of Meplato API calls) the process?

Many thanks

p.s. I've found option to trigger export: https://store.meplato.com/catalogs/{catalogId}/products/export/csv
Is there API documentation for this endpoint?

[olivere]

Hello @YSavitski! Your question is interesting in the sense that it's a use-case we didn't expect but hear from time to time. What's unexpected for us is that we assume you to have full control and all data about your products and prices. What is the reason you need to download your own catalogs from Store?

P.S.:
Notice that we have recently updated the documentation site. You can visit it here.

[YSavitski]

Hi @olivere
Thank you for sharing link to renewed documentation.

Regarding your main qeustion, in short, I may say that this is an issue of our processes.

We use Meplato API as additional tool for manipulate with product assortment, like creation new products for specific use-cases.
But in general our account/sales managers recreate product assortmet via file upload. They may inform us but also may forget to do this.

Therefore, we'd like to have an option to download existing product assortment from the Meplato side.
Now we use Product Scroll API, but I defenitelly guess, that Scroll hasn't been designed for such needs. Especially, when we toalking about catalogs > +1mio products.

And another problem with Scroll API - in 50%+ (sometimes during the whole day) executions we're getting timeout execeptions, regardless of whether we use our own client implementation or Meplato Java API.

[YSavitski]

Hey @olivere
I'd like to correct a bit my initial request.

It seems like on Meplato GUI there is an option to dowload catalog as CSV file, but this option is limitted to 10k products.
Is there reason for that?

Btw, I see in operation history, that there is gRPC API which is used (I guess) internaly for catalog download operations

Would it be possible to at least allow to download the comple catalog assortment via API https://store.meplato.com/catalogs/%7BcatalogId%7D/products/export/csv ? Or at least provide additional RequestParamets which we can use for that to avoid changing the default behavior?

Many thanks for any feedback

[YSavitski]

Hi dear collegues,

Would you be able to check my last comment?
#42 (comment)

[olivere]

@YSavitski I'll loop in the product manager.

[YSavitski]

@YSavitski I'll loop in the product manager.

Hi @olivere

Did you have any option to loop this request to the product manager? Could we expect any further steps?

Many thanks

[olivere]

@YSavitski I no longer work for Meplato. Maybe you'll find someone getting back to you.

[XoseRamon]

Hi @YSavitski

The idea behind the CSV export is to provide a way to export products with errors, so they can be consulted outside the application. It was not though with your use-case in mind and might be difficult to adapt.

As you might imagine, the Product API has been thought to be able to ingest fast, but not with the data export use case in mind. I will double-check with the product manager, and see if there is an easy way to support this.

About the timeouts: we are working currently to increase the capacity, so it should improve in the next month or so.

Thanks for the feedback and the patience!

[YSavitski]

Hi @YSavitski

The idea behind the CSV export is to provide a way to export products with errors, so they can be consulted outside the application. It was not though with your use-case in mind and might be difficult to adapt.

As you might imagine, the Product API has been thought to be able to ingest fast, but not with the data export use case in mind. I will double-check with the product manager, and see if there is an easy way to support this.

About the timeouts: we are working currently to increase the capacity, so it should improve in the next month or so.

Thanks for the feedback and the patience!

Hi @XoseRamon

Many thanks for your feedback! We're looking forward for any updates!

[matze-sr]

Hi @YSavitski,

can we have an online call to discuss your use case in detail? This will help to decide if your request will fit into our product roadmap.

Best regards

[YSavitski]

Hi @YSavitski,

can we have an online call to discuss your use case in detail? This will help to decide if your request will fit into our product roadmap.

Best regards

Hey @matze-sr
Sure, we can have call. Just send me an invite on the mail yauheni.savitski@ext.conrad.com.
I'm working on CET time-zone starting 8am till 6pm.

I have a lot of open slots next week - feel free to choose any. Meantime, I can also shortly prepare myself to describe our workflows and current problems