GetAllCharts takes too much time when there is a considerable amount of charts in backend storage #40
Description
The time analysis of the /api/:repo/charts api as follows:
2020-03-11T19:18:55.184+0800 DEBUG [1] Incoming request: /api/datacollect/charts {"reqID": "a543f2fe-53b5-469a-b90d-cc80a7f36657"}
2020-03-11T19:19:06.957+0800 DEBUG [1] index-cache.yaml loaded {"repo": "datacollect", "reqID": "a543f2fe-53b5-469a-b90d-cc80a7f36657"}
2020-03-11T19:19:06.957+0800 DEBUG [1] Fetching chart list from storage {"repo": "datacollect", "reqID": "a543f2fe-53b5-469a-b90d-cc80a7f36657"}
2020-03-11T19:19:14.519+0800 DEBUG [1] start get object slice {"reqID": "a543f2fe-53b5-469a-b90d-cc80a7f36657"}
2020-03-11T19:19:14.531+0800 DEBUG [1] objects length {"o1": 57376, "o2": 57370, "reqID": "a543f2fe-53b5-469a-b90d-cc80a7f36657"}
2020-03-11T19:19:14.531+0800 DEBUG [1] start get object slice diff {"reqID": "a543f2fe-53b5-469a-b90d-cc80a7f36657"}
2020-03-11T19:19:58.470+0800 DEBUG [1] Change detected between cache and storage {"repo": "datacollect", "reqID": "a543f2fe-53b5-469a-b90d-cc80a7f36657"}
Note: the bold parts are added by myself for easy analysis.
When the back-end storage (BOS) has 57376 charts, after time-consuming analysis, it takes 8 seconds to get all the files from the back-end. It takes 44s to calculate the difference between cache and back-end data by cm_storage.GetObjectSliceDiff(objects, fo.objects).
