issue_export page size parameter needed

[vikoalucard]

Hello, First, thanks a lot for this lib !
Are you still supporting it ?
Is there any way to change the page size for the issue_export method ?
I have some instances so large that the "default max" (1k isses per page) will crash, especially in cloud:

Downloading issue export in CSV format.
<Response [200]> OK ::downloading issues at page: 0 of 4
<Response [200]> OK ::downloading issues at page: 1 of 4
<Response [200]> OK ::downloading issues at page: 2 of 4
<Response [504]> Gateway Timeout ::downloading issues at page: 3 of 4
<Response [200]> OK ::downloading issues at page: 4 of 4

=> results in a crash in the merge (1 file containing the CSV error...)

Thank you in advance !

[princenyeche]

Hey @vikoalucard

The page size default is 1k and it cannot be reduced or increased. Since page 3 is the one breaking things from your example, you can omit it by doing the below

# import statement
issue_export(jql, page=(0, 2), final_file="file1.csv")

do another export for page 4

# import statement
issue_export(jql, page=(4, 4), final_file="file3.csv")

Then attempt page 3 export alone

# import statement
issue_export(jql, page=(3, 3), final_file="file2.csv")

Once done, you can use the merge_files argument to combine the 3 files or simply combine the first two.

# import statement
files = ["file1.csv", "file3.csv"]
issue_export(merge_files=files, check_auth=False)

OR

# import statement
files = ["file1.csv", "file2.csv", "file3.csv"]
issue_export(merge_files=files, check_auth=False)

If the download for certain pages keeps timing out, probably some data within the export is attempting to download some content from the project that might be taking too much time for your Jira environment to respond.

[vikoalucard]

Hey @princenyeche , thanks a lot for your help!
Yes it always crashes on the same page 3 (all pages are long anyway, though), and I have no other way to get it properly using the current script.
That's a big instance (around 3k custom fields, with many team managed projects...), so all operations are slower.
From reporting.py, I see there are many hard coded "1000" for page size and searches, and the "limit" param in issue_export, from access.py, isn't used yet
I was thinking about allowing a different page size for that, using a parameter passed down to this method instead of hard coded 1000. Even though the code is long, it seems there are around 20-30 lines to change to support custom page sizes, but I'm not the expert.
If you think that's doable as I describe, I could even try the modification on my side.

[princenyeche]

About your question on a limit parameter, that isn't used in the issue_export yet so you can't manipulate it except you wanted to use it directly from the endpoint constant. Although to this issue, I think the problem might be the 3K custom fields you mentioned. If that's the case then for page 3, you have two options:

Include Specific Custom Fields in the Export:

• Set up all the fields you need, but this may require fine-tuning. Using the include_fields parameter

Export Only Current Fields:

• This is more straightforward.
• Configure the fields you want by following these steps:
  • Go to your Advanced issue search on your UI.
  • Perform a JQL search (the search you're doing).
  • Add the columns you need for the export (not all fields).

In your script, construct it like this:

# Import statement
issue_export(jql, field_type="current", page=(3, 3))

What this does:

• Exports only the necessary fields added on the UI.
• Downloading 3K custom fields will take time given the situation, so consider reducing the custom field size for a faster export to prevent that 504 error as I think that's the problem here rather than reducing the rows exported.