Welcome to TsaThroughput Discussions!

[mikelor]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[dingyif]

Hi Mike,

Thanks for putting up this amazing github :) This is super helpful for me to use. May I ask when will be generally data update for this git?

Best,
Dingyi

[dingyif]

" basename: missing operand
Try 'basename --help' for more information. "
This is the error message that I'm getting with I run sh update_data.sh

[mikelor]

Hmm, yeah the readme doesn't have everything you need to do. :(

I'll see about updating it, but basically you will need:

1. Linux
2. Install Python 3.8x
3. Install dotnet 6.0
4. Create a Python Environment for tsathroughput (python -m venv tsathroughput)
5. Set your environment to the new environment (source ~/tsathroughput/bin/activate)
6. Install PIP
7. Use PIP to install the necessary python libraries (pip install -r ./src/tsa_throughput/requirements.txt
8. From the ./scripts directory try to run the ./cvtJsonToCsv.sh tsa-throughput-april-19-2020-to-april-25-2020

In order to convert PDF to JSON you will need to have access to an Azure account as of now and setup your own infrastructure. I am working on a local implementation that removes the dependency, but it's not ready yet.

The repo has the latest TSA data (through July 30) that the TSA has published on their website. Right now I check every Monday/Tuesday manually, but working on an automated process too. Are you looking for a weekly file with just the latest TSA data as a CSV?

Happy to help with any or all of above. Are you on a Mac or PC or??

[dingyif]

Hi Mike,

Thanks for the quick response :) I see now what is going on; I was wondering why I couldn't convert the new pdf to JSON from my end, and it seems like I need to set up an azure account. In the meantime, I have a project that depends on this weekly result, would like to know what will be the best approach to get the latest file weekly. I don't want to bother you all the time just to run the update. Do you know what will be the best approach? Please also keep us updated on the local implementation. I think this will be a long-run solution. I'm currently using the windows system.

Best,
Dingyi

[mikelor]

Good morning @dingyif,
I've updated the TsaThroughputApp (the component that extracts data from PDF -> JSON). It now uses https://github.com/BobLd/tabula-sharp instead of Azure Form Recognizer.

You will need to git fetch the latest. I also added a cvtPdfToJson.cmd file to run from the Windows command prompt. You will need to pass in the name of the pdf file you want to read from.

[dingyif]

Hi Mike,

This is really helpful & amazing! Thanks so much for the new update and for removing the dependency on Azure!

Thanks,
Dingyi