Training for inlining is causing issue

[itsmanisk]

Hi
I was trying to train the model for my code , during training process from corpus is causing issue . I am getting weird error like
tensorflow.python.framework.errors_impl.InvalidArgumentError

I am using tensorflow 2.11 And protobuf
Has anyone faced same issue?

[boomanaiden154]

You need to use the dependencies specified in the lockfile (which can be installed as outlined in the README). We do not support arbitrary Tensorflow installations.

We're currently using Tensorflow v2.15.0, not v2.11.

[itsmanisk]

@boomanaiden154 , can you suggest me hardware requirements and optimisation i can make to train my model , i somehow generated a corpus earlier which was generating the model but it showed about 6000 hours !!

[boomanaiden154]

The recommended hardware/system setup for PPO training according to README.md are the following:

Currently, the assumptions for the system are:

Recent Ubuntu distro, e.g. 20.04
python 3.10.x/3.11.x
for local training, which is currently the only supported mode, we recommend a high-performance workstation (e.g. 96 hardware threads).

[itsmanisk]

96 threads as in 96 cpu on machine ?

[itsmanisk]

@boomanaiden154 can i use gpu for training, like are there additional settings required for that ?

[boomanaiden154]

can i use gpu for training, like are there additional settings required for that ?

You can, and you just need to set Tensorflow to use the GPU as a device. It's typically slower than just training on the CPU with how we have it setup though (we end up with lots of mem copies between the host and device). We haven't looked too deeply into fixing it as CPU training has been performant enough for us given the bottleneck is the RL data collection rather than training speed.

[itsmanisk]

@boomanaiden154 , i was training on 8core linux pc, but total iteration of training is 9x10^5 , so it is taking ages for complete training, also the system got crashed after running for 2 days !
On gpu with 40 cores , it is still running with same speed , can you suggest something to speed up training process

[boomanaiden154]

i was training on 8core linux pc, but total iteration of training is 9x10^5 , so it is taking ages for complete training, also the system got crashed after running for 2 days !

You typically get reasonable convergence before training completes with the number of iterations set by default.

On gpu with 40 cores , it is still running with same speed , can you suggest something to speed up training process

The bottleneck for RL training is compilation time, not generating new policies. This is purely CPU bound. We typically notice slowdowns when doing training on the GPU due to data movement.

[itsmanisk]

@boomanaiden154 , so can i decrease the number of iteration of training, like currently the value comes around 9*10^5,?

[boomanaiden154]

Yes. You want to change this number/equivalent in other problems:

ml-compiler-opt/compiler_opt/rl/regalloc/gin_configs/ppo_nn_agent.gin

Line 15 in 5eeb719

train_eval.num_policy_iterations=3000

[cclauss]

Why does the README recommend Ubuntu 20.04 instead of something more modern?

[boomanaiden154]

Why does the README recommend Ubuntu 20.04 instead of something more modern?

Because those requirements were written five years ago according to the git blame. Wouldn't hurt to update, but the wording there now ("recent distro") is not completely terrible in my opinion.

[mtrofin]

Why does the README recommend Ubuntu 20.04 instead of something more modern?

Because those requirements were written five years ago according to the git blame. Wouldn't hurt to update, but the wording there now ("recent distro") is not completely terrible in my opinion.

Also because that's what we know (but may need updating) the setup was tested on. We could link to the version we use for the llvm buildbots (e.g. https://lab.llvm.org/buildbot/#/workers/48) as "this is what it's known to currently work on - see also buildbot setup script"