Computation at HPC system: First time computing Jacobian takes incredibly long, subsequent computations are OK #20164
Answered
by
lindsayad
matthiasneuner
asked this question in
Q&A General
-
|
Hi there, I am facing a very strange issue, for which I do no have a real explanation. The screenshot shows this behavior for a restarted simulation with 1e7 dof computed on 256 cpu cores: Of course, it is not really a problem, since the computation of the Jacobian seems correct and all subsequent steps work as expected. However, I am really curious what might causing this behavior. Has anybody an idea on this behavior :) ? |
Beta Was this translation helpful? Give feedback.
Answered by
lindsayad
Jan 29, 2022
Replies: 4 comments 9 replies
-
|
I have no idea. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
What is your physics? Do you have contact? This kind of behavior is often seen when the initial sparsity pattern is not correct and PETSc has to allocate memory on the fly during Jacobian assembly. Once the first Jacobian is computed, typically the memory is sufficient for all subsequent Jacobian calculations. Our traditional contact methods are known to have some issues with the initial sparsity pattern, at least based on some recent reports |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
GiudGiud
-
|
Thank you for both answers! My application is essentially a TensorMechanis App, using my own (stress divergence) kernels. I could try to run the same example without the Furthermore, I am using no threading, just pure MPI parallelization. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Can you add this to your input file? If the slowness in your original run is due to my hypothesis, then with this option your run should error sometime during that initial Jacobian assembly. |
Beta Was this translation helpful? Give feedback.
-
|
I think you were right: |
Beta Was this translation helpful? Give feedback.
8 replies
-
|
How to accomplish that? |
Beta Was this translation helpful? Give feedback.
-
|
That's something we have to fix at the framework level. I don't suppose your input is something you could send to me? If not, I can try to drum up a test problem that has the same issues |
Beta Was this translation helpful? Give feedback.
-
|
@lindsayad any update on this |
Beta Was this translation helpful? Give feedback.
-
|
Nodal constraints are not a focus for our funders, so unless there has been some fix by happenstance, this is likely still an issue. Nodal constraints do not use our modern relationship manager system which is likely why the sparsity pattern estimation is so poor |
Beta Was this translation helpful? Give feedback.
-
|
I'll add on this old discussion that there is a first-class solution (such that I won't just call it a work-around) in #28979. You can set |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What is your physics? Do you have contact? This kind of behavior is often seen when the initial sparsity pattern is not correct and PETSc has to allocate memory on the fly during Jacobian assembly. Once the first Jacobian is computed, typically the memory is sufficient for all subsequent Jacobian calculations. Our traditional contact methods are known to have some issues with the initial sparsity pattern, at least based on some recent reports