[Getdp] Parallelization patch for GetDP with PETSc
davide.rondini at gmail.com
Wed Jun 3 11:01:11 CEST 2009
I developed a patch for GetDP to enable it to work properly with parallel
version of PETSc. Essentially, I removed a couple of deadlocks, and introduced
the check for MatGetOwnershipRange in functions which set the elements of
matrices and vectors. This ensures that everything is assembled the right way
even using mpiaij matrices.
I tested it using some sample tests taken from wiki and I've seen that it
works fine. Accuracy is ok, it is within tolerance set between serial and
parallel runs (also compared with sparsekit version). Performance scale though
number of processors as expected.
Anyway, this is a crude patch, it has 2 disadvantages:
1. The check of competence of an element of the matrix to each processor is
done inside LinAlg functions, which forces all processors to check all
elements of the matrix. This is not efficient, this operation should be made in
the cycle that assemples the matrix, but this is not so simple, it requires to
modify GetDP code in multiple points. I hope to have time to remove this
overhead in the future. It wolud be very helpful if someone can show me where
the code has to be modified.
2. The parallel process works fine up to the calculation phase, while post-
processing has some problems to run in parallel. This is because PETSc do not
allow to access the elements of a vector or matrix which pertain to another
processor. So, performing post operations in serial mode inserts spurious
zeros or nan in the solution, and doing parallel post processing generates
messy output file. Implementing parallel I/O is long and complex, for me it is
sufficient to run resolution and post-operation separately, this way:
mpirun -n X getdp simulation.pro -solve Resolution
mpirun -n 1 getdp simulation.pro -pos PostOperation
this workaround works fine.
I hope this can be helpful.
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 20119 bytes
Desc: not available
Url : http://www.geuz.org/pipermail/getdp/attachments/20090603/94295e2d/attachment.patch
More information about the getdp