There are two common formulations for radial lens distortion: balanced and unbalanced. They are mathematically equivalent and make no difference to the algorithm. The balanced distortion formulation "looks" prettier and makes the distortions seem smaller so is commonly used by camera and lens manufacturers. PhotoModeler use the unbalanced form.
The radial lens distortion is radially symmetric about the principal point. A marked point at a radial distance of "r" (in mm from the principal point) has to be compensated by a factor drb. The correction is applied to the x and y components (in mm from the principal point) in this way:
xc = x * ( 1 + drb / r )
yc = y * ( 1 + drb / r )
If you pre-divide the radial distortion formulas by r you have:
dr = drb / r
xc = x * ( 1 + dr )
yc = y * ( 1 + dr )
The formula used by PhotoModeler for dr is given by:
dr = K1 * r^2 + K2 * r^4 + K3 * r^6
where r^2 = x^2 + y^2 (r^2 means r to the power of 2)
where K1 (in mm^-2), K2 (in mm^-4) and K3 (in mm^-6) are the parameters shown in the Camera Viewer Dialog. This unbalanced form takes into account the "calibrated" focal length which is found in the same dialog.
The standard balanced form for radial lens distortion is:
dr = A0 + A1 * r^2 + A2 * r^4 + A3 * r^6
To convert this balanced form to PhotoModeler's unbalanced form you use:
s = (1 - A0)
f = f2 / s
K1 = A1 / s
K2 = A2 / s
K3 = A3 / s
where f2 (in mm) is the focal length (principal distance) given in the original calibration with the balanced distortion and f is the focal length to be used in PhotoModeler.
Do note that some other photogrammetric programs apply K1, K2 and K3 with a different sign than PhotoModeler. If you are not getting good results with the above try changing the sign of K1, K2 and K3. Changing the sign of A0 might also help.
To convert PhotoModeler's unbalanced form to the balanced form is a bit trickier because you need to decide on the balancing point (i.e. where the radial distortion equals zero).
A0 = - ( K1 * r0^2 + K2 * r0^4 + K3 * r0^6 )
s = (1 - A0)
f2 = f * s
A1 = K1 * s
A2 = K2 * s
A3 = K3 * s
where r0 is an arbitrary radius chosen so the radial distortion curve is balanced with equal negative and positive distortion over the extent of the format's radius.