How do I check the accuracy of my project?

The accuracy of your project can be checked in a number of ways. Note that a true accuracy check is always done with reference to an external data source. Accuracy determination for any measurement device (whether it be photogrammetry or some other device) has to be done with respect to external information and cannot be done with internal and self-referenced data only.  There are internal quality measures in PhotoModeler that help determine if a project is good, but these will not and cannot tell you absolute accuracy.

We sometimes use the terms relative accuracy vs absolute accuracy.  Absolute accuracy is in reference to the real world. Relative accuracy has to do with the correct overall shape (e.g. width vs height). The first part of this article talks mostly about absolute accuracy.

External Accuracy Checks
For determining accuracy, external information is used. The external data source can be total stations, laser trackers, laser scanners, coordinate measuring machines, or even tape measures. Ideally you know the accuracy of the data source being used. It will usually be a proven data source and preferably have significantly higher accuracy than the photogrammetric solution (which can be difficult at times due to the high accuracy capabilities of photogrammetry).

The accuracy checks can be done a number of ways. One method is to use distance comparisons – measure the distance between two points that can be identified clearly and precisely with the external device, and then compare with the same distance in the completed PhotoModeler project. The two measurement systems need to share the same scale but they don’t have to have the same origin or rotation. The difference in the two distances (external vs PhotoModeler) is called the ‘error’. Traditional statistical measures such as maximum error, mean error, RMS error, and error deviation are used to determine overall accuracy in the comparison.

If the comparison should include the full coordinates (translation and rotation to be checked as well as the distances) then 3D point to 3D point comparison are done. The external device needs to provide accurate 3D point locations in the same coordinate system as the PhotoModeler project (note, use a multi-point transform or control points in PhotoModeler to match coordinate systems). The comparison can then be differences in distances between the 3d coordinates or a comparison of x, y and z values.

Note that PhotoModeler has the Check Distance and Check Point capability (Imports and Coordinates) to make these comparisons with external data sources easier.

PhotoModeler Quality Feedback
In addition to, prior to, or as a temporary stand-in for an external accuracy assessment, information from PhotoModeler alone can help gauge the quality of the project. While these internal numbers indicate how well the project solved and are good for comparing like projects, they are not true accuracy measurements.

Definitions
A couple of definitions first: Accuracy is the agreement of a measurement with a recognized standard or the “true” value; and Precision is the degree of which similar or repeated measurements show the same results. A good measurement system has both good precision and good accuracy. PhotoModeler’s internal numbers (discussed below) relate to quality and precision more so than accuracy. Ideally a precise PhotoModeler project is also accurate but this is not known until a comparison to a known external data source is made.

PhotoModeler Internal Sources
Total Error: A total error number appears after processing and can also be found in the Processing Report. This value can be described as “goodness of fit”.  It measures how well all the input data (camera parameters, mark locations, and 3d points) ‘agree’ with each other. As a general guideline, a project should have a total error under a value of 1.0.

Residuals: An excellent and quick measure of project quality (because it appears at all times on the lower status bar in the program) is maximum residual. A residual is the distance in pixels between where the points was marked (by user or automatically) on a photo and where the projection of the 3d point, associated with that marked point, falls on the photo. The maximum residual is the largest such residual across all marks in the project. A high-quality project would typically have a maximum residual under 1.0 pixels. Projects with manually marked points, use inverse camera, or have control points etc., can still be high quality and have a maximum residual of 5 to 10 pixels. Note that this is measure is based on the assumption that the high residual is not due to a mismarking or misreference. Sometimes correcting the one or two misreferences can reduce the maximum residual of a project significantly.

Precision Values: Point Precision values, found in the Point Table under the Quality configuration, represent what PhotoModeler expects for the precision of the point coordinates. These numbers should be very low in a strong project. Note that the following assumptions are made when calculating these values: a) there are no “gross errors”, meaning there are no mismarked or misreferenced points and the camera calibration is good, and b) the input marking precisions reflect the true precision (in the Preferences / Process tab, the default Point Standard Deviation Weight values for normal marks should be “1”).

Further detail can be found in the PhotoModeler Help File section under Accuracy and Precision, and on Wikipedia under Accuracy. Also, see the Knowledge Base article Factors affecting accuracy.