SCORING PAPER TARGETS WITH A COMPUTER
Orion Scoring System Now Available

The Orion Scoring System is a new software product that automates the task of scoring paper targets. It is now commercially available from Shooters Technology, LLC, of Fairfax, Virginia for club, team and individual use. Orion has successfully applied modern computer technology to the challenges of conducting competitions with paper targets and it promises to make match sponsors’ work easier and the conduct of matches more interesting. Four years in development, Orion is one of the most accurate electronic scoring systems available anywhere in the world. It is currently available for 10-meter air rifle scoring, but future versions due out in coming months will score National Match Air Rifle, 50-foot smallbore and 10-meter air pistol.

Orion uses a computer vision technique known as Visual Image Scoring (VIS). VIS works by reading or interpreting scanned digital images of fired paper targets. It then uses a complex series of algorithms to locate the exact centers of the aiming bull and shot hole for each record shot, calculate the distances between the centers of the aiming bulls and the shot holes and determine their scores.

The commercial version of Orion VIS software was released in August and can now be purchased from Shooters’ Technology. The first year price for a single computer 10m air rifle license is $398.00. A renewal license will be $79.00 per year. Detailed information on purchasing Orion or on obtaining licenses for multi-computers can be found at http://www.orionscoringsystem.com. Information is also available by contacting support@shooterstech.net.

When compared with scoring paper targets manually or electronic scoring targets (EST), VIS represents an advantageous middle ground. Manual scoring is cheap if lots of non-paid volunteer scorers can be recruited, but it is fraught with inaccuracies, inconsistencies, slow scoring and a recurring need to train new scorers. Electronic targets are much quicker and more accurate than manual scoring, but they are prohibitively expensive for most shooting teams and clubs. VIS offers the scoring speed and accuracy of ESTs at a fraction of the EST price.

VIS scoring offers significant advantages over manual scoring since it is consistent, fair and accurate. Shooters frequently criticize the differences between “hard” and “soft” scorers. As a result, scores fired in one competition may not be directly comparable to scores in other competitions. This difference even occurs within competitions when different scoring teams score different targets. With Orion, all shooters are treated equally since their targets are scored according to the same standard.

Both VIS and EST systems have the advantage of making it possible to quickly post scores and shot displays on the Internet so that a much larger shooting public (fans) can be informed about match results. Orion also has advantages over both EST and manual target scoring systems when personnel requirements are analyzed. Orion requires far fewer personnel than manual paper target scoring. It also requires fewer personnel with technical training than EST systems. Orion requires a verifier who understands scoring and has good computer skills. One verifier and one computer loaded with Orion is recommended for each 20-30 active firing points. EST targets require a highly trained technical officer to operate the central ranking computer plus additional trained technical personnel to operate and maintain the targets and operate the range during firing.

In addition, and unlike manual scoring that only evaluates the inner or outermost point of the shot hole, VIS uses the entire circumference of the shot hole or as much of the shot hole as it can locate to determine its location. Orion starts with a color scan of the entire 10-bull target. Most 300 dpi or higher scanners may be used to produce scanned images that Orion can score. Orion uses a complex system of algorithms to identify the outer edge of the aiming black and the outer edge of the shot hole. It scores based on the exact center of what the shooter sees when aiming; EST systems sometimes have acoustical centers that are slightly off-center.

Orion does not, however, have to find a complete round shot hole. In cases where there is a torn or altered shot hole, it can pin-point the shot hole location if it finds a curved shot hole edge that constitutes as little as 30 percent of a 4.5mm diameter ring. After locating the exact center of the shot hole, Orion calculates the x and y coordinates of the shot relative to the target center and derives the radial distance from the center of the target to the center of the shot hole to score the shot.

With this information, Orion produces a record for each shot on the target with 1) the scores, 2) the scores in tenth ring values, 3) the x and y coordinates and 4) the radial distances of the shots from the center. To complete the scoring process, a human verifier reviews Orion’s scoring data to see whether there are any scoring issues caused by excessive or insufficient shots, torn or altered shot holes or a possible double. The verifier, who must have a knowledge of scoring rules, must resolve these issues and may make manual interventions to produce the correct scores.

Orion’s ability to score shots by using only the unaltered part of a shot hole to locate its center has had the unintended consequence of facilitating the detection of deliberately altered shot holes. With VIS scoring, even shot holes that are intentionally altered to make them appear to score higher can still be scored accurately. In at least two cases so far, Orion scans were used to catch shot hole manipulators.

The ability to score paper targets electronically, although novel, is by itself not adequate to meet match sponsor needs. What is required in addition to electronic target scoring is the ability to create competition records, organize shooters and produce results lists. Orion provides the functionality to do all this so that it can be used to run matches and score targets.

The first step in using Orion match management software is to create a match. This match record holds all essential information about the courses of fire, shooters, teams, targets, scores and rankings. The match director or statistical officer must then add or register all shooters and teams in the match record. In an upcoming version of Orion, match sponsors, whose computers are online with the CMP Competitor Tracker system, will only have to enter a competitor’s CMP number and Competitor Tracker will automatically fill in the shooters name, handicap, and number of EIC points the shooters has earned. Each participant can be assigned qualifying attributes, such as whether they are shooting a precision or sporter air rifle, the organization or team they represent and any other category information. This data is used to create class and category results lists. Once shooters are added, the user may scan and score targets. Targets must be assigned to the shooter who fired them and the statistical officer or a verifier must review them. After that, Orion automatically calculates individual and team results. Orion currently supports the National Three-Position Air Rifle Council rulebook, but it can create matches and produce results for standard 40 and 60-shot air rifle standing events.

A feature just added to Orion now allows match sponsors to upload scores directly from Orion to the CMP Competitor Tracker system. This means that three-position air rifle match sponsors no longer have to create a separate file or report and send it to the CMP. With Orion, they can report scores to the CMP electronically just as soon as the match is over.

In order to use the Orion Scoring System, the user must have a Windows based computer, a scanner and Orion software. A printer is also recommended. Most computer systems purchased in the last two years will run Orion with ease. The recommended system is, however, a dual core 2.6MHz machine with 4GB of memory and, ideally, 100GB of free hard drive space.

If Orion is to be used for club shooting and small matches, a single computer and a flat-bed scanner will work nicely to keep up with targets being fired on ranges with up to eight to ten firing points. Once the scoring demand becomes greater than that, it will be necessary to acquire a high-speed scanner. There are currently two scanners that are fully supported by Orion. They are the Canon DR-3010C and DR-4010C. Both scanners have automatic document feeders and have been tested with Orion. The DR-3010C can scan 15 targets a minute and costs about $800; the 4010C can scan 40 targets a minute and costs about $2600. Other scanners, including flat bed scanners, that are capable of producing 24 bit color JPEG images scanned at 300DPI and can handle the heavier (200gsm) target paper may also work.

Most current Orion users are focusing on using it to score large quantities of competition targets, but Orion also provides technical data about shot locations and groups that could make it a valuable training tool for clubs, teams and individual shooters. With tenth ring scoring and radial distance information, shooters and coaches can have a much clearer picture of how well their shooters are really performing. A common error of inexperienced shooters is not centering their shot groups. This is difficult to pick up when shooting one shot per bull on 10-bull paper targets, but a target scored by Orion with its individual shooter printout can provide graphic proof of sight adjustment errors.

During the past three years when Orion was being developed, earlier versions were used to score over one hundred thousand CMP postal targets. Orion made it possible, for the first time, to score every postal target to the same standard. Before that, the CMP had to recruit 25 to 30 people to score its annual JROTC postal targets. Now the CMP uses three or four people to do the same work. The American Legion now uses Orion to score all of its annual postal and competition targets. Orion was also used to score several other major air rifle competitions, including the 2008 National JROTC Service Championships, the 2008 National Junior Olympic Three-Position Air Rifle Championship and the last two Montgomery Bell Classics. The list of air rifle matches that are now being scored with Orion is growing quickly.

Orion was originally developed in response to the challenge of determining whether modern computer technology could be used to score 10-bull indoor rifle targets. VIS systems have been around since the mid-1980s. When the ISSF inaugurated finals at the 1988 Olympic Games, a VIS system from the German DISAG company was used to score final round rifle and pistol targets. Systems of this type are used in Europe to score single bull targets, but Orion is the first VIS system that can score multiple-bull targets and its scoring algorithms are newer and take full advantage of recent research in the field of visual image systems.

Dr. Erik Anderson, who was a member of the University of Kentucky Rifle Team as an undergraduate and later earned his PhD. in electrical engineering from Kansas University, designed and developed Orion. Anderson was also the lead developer of the CMP’s Competitor Tracker Internet-based results system that made the CMP a pioneer in utilizing technology to manage competitions and quickly post results on the Internet where they are available to public viewers.

How accurate is Orion? The answer—extremely accurate. Of the three scoring methods now in use, Orion is probably the most consistently accurate. With shot holes on good target paper, Orion has a theoretical accuracy of +/- 0.04mm. The theoretical accuracy of electronic or EST targets is +/- 0.08mm while their practical accuracy is in the +/- 0.12mm range. The target tolerance standard for 10-meter paper targets is +/- 0.1mm, and even if targets were absolutely consistent, the trained human eye in good light is probably capable of a scoring tolerance only slightly less than that. And even if manual and Orion scoring were both theoretically capable of the same degree of precision, Orion never gets distracted or tired.

Does Orion make mistakes? Of course--no scoring system is perfect and Orion will on rare occasions misread a badly torn shot hole, miss a double shot, or fail to pick up a shot in the white. The cause of the first error is obvious; shot holes must be reasonably well formed to score correctly. The most common cause of this error is poorly maintained pneumatic sporter air rifles with muzzle velocities below 450 fps (minimum factory standard at Daisy). The second error stems from the fact that Orion expects to find only one shot on each aiming bull. If a shooter fires two shots on a bull, Orion will reliably score the first shot and is likely to find the second. However, because Orion only expects to find one shot per bull, after it finds and scores the first shot, it will not look as hard for additional shots on the same bull. Orion will score two shots that overlap, but the two shots have to be offset by about a millimeter. The final error, missing shots outside the scoring rings, usually is caused by the white background of a scan making it impossible to distinguish the shot hole with a white background from the white target.

To protect against errors like this, the Orion Scoring System alerts the user of any abnormalities on the targets. After a target is scored, a verifier can see on the computer screen whether Orion found and scored ten shot holes. The screen also gives Orion’s confidence level for the accuracy of each shot scored. The verifier can then call up the scanned image of each bull that is missing a score to see if somehow a shot was missed. The verifier can also view any bull that displays a low confidence level to see if there is something irregular about that shot hole. If the verifier finds a mistake, it is possible to make a manual adjustment to assist Orion in finding the shot hole.

One of the initial debates concerned how to handle scoring protests when Orion was used to score matches. At first, competitors were allowed to challenge shots that were then rescored manually. Trying to make sense out of that challenge system led the National Three-Position Air Rifle Council to realize how unfair and incorrect it is to use one method of scoring to rescore targets scored by another method. Orion determines the distance from the target center to the shot center, while manual target scoring analyzes the locations of the edge of the shot hole and edge of a scoring ring. To be fair to all competitors, all targets from a match must be scored with the same scoring method. Scoring rings are printed on Orion targets, but they are there for shooters’ reference only; they are not to be used to rescore challenged shots.

To facilitate challenges, Orion now offers the possibility of rescoring the original scan with a much slower, higher resolution scoring pass, but that may soon become unnecessary. The rules for challenging shots scored by Orion are being reevaluated. No European shooting federation that uses VIS systems to score paper targets permits challenging scored shots unless there is an obvious error in scoring. In manual target scoring, shots scored with a gauge may not be rescored. Shots scored by EST systems also cannot be reevaluated unless there is an obvious error. To apply the same standards to Orion, the future adoption of a scoring protest procedure similar to that used for electronic targets can be expected.

There were several challenges in developing Orion. Orion scores scanned images of targets so to be commercially viable, it had to be compatible with scanners that are affordable and readily available. Indeed, a goal was to produce a scoring system that utilized images from common flatbed scanners. The problem with them was that normal 10-bull targets are nine inches wide while most scanners will not handle paper wider than 8 ½ inches. That required designing a new target to work with commodity scanners. The first version of a “VIS Target” was printed in the U. S. on 8 ½ x 11 paper. A far superior target printed in Europe on slightly narrower A4 paper (8 ¼” width) succeeded that target and is now the approved target for VIS scoring.

Indeed, one of the most vexing problems faced by Orion stemmed from the poor quality of paper targets printed in the U. S. These targets do not produce clean shot holes and in the case of air guns, produce oversize shot holes. When targets designed for the Orion Scoring System were printed on paper used by U. S. target companies, competitors who looked closely at their targets often thought their scores should have been higher than Orion scores. A target paper was needed where shots cut clean holes that were the same size as the air gun pellet. Despite exhaustive efforts, no target manufacturer or paper company in the U. S. offered a paper that compares with the paper used to print targets in Europe. The solution came from going to a European target company and getting them to print 10-bull air rifle targets on paper sized to feed through regular and high-speed scanners. The Krüger Target Company in Dillingen, Germany and Kromer Print AG in Lenzburg, Switzerland respectively were contracted to print 10-bull targets designed for scoring by the Orion system and these targets may now be purchased from either the CMP (http://www.odcmp.com/Programs/targets.htm) or Shooters Tech (http://store.orionscoringsystem.com/targets.html).

By leveraging advances in computer science, the Orion Scoring System offers a solution to one of the most vexing problems in competition management, that is, ensuring that shooters’ scores are evaluated fairly and quickly and at an affordable price. As Orion grows to facilitate the scoring of targets in other shooting disciplines, the day may not be far away when we ask ourselves, “do you remember the time when we used to score targets by hand?”

For full information on purchasing Orion or on obtaining licenses for multi-computers go to the Shooters Tech website at http://www.orionscoringsystem.com. Information is also available by contacting support@shooterstech.net.

NEXT ARTICLE